This document details the architecture of the clusters deployed by Kublr in AWS.
One AWS location in Kublr cluster spec always corresponds to a single CloudFormation stack created by Kublr in an
AWS account in an AWS region specified via the location spec spec.locations[*].aws.awsApiAccessSecretRef
and
spec.locations[*].aws.region
properties.
As the AWS API access secret includes specification of a partition, the region must correspond to it. For example, if the AWS API access secret used is for AWS Gov cloud partition, the region must also be from the AWS Gov cloud.
AWS account ID may also be specified in the spec.locations[*].aws.accountId
property. If it is left empty, it will
be automatically populated with the account ID corresponding to the provided access key.
Kublr cluster AWS network customizations are defined in the Kublr cluster specification
sections spec.locations[*].aws
, spec.master.locations[*].aws
and spec.nodes[*].locations[*].aws
.
The following diagram shows a typical AWS network configuration for a Kublr cluster.
By default Kublr will create a new VPC in that CloudFormation stacks, although this may be overridden and
an existing VPC may be used by specifying the existing VPC ID in the location’s vpcId
property.
The automatically created VPC has NewVpc
CloudFormation logical name and therefore may be referred as { Ref: NewVpc }
.
By default Kublr will create the following subnets:
Master public subnets
These subnets are only created
nodeIpAllocationPolicy
property is unset or
NOT private
), ORnodeIpAllocationPolicy
property is private
)
AND a public load balancer(s) for the master(s) is created (via masterElbAllocationPolicy
or
masterNlbAllocationPolicy
or both properties).One master public subnet is created for each availability zone specified for the master group in the group’s
locations[0].aws.availabilityZones
property. By default, if this property is not specified, the list of
the availability zones is taken from the corresponding AWS location.
Master private subnets
These subnets are only created if the master group is declared private in the cluster spec (nodeIpAllocationPolicy
property is set to private
value).
One master private subnet is created for each availability zone specified for the master group. By default, if this property is not specified, the list of the availability zones is taken from the corresponding AWS location.
Worker public subnets
These subnets are only created if there are public worker groups in the cluster spec.
One worker public subnet is created for each availability zone for which a public worker group exists. By default, if this property is not specified, the list of the availability zones is taken from the corresponding AWS location.
Public worker subnets are shared by all public worker groups that have presence in a corresponding availability zone.
Worker private subnets
These subnets are only created if there are private worker groups in the cluster spec.
One worker private subnet is created for each availability zone for which a private worker group exists. By default, if this property is not specified, the list of the availability zones is taken from the corresponding AWS location.
Private worker subnets are shared by all private worker groups that have presence in a corresponding availability zone.
“Other” public subntes
These subnets are created for NAT gateways and public ELB enpoints to ensure instances in the private groups access to and from the internet.
One “other” public subnet is created for each private subnet.
The automatically generated subnets have CloudFormation logical names in the format
Subnet{Master|Node|Other}{Public|Private}{AZ#}
. Here “AZ#” is the number of a corresponding availability zone, so that
for example AZ us-east-1a
has number 0, us-east-1b
- 1, us-east-1c
- 2, etc.
Thus here are examples of subnets Kublr may generate for a cluster:
SubnetMasterPublic3
- a public master subnet for AZ us-east-1d
(or a corresponding AZ in a different region).SubnetNodePrivate2
- a private workers subnet for AZ us-east-1c
(or a corresponding AZ in a different region).SubnetOtherPublic0
- a public NAT subnet for AZ us-east-1a
(or a corresponding AZ in a different region).See the section Pre-existing Subnets below for advanced use-cases.
Whether a private or a public subnet is used for a specific instance group (master or worker) is defined by the
instance group’s locations[*].aws.nodeIpAllocationPolicy
property.
The property can take three values:
default
(same as privateAndPublic
)private
privateAndPublic
Examples:
spec:
master:
locations:
- aws:
# The master group is public.
# Masters will run in SubnetMasterPublic* subnets.
nodeIpAllocationPolicy: privateAndPublic
nodes:
- locations:
- aws:
# This worker group is private.
# These instances will run in SubnetNodePrivate* subnets.
nodeIpAllocationPolicy: private
- locations:
- aws:
# This worker group is public.
# These instances will run in SubnetNodePublic* subnets.
nodeIpAllocationPolicy: privateAndPublic
A single Kublr location of AWS type may span over several availability zones.
The list of zones is specified in the spec.locations[*].aws.availabilityZones
field. If this field is left empty,
it will be populated with the list of all availability zones accessible under the specified AWS account in the specified
AWS region.
In addition a separate list of availability zones may be specified for each instance group (master or worker) in
the properties spec.master.locations[*].aws.availabilityZones
and spec.nodes[*].locations[*].aws.availabilityZones
.
These lists MUST always be subsets of the list of AZs specified in the corresponding AWS location.
Allocation of master and worker nodes over the availability zones and subnets depends on the instance group type; see more details in the sections Stateless and stateful instance groups.
Examples:
spec:
locations:
# AWS location specification
- name: aws1
aws:
# this location in the cluster is limited to 3 AZs
availabilityZones:
- us-east-1a
- us-east-1c
- us-east-1d
# master group specification
master:
maxNodes: 3 # this is 3-master cluster
locations:
- locationRef: aws1
aws:
availabilityZones:
- us-east-1a # master0 is in us-east-1a
- us-east-1a # master1 is in us-east-1a
- us-east-1c # master2 is in us-east-1c
# worker groups specifications
nodes:
- name: group1
locations:
- locationRef: aws1
aws:
# this group's instances will be distributed over 2 AZs
availabilityZones:
- us-east-1a
- us-east-1c
- name: group2
locations:
- locationRef: aws1
aws:
availabilityZones:
- us-east-1b # ERROR! this AZ is not specified in the location's AZ list
By default Kublr creates an Internet Gateway with a CloudFormation logical name Gw
for a VPC. The gateway is required
for VMs inside the VPC to access the public internet. This is true for NAT gateways as well. See the section
Fully private VPC without NAT and IGW below for more details on building
fully private clusters.
Kublr creates one private subnet for each AZ in which there are/is (a) private node group(s).
Such private subnets require a NAT gateway created in a public subnet.
NAT gateways are created by Kublr by default in every AZ where worker or master nodes may run in private instance groups.
The NAT gateways are placed in corresponding SubnetOtherPublc{AZ#}
subnets
While a NAT gateway for a private subnet does not have to be in the same AZ, if the NAT gateway is in a different AZ, the private subnet internet accessibility is vulnerable to the NAT gateway AZ failures.
By default Kublr will create NAT gateway in each AZ with private node groups.
spec.locations[*].aws.natAvailabilityZones
property allows overriding the default behavior. When natAvailabilityZones
property is specified, for each AZ availabilityZones[i]
NAT gateway from the AZ
natAvailabilityZones[i % len(natAvailabilityZones)]
will be used. So for example
natAvailabilityZones == ['us-east-1c']
, then a single NAT gateway in AZ ‘us-east-1c’
will be used for all private subnets.natAvailabilityZones == ['us-east-1c', 'us-east-1a']
, and
availabilityZones == ['us-east-1a', 'us-east-1b', 'us-east-1d']
then NAT gateways in AZs
‘us-east-1c’, ‘us-east-1a’, and ‘us-east-1c’ (again) will be used for private subnets in AZs
‘us-east-1a’, ‘us-east-1b’, and ‘us-east-1d’ correspondingly.natAvailabilityZones
is undefined, null or empty, NAT gateways will be created in each
AZ with private subnets and private subnet in each AZ will be setup with a NAT gateway in
the same AZ.NAT gateway creation can also be completely disabled via spec.locations[*].aws.natMode
property.
NAT mode can be legacy
, multi-zone
or none
(default: multi-zone
for new clusters, legacy
for
pre-existing ones):
legacy
mode is supported for compatibility with AWS clusters created by pre-1.19 Kublr releases;multi-zone
mode is the default for all new cluster.none
mode is used to avoid automatic creation of NAT gateways.Migration from legacy
to multi-zone
is possible but may affect the cluster public egress addresses,
requires manual operation, and cannot be easily rolled back.
With legacy
NAT mode only one NAT gateway is created in one of the availability zone, which is not
AZ fault tolerant. Public subnet used for the NAT gateway in legacy
mode can change depending on the
configuration of master and worker node groups, which may prevent CloudFormation stack from updating in
some situations. NAT gateways created in multi-zone
mode do not create any issues with any configuration
changes in the clusters, thus never preventing CloudFormation stacks from updating.
Examples:
Skip creating NAT gateways (note that the cluster creation will most probably fail in this case if special measures not taken to enable loading docker images from a private repository accessible from the nodes in the private subnets
spec:
locations:
# AWS location specification
- name: aws1
aws:
natMode: none
The following routing tables are created by Kublr by default:
Public subnet routing table
One routing table is created for all public subnets with an internet route via the Internete gateway.
Private subnet routing tables
For each availability zone in which there are private subnets (master or worker), one private routing tables is created with an internet route pointing at a corresponding NAT gateway.
Master and worker node instance groups in the Kublr cluster specification are mapped onto AWS AutoScaling groups.
Specific AWS objects created for an instance group depend on several instance group parameters:
stateful
- true/false, specifying whether this group is stateful
Stateful groups are mapped onto as many AWS AutoScaling Groups of size one as the size of the group.
Stateful instance groups have fixed size (minNodes
, initialNodes
and maxNodes
must be the same), do not support
autoscaling, but allow to maintain each node’s identity over time and across restarts.
Each node in a stateful group will be assined a unique number from 0
to <group-size> - 1
called “ordinal”. The node’s
ordinal is passed to Kubernetes as a node label kublr.io/node-ordinal
.
Master group is always stateful.
Stateless (non-stateful) groups are mapped onto a single AWS AutoScaling Group of the same (and potentially varying, if autoscaling is enabled) size as specified for the instance group.
Stateless instance groups may have different minNodes
and maxNodes
values, and support autoscaling.
locations[*].aws.groupType
- the AWS archintecture of the instance group. The following values are supported:
asg-lt
- Auto-Scaling Group with a Launch Template
asg-mip
- Auto-Scaling Group with a Launch Template and a Mixed Instance Policy
asg-lc
- Auto-Scaling Group with a Launch Configuration
elastigroup
- Spotinst Elastigroup (see the section Integration with spotinst elastigroups
for more information).
asg
- same as asg-lc
, used in the clusters created by Kublr 1.17 and earlier
locations[*].aws.pinToZone
- whether instances in this instance group are pinned to a specific Availability Zone or not.
Possible values:
pin
- instances are pinned to a specific AZ from the availabilityZones
list.
Due to the fact that only stateful group instances have persistent identity (node ordinal), this value only makes sense and is only allowed for stateful groups.
span
- instances are not pinned to a specific AZ and can be created in any availability
zone from the availabilityZones
list.
This value may be specified for any non-master stateless or stateful group.
default
(default value) - is the same as pin
for stateful groups (including the master group, which is always stateful);
and span
for all the other groups (i.e. non-master stateless groups).
TBD: illustration here
The following instance group properties may be used to customize master and worker instance groups (the properties
may be specified in the spec.master
section for the master group, and spec.nodes[*]
sections for the worker
node groups):
minNodes
, initialNodes
, maxNodes
- instance group size.
Note that for the master group, and more generally for any stateful group these properties MUST all have the same value.
autoscaling
- true/false, if enabled, the AWS autoscaling groups is tagged so that it can be indentified by
the Kubernetes node autoscaler.
locations[*].aws.nodeIpAllocationPolicy
- whether this group is considered private or public.
The property can take three values:
default
(same as privateAndPublic
) - public groupprivate
- private groupprivateAndPublic
- public groupThis property defines whether the instances in this group will be started in private or public subnets, and whether public IPs will be associated with them.
locations[*].aws.sshKey
- specifies the AWS managed SSH key to be provisioned for the instances in this instance
group. If left undefined, not SSH key will be provisioned.
locations[*].aws.instanceType
- specifies the AWS instance type to be used for this instance group (e.g. t3.medium
etc).
See the sesction Autoscaling Groups and Instance Group Types for more
detail on using AWS mixed instance type policy with Kublr instance groups.
locations[*].aws.overrideImageId
, locations[*].aws.imageRootDeviceName
- specifies the AWS AMI image ID
and the image root device name for this instance group. If imageRootDeviceName
is left empty (recommended),
Kublr will automatically populate correct value from the AMI metadata.
locations[*].aws.rootVolume
, locations[*].aws.masterVolume
- specifies the root EBS volume and the master data
(etcd data) parameters for the instances in this instance group. The masterVolume
property is only taken into
account for the master instance group, and is ignored for worker instance groups.
Both properties are structures with the following properties with the direct correspondance to CloudFormation EBS Volume structure properties:
type
- one of the EBS Volume types, e.g. gp2
, io1
, io2
, sc1
, st1
, standard
etc.size
- size in GiBsiops
- provisioned IOPS per GiBencrypted
- the flag indicates if EBS volume should be encryptedkmsKeyId
- ARN or alias of the AWS KMS master key used to create the encrypted volume.snapshotId
- optional snapshot ID to create EBS volume fromdeleteOnTermination
- deleteOnTermination property for ASG EBS mapping volumesproperties directly passed to the AWS AutoScaling Group objects corresponding to this instance group in the CloudFormation template:
cooldown
- stringloadBalancerNames
- array of structurestargetGroupARNs
- array of structuresproperties directly passed to the AWS Launch Configuration and/or Launch Template objects corresponding to this instance group in the CloudFormation template:
blockDeviceMappings
- array of structuresebsOptimized
- booleaninstanceMonitoring
- booleanplacementTenancy
- stringspotPrice
- stringIn addition to specifying whether public IP addresses should be assigned to instances in Kublr instance groups, Kublr allows specifying whether these public IP addressed should be static AWS elastic IPs (EIPs).
This may be customized via spec.master.locations[*].aws.eipAllocationPolicy
property.
Allowed values:
none
- do not create and assign EIP to the instances of this grouppublic
- create and assign EIP to the instances of this groupdefault
- use the following automatic behaviour:none
for private multi-master groupsnone
for private single-master groups (nodeIpAllocationPolicy === 'private'
)public
for public single-master groups (nodeIpAllocationPolicy !== 'private'
)none
for private stateful worker groups (nodeIpAllocationPolicy === 'private' && stateful === true
)public
for public stateful worker groups (nodeIpAllocationPolicy === 'private' && stateful === true
)none
for non-stateful worker groups (nodeIpAllocationPolicy === 'private' && stateful === true
)Constraints:
eipAllocationPolicy
may not be set to public
if the group is private (nodeIpAllocationPolicy === 'private'
),
Kublr will generate an error in this case.eipAllocationPolicy
may not be set to public
if the group is not stateful (stateful === false
)Examples:
spec:
master:
locations:
- aws:
eipAllocationPolicy: public # assign EIPs to the master(s)
nodes:
- stateful: true
locations:
- aws:
eipAllocationPolicy: public # assign EIPs to the workers in this public stateful instance group
- stateful: false
locations:
- aws:
eipAllocationPolicy: public # ERROR! the group is not stateful
- stateful: true
locations:
- aws:
nodeIpAllocationPolicy: private
eipAllocationPolicy: public # ERROR! the group is not public
It is important for Kubernetes clusters to have stable addresses/endpoints for the master node(s). In Kublr AWS clusters it is implemented via AWS load balancers and elastic IPs.
By default Kublr automatically creates public load balancers for masters in multi-master clusters, and assigns public EIP to the master in single-master clusters.
This default behavior may be customized via several properties in the custom cluster specification.
spec.master.locations[*].aws.masterNlbAllocationPolicy
Allowed values:
none
- do not create NLB for mastersprivate
- create a private NLB for masterspublic
- create a public NLB for mastersprivateAndPublic
- create both public and private NLBs for mastersauto
- same as public
for a multi-master cluster, and none
for a single-master clusterdefault
- same as auto
, deprecatedThe property is ignored in non-master instance groups.
By default (value empty):
spec.master.locations[*].aws.masterElbAllocationPolicy
Allowed values:
none
- do not create ELB for mastersprivate
- create a private ELB for masterspublic
- create a public ELB for mastersprivateAndPublic
- create both public and private ELBs for mastersauto
- same as public
for a multi-master cluster, and none
for a single-master clusterdefault
- same as auto
, deprecatedThis property is ignored in non-master instance groups.
By default (value empty):
The use of ELB for masters is deprecated, prefer NLB
spec.master.locations[*].aws.eipAllocationPolicy
property may be used to customize EIP assignement
to master instances.
By default this property will assign EIPs to public single-master master groups, an will not assign EIP in all other cases (private or multi-master master groups). See the section Automatic Elastic IP management above for more detail on the property use.
In AWS when a public load balancer is used to access instances in private subnets, public subnets in the corresponding availability zones must be created. Kublr uses public master subnets for this purpose. As a result, in a cluster that is configured with private masters and (a) public load balancer(s) Kublr will by default generate public master subnets automatically in addition to the private master subnets in which masters are started.
Custom public subnets may be specified for public master load balancers via spec.master.locations[*].aws.privateMasterPublicLbSubnetIds
property. This property will be ignored in any other situation (e.g. for a non-master group, or for public groups,
or when no public ELB or NLB is created).
privateMasterPublicLbSubnetIds
values may be string IDs of specific subnets, or objects allowed for inclusion into
a CloudFormation stack templates, such as { Ref: MySubnet }
It is important to remember that if a subnet ID is specified in a certain position of privateMasterPublicLbSubnetIds
array,
a correct AZ in which this subnet is located MUST also be specified in corresponding position of availabilityZones
array.
Examples:
all possible master endpoints enabled (public/private ELB/NLB, and EIPs)
spec:
master:
locations:
- aws:
eipAllocationPolicy: public
masterNlbAllocationPolicy: privateAndPublic
masterElbAllocationPolicy: privateAndPublic
only create private NLB for public masters
spec:
master:
locations:
- aws:
eipAllocationPolicy: none
masterNlbAllocationPolicy: private
masterElbAllocationPolicy: none
use a custom public subnet for publc LBs in one of the AZs for a private master group
spec:
master:
locations:
- aws:
nodeIpAllocationPolicy: private # private master groups
masterNlbAllocationPolicy: public # create a public NLB
availabilityZones: # masters in the following 2 zones
- us-east-1b
- us-east-1d
privateMasterPublicLbSubnetIds:
- '' # regular Kublr-generated SubnetMasterPublic1 for AZ us-east-1b
- subnet-78178267345 # a custom public subnet for AZ us-east-1d
By default Kublr tags automatically created AWS subnets for the worker and master nodes to ensure proper Kubernetes AWS cloud provider integration for Kubernetes Services of the LoadBalancer type.
See AWS Kubernetes load balancing documentation for more details on subnet tagging for Kubernetes on AWS.
TBD
/**
* This map allows to specify Kublr generator behavior for resources created per AZ(such as
* subnets for example).
*/
availabilityZoneSpec?: {
[azName: string]: AwsLocationAvailabilityZoneSpec;
};
TBD
/**
* If defined, existing AWS IAM objects will be used
* If not defined, Kublr will create corresponding objects (roles and instance profiles).
* If role is created by Kublr, its path and name will be '/kublr/<cluster-name>-<location-name>-(master|node)'
*/
iamRoleMasterPathName?: string;
iamInstanceProfileMasterPathName?: string;
iamRoleNodePathName?: string;
iamInstanceProfileNodePathName?: string;
/**
* Additional parameters for Kublr-generated master and node IAM roles
*/
iamRoleMasterCloudFormationExtras?: any;
iamRoleNodeCloudFormationExtras?: any;
TBD
/**
* Skip creating security groups of different types
*
* Kublr can automatically create security groups for cluster instances.
* In some situations it is not desirable or allowed, in which case the following
* properties can be used to skip automatic security groups creation.
*
* See also 'existingSecurityGroupIds' properties in AWS location and node groups'
* AWS location objects.
*/
skipSecurityGroupDefault?: boolean;
skipSecurityGroupMaster?: boolean;
skipSecurityGroupNode?: boolean;
/**
* GroupId of existing security groups that need to be added to all instances.
* More security groups may be added to specific node groups by specifying additional
* GroupIds in `existingSecurityGroupIds` property of specific groups' `AWSInstanceGroupLocationSpec`
* objects.
*
* existingSecurityGroupIds values may be string IDs of specific security groups, or objects
* allowed for inclusion into a Cloudformation stack templates, such as `{ Ref: MySG }`
*/
existingSecurityGroupIds?: any[];
TBD
spec.locations[*].aws.cidrBlocks
TBD
/**
* map of additional CloudFormation resources to include in the CloudFormation stack template
*/
resourcesCloudFormationExtras?: {};
TBD
lc/lt/mip/elastigroup Specifying ASG update / instance termination policy
TBD
/**
* Additional AWS specific parameters for ASG, LaunchConfiguration, LaunchTemplate, and MixedInstancePolicy
*/
asgCloudFormationExtras?: {};
asgPropertiesCloudFormationExtras?: {};
launchConfigurationPropertiesCloudFormationExtras?: {};
launchTemplateDataCloudFormationExtras?: {
NetworkInterfaces?: Array<{}>;
};
mixedInstancesPolicyCloudFormationExtras?: {
LaunchTemplate?: {
Overrides?: Array<{
InstanceType?: string;
WeightedCapacity?: string | number;
}>;
};
};
It is possible to configure a Kublr cluster so that is uses pre-existing subnets instead of creating subnets automatically.
This is done via the instance groups’ property locations[*].aws.subnetIds
.
By default this property is omitted, and subnets for Kublr instances are created as described in the section Auto-generated subnets above.
Non-default subnet allocation may be specified for each instance group (both master and workers) by setting
some of the elements of subnetIds
array.
Elements of the array correspond to the availability zones in this group’s availabilityZones
array, so if
a subnet is specified in a certain position of the subnetIds
array, this subnet MUST be in the same AZ
as the AZ specified in the same position in the group’s availabilityZones
array.
Empty or undefined value in a position of subnetIds
array means that Kublr with use the auto-generated
subnet for this group’s instances in the corresponding availability zone.
For example, if availabilityZones
is ['us-east1a', 'us-east-1c', 'us-east-1d']
and subnetIds
is
['subnet1', '', { Ref: CustomSubnet3 } ]
, then generator will assume that subnet1
exists in AZ us-east-1a
,
{ Ref: CustomSubnet3 }
exists in us-east-1d
, and the group’s instances starting in us-east-1c
will
run in a subnet generated by Kublr.
Note also that if a subnet id is specified in a certain position of subnetIds
array, a correct
AZ in which this subnet is located MUST also be specified in corresponding position of availabilityZones
array.
subnetIds values may be string IDs of specific subnets, or objects allowed for inclusion into
a Cloudformation stack templates, such as { Ref: MySubnet }
.
Kublr allows creating clusters in fully private VPCs that completely block access to and from the public internet due to the fact that all subnets in the cluster are private and no AWS Internet and NAT Gateways are created in the cluster, as well as public internet routes are not created.
Note that for Kublr to be able to provision a Kubernetes cluster in this configuration, it is the user’s responsibility to ensure network connectivity between the private cluster and the Kublr Control Plane provisioning the cluster, as well as the cluster instances’ access to an image and a binary repository hosting Kublr and Kubernetes images and binaries (by default the public repositories are used).
Complete isolation from the public internet may be achived by using the following properties in Kublr AWS location object:
spec.location[*].aws.skipInternetGateway
- true/false, undefined (same as false) by default
If true - skip automatic creation of AWS Internet Gateway for the VPC
spec.location[*].aws.natMode
NAT mode can be legacy
, multi-zone
or none
(default: multi-zone
for new clusters, legacy
for
pre-existing ones):
legacy
mode is supported for compatibility with AWS clusters created by pre-1.19 Kublr releases;multi-zone
mode is the default for all new cluster.none
mode is used to avoid automatic creation of NAT gateways.Migration from legacy
to multi-zone
is possible but may affect the cluster public egress addresses,
requires manual operation, and cannot be easily rolled back.
With legacy
NAT mode only one NAT gateway is created in one of the availability zone, which is not
AZ fault tolerant. Public subnet used for the NAT gateway in legacy
mode can change depending on the
configuration of master and worker node groups, which may prevent CloudFormation stack from updating in
some situation.
With multi-zone
NAT mode by default a NAT gateway is created for each AZ in which private node groups are
present.
It is also possible to only create NAT gateways in some AZs, and to specify which NAT gateways should be used
by which specific private subnets.
NAT gateways created in multi-zone
mode also do not create any issues with any configuration changes in
the clusters, thus never preventing CloudFormation stacks from updating.
spec.location[*].aws.skipPublicSubnetsForPrivateGroups
- true/false, undefined (same as false) by default
If true - skip creating empty public subnets for private node groups.
By default Kublr creates an empty public subnet for each AZ in which there is at least one private node group. CIDRs for such public subnets are taken from cidrBlocks.otherPublic property.
These public subnets are necessary for public ELB/NLB created by Kubernetes for Services of type LoadBalancer to be able to connect to worker nodes running in private subnets in corresponding AZs.
Note that even if skipPublicSubnetsForPrivateGroups
is true, public subnets may still be created
for NAT gateways for private master and/or worker groups;
Public master subnets will also be created for private master groups if masterElbAllocationPolicy or masterNlbAllocationPolicy requires public load blancer.
It is only possible to fully disable public subnet creation in clusters with:
masterElbAllocationPolicy
and masterNlbAllocationPolicy
values that do not require public load
balancer creation (none
, private
, or auto
in single-master cluster)natMode
set to none
skipPublicSubnetsForPrivateGroups
set to trueTBD
/**
* Elastigroup specification
* This property is ignored if groupType != 'elastigroup'
*/
elastigroup?: ElastigroupSpec;