This document details the architecture of the clusters deployed by Kublr in AWS.
One Azure location in Kublr cluster spec always corresponds to a Resource Group stack created by Kublr in an
Azure subscription in an Azure region specified via the location spec
Kublr cluster Azure deployment customizations are defined in the Kublr cluster specification
The following diagram shows a typical Azure deployment configuration for a Kublr cluster.
Kublr uses deployments and ARM templates to provision the infrastructure.
The following main components comprise a typical Kublr Azure Kubernetes cluster:
Resource group and subscription
By default Kublr creates a cluster within a single resource group in a single region and under one subscription. It is possible to set up multi-region multi-subscription and multi-resource-group deployments via advanced custom configuration.
The resource group name is by default the same as the cluster name.
Storage account and storage contaner for the cluster secrets exchange
Kublr configures cluster master and worker nodes to run Kublr agent - a service responsible for Kubernetes initialization, configuration, and maintenance. Kublr agents running on different nodes exchange information via so called “secret store”, which is implemented as an Azure storage container in case of Azure deployments.
Virtual network, subnet, and route table
A virtual network is set up with one subnet and a default routing table. It is possible to customize it via Kublr cluster specification to any necessary degree - including additional subnets, specific routing rules, additional security rules, VPN peerings etc.
Network and application security groups
Network security group is set up to ensure the virtual network security.
Public and private load balancers and a public master IP
Kublr sets up a public and a private load balancers with a public master IP by default. This can be customized to e.g. disable public load balancer if a completely private cluster needs to be configured.
Other customization options include using different SKU for load balancers (‘standard’ SKU is used by default) and many others.
Master and worker node groups objects
These objects include all or some of the following objects depending on a corresponding node group type and parameters: virtual machines, managed disks, network interfaces, availability sets, and virtual machine scale sets.
Specific objects created for different group types and node group parameters are described below.
Kublr cluster is fully defined by a cluster specification - a structured document usually in YAML format.
As a cluster can potentially use several infrastructure providers, different providers used by the cluster are abstracted as “locations” in the cluster specification. Each location includes provider-specific parameters.
An Azure location can look as follows in the cluster specification:
spec: ... locations: - name: az # the location name used to reference it azure: # Azure-specific location parameters azureApiAccessSecretRef: azure-secret # reference to Azure API creds region: eastus enableMasterSSH: true ... --- kind: Secret metadata: name: azure-secret spec: azureApiAccessKey: tenantId: '***' subscriptionId: '***' aadClientId: '***' aadClientSecret: '***'
Normally an Azure location in a Kublr cluster specification corresponds to one Azure resource group under a certain subscription, that includes all Azure resources required by the cluster, such as virtual machines, load balancers, virtual network, subnet, etc.
Detailed description of all parameters available in a Kublr cluster specification for an Azure location is available in the API Types Reference > AzureLocationSpec.
The main part of a Kubernetes cluster is a compute node that runs containerized applications.
Kublr abstracts Kubernetes nodes as cluster instance groups. There are two main categories of the instance groups in a Kublr Kubernetes cluster: master groups and worker groups.
The master group is a dedicated node groups used to run Kubenretes API server components and etcd cluster. The
master group is decribed in
spec.master sections of the cluster specification.
The worker groups are node groups used to run application container. The worker groups are decribed in
Besides the fact that they are located in different sections of the cluster specification, master and worker group specifications have exactly the same structure documented in the API Types Reference > InstanceGroupSpec.
While generaly speaking andinstance group is just a set of virtual or physical machines and thus has a lot of common
characteristics on different infrastructure providers (such as AWS, Azure, vSphere, GCP etc), certain provider-specific
parameters are necessary to customize group implementation. These parameters are defined in the
supbsection of the group specification. For example, the following is a cluster specification snippet showing
a worker group specification with generic and Azure specific parameters:
spec: ... # cluster spec parameters not related to the example are omitted locations: # cluster may potentially span several locations - name: azure azure: ... # Azure-specific parameters for the location nodes: # an array of worker node groups - name: example-group1 # worker node group name stateful: true # groups may be stateful or stateless autoscaling: false # whether autoscaling is enabled for this group locations: - locationRef: azure # azure: groupType: VirtualMachineScaleSet # the group type defining the group implementation zones: ["1", "2"] # Azure-specific parameters pinToZone: pin # Azure-specific parameters ...
Detailed description of all parameters available in a Kublr cluster specification for Azure instance groups is available in the API Types Reference > AzureInstanceGroupLocationSpec.
Types of Azure node groups:
|VirtualMachine||Stateful||Master or Worker||A set of standalone virtual machines|
|AvailabilitySet||Stateful||Master or Worker||A set of standalone virtual machines in an Availability Set|
|VirtualMachineScaleSet||Stateful||Master or Worker||A set of VMSS’ (Virtual Machine Scale Sets) each of size 1|
|VirtualMachineScaleSet||Stateless||Worker only||One VMSS of size defined by the node group size|
|AvailabilitySetLegacy||Stateful||Master or Worker||A set of standalone virtual machines in one cluster-wide Availability Set; the model used by Kublr pre-1.20|
As shown in the table the only group type that supports stateless groups is VirtualMachineScaleSet. All other group types MUST be stateful.
One cluster can contain multiple groups of different types, although some restrictions are in place when using ‘Basic’ SKU load balancers, see Load Balancers section for more details.
The group type may be omitted in the initial cluster specification submission, in which case Kublr will automatically assign the group type as follows:
|Description||Default group type|
|Master and stateful worker groups with zones||VirtualMachine|
|Master and stateful worker groups without zones||AvailabilitySet|
VirtualMachine group type is implemented as a set of regular Azure virtual machines. Each machine tagged with the group name and the unique machine ID within the group (from 0 to max-1).
VirtualMachineAvailabilitySet group type is implemented as a set of regular Azure virtual machines in a single availability set. Each machine is tagged with the group name and the machine ID within the group (from 0 to max-1).
Stateless VirtualMachineScaleSet group type is implemented as a single VMSS of the size between min and max group size, and containing all VMs of the group. The VMSS is tagged with the group name.
Stateful VirtualMachineScaleSet group type is implemented as a set of VMSS each of size 1. Each VMSS is tagged with the group name and the unique machine ID within the group (from 0 to max-1).
While in many aspects master groups are exactly same as worker groups, there are several important differences:
master group name is always ‘master’ - Kublr will enforce this constraint
master group is always stateful
master group size can only be 1, 3, or 5
master group cannot be auto-scaled
persistend data disks are created by Kublr for each master node
Every node, master or worker, has an ephemeral OS root disk that exists only for the lifetime of the node. Master nodes, in addition to OS disks, have fully persistent managed disks used to store etcd and other cluster system data. This makes it possible to delete and restore master nodes / VMs without losing cluster system data.
master VMs may have a public IP and a load balancing rule on the public load balancer to enable public Kubernetes API endpoint (optional)
master VMs may have a load balancing rule on the private load balancer to enable private Kubernetes API endpoint
master VMs may have public NAT rules on the public load balancer to enable SSH access (disabled by default)
All group types are supported for masters (except for stateless VMSS), as well as zones and zone pinning (see Zone Support section for more details).
Kublr 1.20 introduced a number of improvements in Azure clusters infrastructure architecture that were not directly compatible with previously used architecture. To maintain compatibility and enable smooth migration from Kublr 1.19 to Kublr 1.20 the pre-1.20 architecture is supported through specific combination of parameters in the cluster specification. Migration of existing pre-1.20 cluster specifications to Kublr 1.20 format is performed automatically on the first cluster update in Kublr 1.20.
The following combination of parameters in Kublr 1.20 allows to achieve compatibility with pre-1.20 Azure archietcture:
spec: locations: - azure: loadBalancerSKU: Basic enableMasterSSH: true kublrAgentConfig: kublr_cloud_provider: azure: vm_type: '' master: locations: - azure: groupType: AvailabilitySetLegacy masterLBSeparate: true nodes: - locations: - azure: groupType: AvailabilitySetLegacy
A pre-1.20 Kublr Azure Kubernetes cluster can be imported into a KCP 1.20+ or can be used after KCP upgrade to 1.20+ as follows:
It is still recommended to migrate a pre-1.20 Kublr Azure Kubernetes cluster to a new Azure architecture as soon as convenient. Use the procedure described on the support portal Migrate Pre-Kublr-1.20 Azure Cluster Architecture to 1.20 Architecture for migration.