Azure Deployment Architecture

This document details the architecture of the clusters deployed by Kublr in AWS.

Overview

One Azure location in Kublr cluster spec always corresponds to a Resource Group stack created by Kublr in an Azure subscription in an Azure region specified via the location spec spec.locations[*].azure.azureApiAccessSecretRef and spec.locations[*].azure.region properties.

Kublr cluster Azure deployment customizations are defined in the Kublr cluster specification sections spec.locations[*].azure, spec.master.locations[*].azure and spec.nodes[*].locations[*].azure.

Deployment layout

The following diagram shows a typical Azure deployment configuration for a Kublr cluster.

Azure Kublr Deployment

Kublr uses deployments and ARM templates to provision the infrastructure.

The following main components comprise a typical Kublr Azure Kubernetes cluster:

  • Resource group and subscription

    By default Kublr creates a cluster within a single resource group in a single region and under one subscription. It is possible to set up multi-region multi-subscription and multi-resource-group deployments via advanced custom configuration.

    The resource group name is by default the same as the cluster name.

  • Storage account and storage contaner for the cluster secrets exchange

    Kublr configures cluster master and worker nodes to run Kublr agent - a service responsible for Kubernetes initialization, configuration, and maintenance. Kublr agents running on different nodes exchange information via so called “secret store”, which is implemented as an Azure storage container in case of Azure deployments.

  • Virtual network, subnet, and route table

    A virtual network is set up with one subnet and a default routing table. It is possible to customize it via Kublr cluster specification to any necessary degree - including additional subnets, specific routing rules, additional security rules, VPN peerings etc.

  • Network and application security groups

    Network security group is set up to ensure the virtual network security.

  • Public and private load balancers and a public master IP

    Kublr sets up a public and a private load balancers with a public master IP by default. This can be customized to e.g. disable public load balancer if a completely private cluster needs to be configured.

    Other customization options include using different SKU for load balancers (‘standard’ SKU is used by default) and many others.

  • Master and worker node groups objects

    These objects include all or some of the following objects depending on a corresponding node group type and parameters: virtual machines, managed disks, network interfaces, availability sets, and virtual machine scale sets.

    Specific objects created for different group types and node group parameters are described below.

Cluster Architecture

Azure location

Kublr cluster is fully defined by a cluster specification - a structured document usually in YAML format.

As a cluster can potentially use several infrastructure providers, different providers used by the cluster are abstracted as “locations” in the cluster specification. Each location includes provider-specific parameters.

An Azure location can look as follows in the cluster specification:

spec:
  ...
  locations:
  - name: az # the location name used to reference it
    azure: # Azure-specific location parameters
      azureApiAccessSecretRef: azure-secret # reference to Azure API creds
      region: eastus
      enableMasterSSH: true
      ...
---
kind: Secret
metadata:
  name: azure-secret
spec:
  azureApiAccessKey:
    tenantId: '***'
    subscriptionId: '***'
    aadClientId: '***'
    aadClientSecret: '***'

Normally an Azure location in a Kublr cluster specification corresponds to one Azure resource group under a certain subscription, that includes all Azure resources required by the cluster, such as virtual machines, load balancers, virtual network, subnet, etc.

Detailed description of all parameters available in a Kublr cluster specification for an Azure location is available in the API Types Reference > AzureLocationSpec.

Instance Groups

The main part of a Kubernetes cluster is a compute node that runs containerized applications.

Kublr abstracts Kubernetes nodes as cluster instance groups. There are two main categories of the instance groups in a Kublr Kubernetes cluster: master groups and worker groups.

The master group is a dedicated node groups used to run Kubenretes API server components and etcd cluster. The master group is decribed in spec.master sections of the cluster specification.

The worker groups are node groups used to run application container. The worker groups are decribed in spec.nodes array.

Besides the fact that they are located in different sections of the cluster specification, master and worker group specifications have exactly the same structure documented in the API Types Reference > InstanceGroupSpec.

While generaly speaking andinstance group is just a set of virtual or physical machines and thus has a lot of common characteristics on different infrastructure providers (such as AWS, Azure, vSphere, GCP etc), certain provider-specific parameters are necessary to customize group implementation. These parameters are defined in the location[*].<provider> supbsection of the group specification. For example, the following is a cluster specification snippet showing a worker group specification with generic and Azure specific parameters:

spec:
  ... # cluster spec parameters not related to the example are omitted

  locations: # cluster may potentially span several locations
  - name: azure
    azure:
      ... # Azure-specific parameters for the location

  nodes: # an array of worker node groups
  - name: example-group1 # worker node group name
    stateful: true       # groups may be stateful or stateless
    autoscaling: false   # whether autoscaling is enabled for this group
    locations:
    - locationRef: azure # 
      azure:
        groupType: VirtualMachineScaleSet # the group type defining the group implementation
        zones: ["1", "2"] # Azure-specific parameters
        pinToZone: pin    # Azure-specific parameters
        ...

Detailed description of all parameters available in a Kublr cluster specification for Azure instance groups is available in the API Types Reference > AzureInstanceGroupLocationSpec.

Types of Azure node groups:

Group TypeStateful/StatelessMaster/WorkerDescription
VirtualMachineStatefulMaster or WorkerA set of standalone virtual machines
AvailabilitySetStatefulMaster or WorkerA set of standalone virtual machines in an Availability Set
VirtualMachineScaleSetStatefulMaster or WorkerA set of VMSS’ (Virtual Machine Scale Sets) each of size 1
VirtualMachineScaleSetStatelessWorker onlyOne VMSS of size defined by the node group size
AvailabilitySetLegacyStatefulMaster or WorkerA set of standalone virtual machines in one cluster-wide Availability Set; the model used by Kublr pre-1.20

As shown in the table the only group type that supports stateless groups is VirtualMachineScaleSet. All other group types MUST be stateful.

One cluster can contain multiple groups of different types, although some restrictions are in place when using ‘Basic’ SKU load balancers, see Load Balancers section for more details.

The group type may be omitted in the initial cluster specification submission, in which case Kublr will automatically assign the group type as follows:

DescriptionDefault group type
Master and stateful worker groups with zonesVirtualMachine
Master and stateful worker groups without zonesAvailabilitySet
Stateless groupsVirtualMachineScaleSet

Group type VirtualMachine

VirtualMachine group type is implemented as a set of regular Azure virtual machines. Each machine tagged with the group name and the unique machine ID within the group (from 0 to max-1).

Group type VirtualMachine

Group type VirtualMachineAvailabilitySet

VirtualMachineAvailabilitySet group type is implemented as a set of regular Azure virtual machines in a single availability set. Each machine is tagged with the group name and the machine ID within the group (from 0 to max-1).

Group type VirtualMachineAvailabilitySet

Group type VirtualMachineScaleSet (stateless)

Stateless VirtualMachineScaleSet group type is implemented as a single VMSS of the size between min and max group size, and containing all VMs of the group. The VMSS is tagged with the group name.

Group type VirtualMachineScaleSet (stateless)

Group type VirtualMachineScaleSet (stateful)

Stateful VirtualMachineScaleSet group type is implemented as a set of VMSS each of size 1. Each VMSS is tagged with the group name and the unique machine ID within the group (from 0 to max-1).

Group type VirtualMachineScaleSet (stateful)

Master Groups

While in many aspects master groups are exactly same as worker groups, there are several important differences:

  • master group name is always ‘master’ - Kublr will enforce this constraint

  • master group is always stateful

  • master group size can only be 1, 3, or 5

  • master group cannot be auto-scaled

  • persistend data disks are created by Kublr for each master node

    Every node, master or worker, has an ephemeral OS root disk that exists only for the lifetime of the node. Master nodes, in addition to OS disks, have fully persistent managed disks used to store etcd and other cluster system data. This makes it possible to delete and restore master nodes / VMs without losing cluster system data.

  • master VMs may have a public IP and a load balancing rule on the public load balancer to enable public Kubernetes API endpoint (optional)

  • master VMs may have a load balancing rule on the private load balancer to enable private Kubernetes API endpoint

  • master VMs may have public NAT rules on the public load balancer to enable SSH access (disabled by default)

All group types are supported for masters (except for stateless VMSS), as well as zones and zone pinning (see Zone Support section for more details).

Master group of type VirtualMachine

Load Balancers

TBD

Zone Support

TBD

Legacy (pre-Kublr 1.20) Clusters

Kublr 1.20 introduced a number of improvements in Azure clusters infrastructure architecture that were not directly compatible with previously used architecture. To maintain compatibility and enable smooth migration from Kublr 1.19 to Kublr 1.20 the pre-1.20 architecture is supported through specific combination of parameters in the cluster specification. Migration of existing pre-1.20 cluster specifications to Kublr 1.20 format is performed automatically on the first cluster update in Kublr 1.20.

The following combination of parameters in Kublr 1.20 allows to achieve compatibility with pre-1.20 Azure archietcture:

spec:
  locations:
    - azure:
        loadBalancerSKU: Basic
        enableMasterSSH: true
      kublrAgentConfig:
        kublr_cloud_provider:
          azure:
            vm_type: ''
  master:
    locations:
      - azure:
          groupType: AvailabilitySetLegacy
          masterLBSeparate: true
  nodes:
    - locations:
        - azure:
            groupType: AvailabilitySetLegacy

A pre-1.20 Kublr Azure Kubernetes cluster can be imported into a KCP 1.20+ or can be used after KCP upgrade to 1.20+ as follows:

  1. Kublr control plane is upgraded to 1.20
  2. Pre-1.20 cluster is updated without changes in the cluster spec. This will automatically upgrate the cluster specification structure without changing the cluster architecture.

It is still recommended to migrate a pre-1.20 Kublr Azure Kubernetes cluster to a new Azure architecture as soon as convenient. Use the procedure described on the support portal Migrate Pre-Kublr-1.20 Azure Cluster Architecture to 1.20 Architecture for migration.

Advanced Use-Cases

Enabling Master SSH Access

TBD

Customizing Kublr-generated Azure Resources

TBD

Specifying Additional Azure Resources

TBD

ARM Template Structure Reference

TBD