Scaling the master group

Overview

Worker node groups may be scaled up and down at any time just by changing the number of nodes in the group in the cluster spec.

Unlike workers, the master group nodes are running etcd cluster, which keeps all the Kubernetes cluster data, and scaling it up and down is more involved.

This article describes procedure for scaling the master group up.

Constraints and pre-requisites

  1. Currently the procedure requires a cluster downtime.

  2. Only iterative group size increase is supported, e.g. 1->3 and 3->5

  3. Only scaling the master group up is supported.

  4. Kubernetes cluster must be in a healthy state with all master and worker nodes up and running and all conditions in green state before the master group can be scaled.

  5. Kublr agent of the following versions (or with a higher build number) must be used:

    • 1.17.4-9 (Kubernetes 1.17.4)
    • 1.17.8-2 (Kubernetes 1.17.8)
    • 1.18.5-2 (Kubernetes 1.18.5)
  6. Etcd cluster must be in a healthy state with all members up and running.

    You can validate it by running etcdctl command in etcd pods:

    • List etcd pods:

      kubectl get pods -n kube-system -o name | grep etcd
      

      Example output:

      pod/k8s-etcd-3f2598adc74694e693d2efa5733868229158fa22aa0388ed5d89ac53c1464159-ip-172-16-13-136.ec2.internal
      pod/k8s-etcd-bb9ac49174362ec876f2f9be250d14c9279cd8883f8376768dd808337961d2a9-ip-172-16-12-207.ec2.internal
      pod/k8s-etcd-f90d086594e05b363c9636dc21036fbe6aac21b84fa94b2eba3fc521ff681af5-ip-172-16-12-214.ec2.internal
      
    • Check etcd cluster members on each master:

      for P in $(kubectl get pods -n kube-system -o name | grep etcd) ; do \
        echo -e "\nEtcd members on pod ${P}:\n"
        kubectl exec -n kube-system "${P}" -- etcdctl member list 2>/dev/null
      done
      

      Example output

      Etcd members on pod pod/k8s-etcd-bb9ac49174362ec876f2f9be250d14c9279cd8883f8376768dd808337961d2a9-ip-172-16-12-207.ec2.internal:
      
      d91c8b1e8e5a44dc, started, etcd0, https://etcd0.kublr.local:2380, https://172.16.12.207:2379, false
      
    • Check etcd cluster endpoints health on each master:

      for P in $(kubectl get pods -n kube-system -o name | grep etcd) ; do \
        echo -e "\nEtcd endpoints on pod ${P}:\n"
        kubectl exec -n kube-system "${P}" -- etcdctl endpoint health --cluster
      done
      

      Example output

      Etcd endpoints on pod pod/k8s-etcd-bb9ac49174362ec876f2f9be250d14c9279cd8883f8376768dd808337961d2a9-ip-172-16-12-207.ec2.internal:
      
      2020-07-05 22:12:43.153435 W | pkg/flags: unrecognized environment variable ETCDCTL_KEY_FILE=/srv/kubernetes/current/etcdclient.key
      2020-07-05 22:12:43.153469 W | pkg/flags: unrecognized environment variable ETCDCTL_CA_FILE=/srv/kubernetes/current/ca.crt
      2020-07-05 22:12:43.153475 W | pkg/flags: unrecognized environment variable ETCDCTL_CERT_FILE=/srv/kubernetes/current/etcdclient.crt
      https://172.16.12.207:2379 is healthy: successfully committed proposal: took = 10.515741ms
      

Scaling up from 1 to 3 masters

  1. Change cluster specification as follows:

    • Set etcd initial cluster state parameter to existing:

      spec:
        ...
        master:
          kublrAgentConfig:
            kublr:
              etcd_flag:
                initial_cluster_state: '--initial-cluster-state=existing'
      
    • Change the upgrade strategy to disable node draining and upgrading all nodes at once.

      Upgrade strategy can be changed back to normal after scaling master group is complete.

      spec:
        ...
        master:
          ...
          updateStrategy:
            drainStrategy:
              skip: true             # skip draining master nodes
            rollingUpdate:
              maxUnavailable: 100%   # upgrade all nodes at once
            type: RollingUpdate
        ...
        nodes:
          ...
          -
            ...
            updateStrategy:
              drainStrategy:
                skip: true           # skip draining worker nodes
              rollingUpdate:
                maxUnavailable: 100% # upgrade all nodes at once
              type: RollingUpdate
        ...
        updateStrategy:
          rollingUpdate:
            maxUpdatedGroups: 1
          type: ResetUpdate          # upgrade all groups at once
      
    • Increase the number of master nodes from 1 to 3 in the cluster specification (in addition to common changes described in the previous section).

      For on-prem (“bring your ouw infrastructure”) clusters add new masters’ addresses as follows:

      spec:
        ...
        master:
          ...
          initialNodes: 3 # change from 1 to 3
          maxNodes:     3 # change from 1 to 3
          minNodes:     3 # change from 1 to 3
          
          locations: # for on-prem/BYOI clusters only ...
            -
              baremetal:
                hosts:
                  - address: 192.168.56.21
                  - address: 192.168.56.22 # specify additional master nodes' addresses
                  - address: 192.168.56.23 # specify additional master nodes' addresses
      

      For on-prem (vSphere) clusters add new masters’ addresses as follows:

      spec:
        ...
        master:
          ...
          initialNodes: 3 # change from 1 to 3
          maxNodes:     3 # change from 1 to 3
          minNodes:     3 # change from 1 to 3
          
          locations: # for on-prem/BYOI clusters only ...
            -
              vSphere:
                ipAddresses:
                  - 192.168.56.21
                  - 192.168.56.22 # specify additional master nodes' addresses
                  - 192.168.56.23 # specify additional master nodes' addresses
      

      For on-prem (vCloud Director) clusters add new masters’ addresses as follows:

      spec:
        ...
        master:
          ...
          initialNodes: 3 # change from 1 to 3
          maxNodes:     3 # change from 1 to 3
          minNodes:     3 # change from 1 to 3
      
          locations: # for on-prem/BYOI clusters only ...
            -
              vcd:
                ipAddresses:
                  - 192.168.56.21
                  - 192.168.56.22 # specify additional master nodes' addresses
                  - 192.168.56.23 # specify additional master nodes' addresses
      
  2. Run cluster update and wait until cluster is back to green healthy state.

  3. Expanding etcd cluster from 1 to 3 members; run the following commands from the cluster CLI:

    List etcd pods:

    # List etcd pods
    $ kubectl get pods -n kube-system -o name | grep etcd
    
    pod/k8s-etcd-01051b38137ab69ab204db17e586f37abdf968f01a997b568c2e00338fbdf4e8-192.168.56.23
    pod/k8s-etcd-1625baa2258e299969c1fa2c9c7765620957dc118eb300c508fadcc59f45133e-192.168.56.22
    pod/k8s-etcd-c6c3f19ea1c3126621056d9fc1906064f547fb62d22c89fd3f8d5bb372b287de-192.168.56.21
    

    Pick the pod corresponding to the first member of the cluster and connect to a console into it

    # Pick the pod corresponding to the first member of the cluster and connect to a console into it
    $ ETCD0_POD=k8s-etcd-c6c3f19ea1c3126621056d9fc1906064f547fb62d22c89fd3f8d5bb372b287de-192.168.56.21
    $ kubectl exec -n kube-system -it "${ETCD0_POD}" -- sh
    

    Check that the list of members is available in that pod, and the list only includes one member:

    # Check that the list of members is available in that pod, and the list only includes one member
    $ etcdctl member list
    
    7d9376dfea92b837, started, etcd0, https://etcd0.kublr.local:2380, https://192.168.56.21:2379, false
    

    Check the health status of the member endpoint

    # Check the health status of the member endpoint
    $ etcdctl endpoint health --cluster
    
    https://192.168.56.21:2379 is healthy: successfully committed proposal: took = 21.564722ms
    

    Add the second and the third members to the cluster

    # Add the second and the third members to the cluster
    $ etcdctl member add etcd1 --learner --peer-urls=https://etcd1.kublr.local:2380
    $ etcdctl member add etcd2 --peer-urls=https://etcd2.kublr.local:2380
    
    # Promoting a member from learner status is possible only after the learner
    # has caught up with the cluster, which may require some time.
    # If the promotion is not successful on the first attempt, wait for some time
    # and try again.
    $ etcdctl member promote "$(etcdctl member list | grep etcd1 | cut -d, -f1)"
    

    Verify that all members are in the cluster and all endpoints are healthy

    # Verify that all members are in the cluster and all endpoints are healthy
    $ etcdctl member list
    
    63b2fc83492413f2, started, etcd2, https://etcd2.kublr.local:2380, https://192.168.56.23:2379, false
    7d9376dfea92b837, started, etcd0, https://etcd0.kublr.local:2380, https://192.168.56.21:2379, false
    b2f3ff8746e503f5, started, etcd1, https://etcd1.kublr.local:2380, https://192.168.56.22:2379, false
    
    $ etcdctl endpoint health --cluster
    
    https://192.168.56.21:2379 is healthy: successfully committed proposal: took = 20.261465ms
    https://192.168.56.23:2379 is healthy: successfully committed proposal: took = 23.343856ms
    https://192.168.56.22:2379 is healthy: successfully committed proposal: took = 24.217642ms
    

Scaling up from 3 to 5 masters

When scaling up from 3 to 5 masters nodes, the procedure should be adjusted accordingly

  1. Change cluster specification as follows:

    • Set etcd initial cluster state parameter to existing - same as for scaling up from 1 to 3 nodes

    • Change the upgrade strategy to disable node draining and upgrading all nodes at once - same as for scaling up from 1 to 3 nodes.

      Upgrade strategy can be changed back to normal after scaling master group is complete.

    • Increase the number of master nodes from 3 to 5 in the cluster specification (in addition to common changes described in the previous section).

      For on-prem (“bring your ouw infrastructure”) clusters add new masters’ addresses as follows:

      spec:
        ...
        master:
          ...
          initialNodes: 5 # change from 3 to 5
          maxNodes:     5 # change from 3 to 5
          minNodes:     5 # change from 3 to 5
          
          locations: # for on-prem/BYOI clusters only ...
            -
              baremetal:
                hosts:
                  - address: 192.168.56.21
                  - address: 192.168.56.22
                  - address: 192.168.56.23
                  - address: 192.168.56.24 # specify additional master nodes' addresses
                  - address: 192.168.56.25 # specify additional master nodes' addresses
      

      For on-prem (vSphere) clusters add new masters’ addresses as follows:

      spec:
        ...
        master:
          ...
          initialNodes: 5 # change from 3 to 5
          maxNodes:     5 # change from 3 to 5
          minNodes:     5 # change from 3 to 5
          
          locations: # for on-prem/BYOI clusters only ...
            -
              vSphere:
                ipAddresses:
                  - 192.168.56.21
                  - 192.168.56.22
                  - 192.168.56.23
                  - 192.168.56.24 # specify additional master nodes' addresses
                  - 192.168.56.25 # specify additional master nodes' addresses
      

      For on-prem (vCloud Director) clusters add new masters’ addresses as follows:

      spec:
        ...
        master:
          ...
          initialNodes: 5 # change from 3 to 5
          maxNodes:     5 # change from 3 to 5
          minNodes:     5 # change from 3 to 5
      
          locations: # for on-prem/BYOI clusters only ...
            -
              vcd:
                ipAddresses:
                  - 192.168.56.21
                  - 192.168.56.22
                  - 192.168.56.23
                  - 192.168.56.24 # specify additional master nodes' addresses
                  - 192.168.56.25 # specify additional master nodes' addresses
      
  2. Run cluster update and wait until cluster is back to green healthy state.

  3. Expanding etcd cluster from 3 to 5 members; run the following commands from the cluster CLI:

    List etcd pods:

    # List etcd pods
    $ kubectl get pods -n kube-system -o name | grep etcd
    
    pod/k8s-etcd-21051b38137ab69ab204db17e586f37abdf968f01a997b568c2e00338fbdf4ea-192.168.56.25
    pod/k8s-etcd-11051b38137ab69ab204db17e586f37abdf968f01a997b568c2e00338fbdf4e9-192.168.56.24
    pod/k8s-etcd-01051b38137ab69ab204db17e586f37abdf968f01a997b568c2e00338fbdf4e8-192.168.56.23
    pod/k8s-etcd-1625baa2258e299969c1fa2c9c7765620957dc118eb300c508fadcc59f45133e-192.168.56.22
    pod/k8s-etcd-c6c3f19ea1c3126621056d9fc1906064f547fb62d22c89fd3f8d5bb372b287de-192.168.56.21
    

    Pick the pod corresponding to the first member of the cluster and connect to a console into it

    # Pick the pod corresponding to the first member of the cluster and connect to a console into it
    $ ETCD0_POD=k8s-etcd-c6c3f19ea1c3126621056d9fc1906064f547fb62d22c89fd3f8d5bb372b287de-192.168.56.21
    $ kubectl exec -n kube-system -it "${ETCD0_POD}" -- sh
    

    Check that the list of members is available in that pod, and the list only includes one member:

    # Check that the list of members is available in that pod, and the list only includes one member
    $ etcdctl member list
    
    7d9376dfea92b837, started, etcd0, https://etcd0.kublr.local:2380, https://192.168.56.21:2379, false
    ad9376dfea92b855, started, etcd1, https://etcd1.kublr.local:2380, https://192.168.56.22:2379, false
    fd9376dfea92b8dd, started, etcd2, https://etcd2.kublr.local:2380, https://192.168.56.23:2379, false
    

    Check the health status of the member endpoint

    # Check the health status of the member endpoint
    $ etcdctl endpoint health --cluster
    
    https://192.168.56.21:2379 is healthy: successfully committed proposal: took = 21.564722ms
    https://192.168.56.22:2379 is healthy: successfully committed proposal: took = 19.324544ms
    https://192.168.56.23:2379 is healthy: successfully committed proposal: took = 20.737437ms
    

    Add the fourth and the fifth members to the cluster

    # Add the fourth and the fifth members to the cluster
    $ etcdctl member add etcd3 --learner --peer-urls=https://etcd3.kublr.local:2380
    $ etcdctl member add etcd4 --peer-urls=https://etcd4.kublr.local:2380
    
    # Promoting a member from learner status is possible only after the learner
    # has caught up with the cluster, which may require some time.
    # If the promotion is not successful on the first attempt, wait for some time
    # and try again.
    $ etcdctl member promote "$(etcdctl member list | grep etcd3 | cut -d, -f1)"
    

    Verify that all members are in the cluster and all endpoints are healthy

    # Verify that all members are in the cluster and all endpoints are healthy
    $ etcdctl member list
    
    7d9376dfea92b837, started, etcd0, https://etcd0.kublr.local:2380, https://192.168.56.21:2379, false
    ad9376dfea92b855, started, etcd1, https://etcd1.kublr.local:2380, https://192.168.56.22:2379, false
    fd9376dfea92b8dd, started, etcd2, https://etcd2.kublr.local:2380, https://192.168.56.23:2379, false
    b2f3ff8746e503f5, started, etcd3, https://etcd3.kublr.local:2380, https://192.168.56.24:2379, false
    63b2fc83492413f2, started, etcd4, https://etcd4.kublr.local:2380, https://192.168.56.25:2379, false
    
    $ etcdctl endpoint health --cluster
    
    https://192.168.56.21:2379 is healthy: successfully committed proposal: took = 20.261465ms
    https://192.168.56.23:2379 is healthy: successfully committed proposal: took = 23.343856ms
    https://192.168.56.22:2379 is healthy: successfully committed proposal: took = 24.217642ms
    https://192.168.56.24:2379 is healthy: successfully committed proposal: took = 19.340043ms
    https://192.168.56.25:2379 is healthy: successfully committed proposal: took = 21.834774ms