My blog

Demystifying High Availability in Kubernetes Using Kubeadm

Written by

admin

in

, , , , , ,

Introduction

The rise of containers has reshaped the way we develop, deploy and maintain the software. Containers allow us to package the different services that constitute an application into separate containers, and to deploy those containers across a set of virtual and physical machines. This gives rise to container orchestration tool to automate the deployment, management, scaling and availability of a container-based application. Kubernetes allows deployment and management of container-based applications at scale. Learn more about backup and disaster recovery for your Kubernetes clusters.

One of the main advantages of Kubernetes is how it brings greater reliability and stability to the container-based distributed application, through the use of dynamic scheduling of containers. But, how do you make sure Kubernetes itself stays up when a component or its master node goes down?
 

 

Why we need Kubernetes High Availability?

Kubernetes High-Availability is about setting up Kubernetes, along with its supporting components in a way that there is no single point of failure. A single master cluster can easily fail, while a multi-master cluster uses multiple master nodes, each of which has access to same worker nodes. In a single master cluster the important component like API server, controller manager lies only on the single master node and if it fails you cannot create more services, pods etc. However, in case of Kubernetes HA environment, these important components are replicated on multiple masters(usually three masters) and if any of the masters fail, the other masters keep the cluster up and running.

Advantages of multi-master

In the Kubernetes cluster, the master node manages the etcd database, API server, controller manager, and scheduler, along with all the worker nodes. What if we have only a single master node and if that node fails, all the worker nodes will be unscheduled and the cluster will be lost.

In a multi-master setup, by contrast, a multi-master provides high availability for a single cluster by running multiple apiserver, etcd, controller-manager, and schedulers. This does not only provides redundancy but also improves network performance because all the masters are dividing the load among themselves.

A multi-master setup protects against a wide range of failure modes, from a loss of a single worker node to the failure of the master node’s etcd service. By providing redundancy, a multi-master cluster serves as a highly available system for your end-users.

Steps to Achieve Kubernetes HA

Before moving to steps to achieve high-availability, let us understand what we are trying to achieve through a diagram:

(Image Source: Kubernetes Official Documentation)

Master Node: Each master node in a multi-master environment run its’ own copy of Kube API server. This can be used for load balancing among the master nodes. Master node also runs its copy of the etcd database, which stores all the data of cluster. In addition to API server and etcd database, the master node also runs k8s controller manager, which handles replication and scheduler, which schedules pods to nodes.

Worker Node: Like single master in the multi-master cluster also the worker runs their own component mainly orchestrating pods in the Kubernetes cluster. We need 3 machines which satisfy the Kubernetes master requirement and 3 machines which satisfy the Kubernetes worker requirement.

For each master, that has been provisioned, follow the installation guide to install kubeadm and its dependencies. In this blog we will use k8s 1.10.4 to implement HA.

Note: Please note that cgroup driver for docker and kubelet differs in some version of k8s, make sure you change cgroup driver to cgroupfs for docker and kubelet. If cgroup driver for kubelet and docker differs then the master doesn’t come up when rebooted.

Setup etcd cluster

1. Install cfssl and cfssljson

$ curl -o /usr/local/bin/cfssl https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 
$ curl -o /usr/local/bin/cfssljson https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
$ chmod +x /usr/local/bin/cfssl*
$ export PATH=$PATH:/usr/local/bin

2 . Generate certificates on master-0

$ mkdir -p /etc/kubernetes/pki/etcd
$ cd /etc/kubernetes/pki/etcd

3. Create config.json file in /etc/kubernetes/pki/etcd folder with following content.

{
    "signing": {
        "default": {
            "expiry": "43800h"
        },
        "profiles": {
            "server": {
                "expiry": "43800h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            },
            "client": {
                "expiry": "43800h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "client auth"
                ]
            },
            "peer": {
                "expiry": "43800h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            }
        }
    }
}

4. Create ca-csr.json file in /etc/kubernetes/pki/etcd folder with following content.

{
    "CN": "etcd",
    "key": {
        "algo": "rsa",
        "size": 2048
    }
}

5. Create client.json file in /etc/kubernetes/pki/etcd folder with following content.

{
    "CN": "client",
    "key": {
        "algo": "ecdsa",
        "size": 256
    }
}

$ cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client client.json | cfssljson -bare client

6. Create a directory  /etc/kubernetes/pki/etcd on master-1 and master-2 and copy all the generated certificates into it.

7. On all masters, now generate peer and etcd certs in /etc/kubernetes/pki/etcd. To generate them, we need the previous CA certificates on all masters.

$ export PEER_NAME=$(hostname)
$ export PRIVATE_IP=$(ip addr show eth0 | grep -Po 'inet K[d.]+')

$ cfssl print-defaults csr > config.json
$ sed -i 's/www.example.net/'"$PRIVATE_IP"'/' config.json
$ sed -i 's/example.net/'"$PEER_NAME"'/' config.json
$ sed -i '0,/CN/{s/example.net/'"$PEER_NAME"'/}' config.json

$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server config.json | cfssljson -bare server
$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer config.json | cfssljson -bare peer

This will replace the default configuration with your machine’s hostname and IP address, so in case if you encounter any problem just check the hostname and IP address are correct and rerun cfssl command.

8. On all masters, Install etcd and set it’s environment file.

$ yum install etcd -y
$ touch /etc/etcd.env
$ echo "PEER_NAME=$PEER_NAME" >> /etc/etcd.env
$ echo "PRIVATE_IP=$PRIVATE_IP" >> /etc/etcd.env

9. Now, we will create a 3 node etcd cluster on all 3 master nodes. Starting etcd service on all three nodes as systemd. Create a file /etc/systemd/system/etcd.service on all masters.

[Unit]
Description=etcd
Documentation=https://github.com/coreos/etcd
Conflicts=etcd.service
Conflicts=etcd2.service

[Service]
EnvironmentFile=/etc/etcd.env
Type=notify
Restart=always
RestartSec=5s
LimitNOFILE=40000
TimeoutStartSec=0

ExecStart=/bin/etcd --name <host_name>  --data-dir /var/lib/etcd --listen-client-urls http://<host_private_ip>:2379,http://127.0.0.1:2379 --advertise-client-urls http://<host_private_ip>:2379 --listen-peer-urls http://<host_private_ip>:2380 --initial-advertise-peer-urls http://<host_private_ip>:2380 --cert-file=/etc/kubernetes/pki/etcd/server.pem --key-file=/etc/kubernetes/pki/etcd/server-key.pem --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.pem --peer-cert-file=/etc/kubernetes/pki/etcd/peer.pem --peer-key-file=/etc/kubernetes/pki/etcd/peer-key.pem --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.pem --initial-cluster master-0=http://<master0_private_ip>:2380,master-1=http://<master1_private_ip>:2380,master-2=http://<master2_private_ip>:2380 --initial-cluster-token my-etcd-token --initial-cluster-state new --client-cert-auth=false --peer-client-cert-auth=false

[Install]
WantedBy=multi-user.target

10. Ensure that you will replace the following placeholder with

  • <host_name> : Replace as the master’s hostname</host_name>
  • <host_private_ip>: Replace as the current host private IP</host_private_ip>
  • <master0_private_ip>: Replace as the master-0 private IP</master0_private_ip>
  • <master1_private_ip>: Replace as the master-1 private IP</master1_private_ip>
  • <master2_private_ip>: Replace as the master-2 private IP</master2_private_ip>

11. Start the etcd service on all three master nodes and check the etcd cluster health:

$ systemctl daemon-reload
$ systemctl enable etcd
$ systemctl start etcd

$ etcdctl cluster-health

This will show the cluster healthy and connected to all three nodes.

Setup load balancer

There are multiple cloud provider solutions for load balancing like AWS elastic load balancer, GCE load balancing etc. There might not be a physical load balancer available, we can setup a virtual IP load balancer to healthy node master. We are using keepalived for load balancing, install keepalived on all master nodes

$ yum install keepalived -y

Create the following configuration file /etc/keepalived/keepalived.conf on all master nodes:

! Configuration File for keepalived
global_defs {
  router_id LVS_DEVEL
}

vrrp_script check_apiserver {
  script "/etc/keepalived/check_apiserver.sh"
  interval 3
  weight -2
  fall 10
  rise 2
}

vrrp_instance VI_1 {
    state <state>
    interface <interface>
    virtual_router_id 51
    priority <priority>
    authentication {
        auth_type PASS
        auth_pass velotiotechnologies
    }
    virtual_ipaddress {
        <virtual ip>
    }
    track_script {
        check_apiserver
    }
}

  • state is either MASTER (on the first master nodes) or BACKUP (the other master nodes).
  • Interface is generally the primary interface, in my case it is eth0
  • Priority should be higher for master node e.g 101 and lower for others e.g 100
  • Virtual_ip should contain the virtual ip of master nodes

Install the following health check script to /etc/keepalived/check_apiserver.sh on all master nodes:

#!/bin/sh

errorExit() {
    echo "*** $*" 1>&2
    exit 1
}

curl --silent --max-time 2 --insecure https://localhost:6443/ -o /dev/null || errorExit "Error GET https://localhost:6443/"
if ip addr | grep -q <VIRTUAL-IP>; then
    curl --silent --max-time 2 --insecure https://<VIRTUAL-IP>:6443/ -o /dev/null || errorExit "Error GET https://<VIRTUAL-IP>:6443/"
fi

$ systemctl restart keepalived

Setup three master node cluster

Run kubeadm init on master0:

Create config.yaml file with following content.

apiVersion: kubeadm.k8s.io/v1alpha1
kind: MasterConfiguration
api:
  advertiseAddress: <master-private-ip>
etcd:
  endpoints:
  - http://<master0-ip-address>:2379
  - http://<master1-ip-address>:2379
  - http://<master2-ip-address>:2379
  caFile: /etc/kubernetes/pki/etcd/ca.pem
  certFile: /etc/kubernetes/pki/etcd/client.pem
  keyFile: /etc/kubernetes/pki/etcd/client-key.pem
networking:
  podSubnet: <podCIDR>
apiServerCertSANs:
- <load-balancer-ip>
apiServerExtraArgs:
  endpoint-reconciler-type: lease

Please ensure that the following placeholders are replaced:

  • <master-private-ip> with the private IPv4 of the master server on which config file resides.</master-private-ip>
  • <master0-ip-address>, <master1-ip-address> and <master-2-ip-address> with the IP addresses of your three master nodes</master-2-ip-address></master1-ip-address></master0-ip-address>
  • <podcidr> with your Pod CIDR. Please read the </podcidr>CNI network section of the docs for more information. Some CNI providers do not require a value to be set. I am using weave-net as pod network, hence podCIDR will be 10.32.0.0/12
  • <load-balancer-ip> with the virtual IP set up in the load balancer in the previous section.</load-balancer-ip>
$ kubeadm init --config=config.yaml

10. Run kubeadm init on master1 and master2:

First of all copy /etc/kubernetes/pki/ca.crt, /etc/kubernetes/pki/ca.key, /etc/kubernetes/pki/sa.key, /etc/kubernetes/pki/sa.pub to master1’s and master2’s /etc/kubernetes/pki folder.

Note: Copying this files is crucial, otherwise the other two master nodes won’t go into the ready state.

Copy the config file config.yaml from master0 to master1 and master2. We need to change <master-private-ip> to current master host’s private IP.</master-private-ip>

$ kubeadm init --config=config.yaml

11. Now you can install pod network on all three masters to bring them in the ready state. I am using weave-net pod network, to apply weave-net run:

export kubever=$(kubectl version | base64 | tr -d 'n') kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$kubever"

12. By default, k8s doesn’t schedule any workload on the master, so if you want to schedule workload on master node as well, taint all the master nodes using the command:

$ kubectl taint nodes --all node-role.kubernetes.io/master-

13. Now that we have functional master nodes, we can join some worker nodes:

Use the join string you got at the end of kubeadm init command

$ kubeadm join 10.0.1.234:6443 --token llb1kx.azsbunpbg13tgc8k --discovery-token-ca-cert-hash sha256:1ad2a436ce0c277d0c5bd3826091e72badbd8417ffdbbd4f6584a2de588bf522

High Availability in action

The Kubernetes HA cluster will look like:

[root@master-0 centos]# kubectl get nodes
NAME       STATUS     ROLES     AGE       VERSION
master-0   NotReady   master    4h        v1.10.4
master-1   Ready      master    4h        v1.10.4
master-2   Ready      master    4h        v1.10.4

[root@master-0 centos]# kubectl get pods -n kube-system
NAME                              READY     STATUS     RESTARTS   AGE
kube-apiserver-master-0            1/1       Unknown    0          4h
kube-apiserver-master-1            1/1       Running    0          4h
kube-apiserver-master-2            1/1       Running    0          4h
kube-controller-manager-master-0   1/1       Unknown    0          4h
kube-controller-manager-master-1   1/1       Running    0          4h
kube-controller-manager-master-2   1/1       Running    0          4h
kube-dns-86f4d74b45-wh795          3/3       Running    0          4h
kube-proxy-9ts6r                   1/1       Running    0          4h
kube-proxy-hkbn7                   1/1       NodeLost   0          4h
kube-proxy-sq6l6                   1/1       Running    0          4h
kube-scheduler-master-0            1/1       Unknown    0          4h
kube-scheduler-master-1            1/1       Running    0          4h
kube-scheduler-master-2            1/1       Running    0          4h
weave-net-6nzbq                    2/2       NodeLost   0          4h
weave-net-ndx2q                    2/2       Running    0          4h
weave-net-w2mfz                    2/2       Running    0          4h

After failing over one master node the Kubernetes cluster is still accessible.

[root@master-0 centos]# kubectl get nodes
NAME       STATUS     ROLES     AGE       VERSION
master-0   NotReady   master    4h        v1.10.4
master-1   Ready      master    4h        v1.10.4
master-2   Ready      master    4h        v1.10.4

[root@master-0 centos]# kubectl get pods -n kube-system
NAME                              READY     STATUS     RESTARTS   AGE
kube-apiserver-master-0            1/1       Unknown    0          4h
kube-apiserver-master-1            1/1       Running    0          4h
kube-apiserver-master-2            1/1       Running    0          4h
kube-controller-manager-master-0   1/1       Unknown    0          4h
kube-controller-manager-master-1   1/1       Running    0          4h
kube-controller-manager-master-2   1/1       Running    0          4h
kube-dns-86f4d74b45-wh795          3/3       Running    0          4h
kube-proxy-9ts6r                   1/1       Running    0          4h
kube-proxy-hkbn7                   1/1       NodeLost   0          4h
kube-proxy-sq6l6                   1/1       Running    0          4h
kube-scheduler-master-0            1/1       Unknown    0          4h
kube-scheduler-master-1            1/1       Running    0          4h
kube-scheduler-master-2            1/1       Running    0          4h
weave-net-6nzbq                    2/2       NodeLost   0          4h
weave-net-ndx2q                    2/2       Running    0          4h
weave-net-w2mfz                    2/2       Running    0          4h

Even after one node failed, all the important components are up and running. The cluster is still accessible and you can create more pods, deployment services etc.

[root@master-1 centos]# kubectl create -f nginx.yaml 
deployment.apps "nginx-deployment" created
[root@master-1 centos]# kubectl get pods -o wide
NAME                                READY     STATUS    RESTARTS   AGE       IP              NODE
nginx-deployment-75675f5897-884kc   1/1       Running   0          10s       10.117.113.98   master-2
nginx-deployment-75675f5897-crgxt   1/1       Running   0          10s       10.117.113.2    master-1

Conclusion

High availability is an important part of reliability engineering, focused on making system reliable and avoid any single point of failure of the complete system. At first glance, its implementation might seem quite complex, but high availability brings tremendous advantages to the system that requires increased stability and reliability. Using highly available cluster is one of the most important aspects of building a solid infrastructure.

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

More posts