Kubernetes allows us to run a containerized application at scale without drowning in the details of application load balancing. You can ensure high availability for your applications running on Kubernetes by running multiple replicas (pods) of the application. All the complexity of container orchestrations is hidden away safely so that you can focus on developing application instead of deploying it. Learn more about high availability of Kubernetes Clusters and how you can use Kubedm for high availability in Kubernetes here.
But using Kubernetes has its own challenges and getting Kubernetes up and running takes some real work. If you are not familiar with getting Kubernetes up and running, you might want to take a look here.
Kubernetes allows us to have a zero downtime deployment, yet service interrupting events are inevitable and can occur at any time. Your network can go down, your latest application push can introduce a critical bug, or in the rarest case, you might even have to face a natural disaster.
When you are using Kubernetes, sooner or later, you need to set up a backup. In case your cluster goes into an unrecoverable state, you will need a backup to go back to the previous stable state of the Kubernetes cluster.
Why Backup and Recovery?
There are three reasons why you need a backup and recovery mechanism in place for your Kubernetes cluster. These are:
- To recover from Disasters: like someone accidentally deleted the namespace where your deployments reside.
- Replicate the environment: You want to replicate your production environment to staging environment before any major upgrade.
- Migration of Kubernetes Cluster: Let’s say, you want to migrate your Kubernetes cluster from one environment to another.
What to Backup?
Now that you know why, let’s see what exactly do you need to backup. The two things you need to backup are:
- Your Kubernetes control plane is stored into etcd storage and you need to backup the etcd state to get all the Kubernetes resources.
- If you have stateful containers (which you will have in real world), you need a backup of persistent volumes as well.
How to Backup?
There have been various tools like Heptio ark and Kube-backup to backup and restore the Kubernetes cluster for cloud providers. But, what if you are not using managed Kubernetes cluster? You might have to get your hands dirty if you are running Kubernetes on Baremetal, just like we are.
We are running 3 master Kubernetes cluster with 3 etcd members running on each master. If we lose one master, we can still recover the master because etcd quorum is intact. Now if we lose two masters, we need a mechanism to recover from such situations as well for production grade clusters.
Want to know how to set up multi-master Kubernetes cluster? Keep reading!
Taking etcd backup:
There is a different mechanism to take etcd backup depending on how you set up your etcd cluster in Kubernetes environment.
There are two ways to setup etcd cluster in kubernetes environment:
- Internal etcd cluster: It means you’re running your etcd cluster in the form of containers/pods inside the Kubernetes cluster and it is the responsibility of Kubernetes to manage those pods.
- External etcd cluster: Etcd cluster you’re running outside of Kubernetes cluster mostly in the form of Linux services and providing its endpoints to Kubernetes cluster to write to.
Backup Strategy for Internal Etcd Cluster:
To take a backup from inside a etcd pod, we will be using Kubernetes CronJob functionality which will not require any etcdctl client to be installed on the host.
Following is the definition of Kubernetes CronJob which will take etcd backup every minute:
`apiVersion: batch/v1beta1kind: CronJobmetadata: name: backup namespace: kube-systemspec: # activeDeadlineSeconds: 100schedule: "*/1 * * * *"
jobTemplate:
spec:
template:
spec:
containers:
- name: backup
# Same image as in /etc/kubernetes/manifests/etcd.yaml
image: k8s.gcr.io/etcd:3.2.24
env:
- name: ETCDCTL_API
value: "3"
command: ["/bin/sh"]
args: ["-c", "etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt --key=/etc/kubernetes/pki/etcd/healthcheck-client.key snapshot save /backup/etcd-snapshot-$(date +%Y-%m-%d_%H:%M:%S_%Z).db"]
volumeMounts:
- mountPath: /etc/kubernetes/pki/etcd
name: etcd-certs
readOnly: true
- mountPath: /backup
name: backup
restartPolicy: OnFailure
hostNetwork: true
volumes:
- name: etcd-certs
hostPath:
path: /etc/kubernetes/pki/etcd
type: DirectoryOrCreate
- name: backup
hostPath:
path: /data/backup
type: DirectoryOrCreateBackup Strategy for External Etcd Cluster:
If you running etcd cluster on Linux hosts as a service, you should set up a Linux cron job to take backup of your cluster.
Run the following command to save etcd backup
ETCDCTL_API=3 etcdctl --endpoints $ENDPOINT snapshot save /path/for/backup/snapshot.dbDisaster Recovery
Now, Let’s say the Kubernetes cluster went completely down and we need to recover the Kubernetes cluster from the etcd snapshot.
Normally, start the etcd cluster and do the kubeadm init on the master node with etcd endpoints.
Make sure you put the backup certificates into /etc/kubernetes/pki folder before kubeadm init. It will pick up the same certificates.
Restore Strategy for Internal Etcd Cluster:
docker run --rm
-v '/data/backup:/backup'
-v '/var/lib/etcd:/var/lib/etcd'
--env ETCDCTL_API=3
k8s.gcr.io/etcd:3.2.24'
/bin/sh -c "etcdctl snapshot restore '/backup/etcd-snapshot-2018-12-09_11:12:05_UTC.db' ; mv /default.etcd/member/ /var/lib/etcd/"
kubeadm init --ignore-preflight-errors=DirAvailable--var-lib-etcdRestore Strategy for External Etcd Cluster
Restore the etcd on 3 nodes using following commands:
ETCDCTL_API=3 etcdctl snapshot restore snapshot-188.db
--name master-0
--initial-cluster master-0=http://10.0.1.188:2380,master-01=http://10.0.1.136:2380,master-2=http://10.0.1.155:2380
--initial-cluster-token my-etcd-token
--initial-advertise-peer-urls http://10.0.1.188:2380
ETCDCTL_API=3 etcdctl snapshot restore snapshot-136.db
--name master-1
--initial-cluster master-0=http://10.0.1.188:2380,master-1=http://10.0.1.136:2380,master-2=http://10.0.1.155:2380
--initial-cluster-token my-etcd-token
--initial-advertise-peer-urls http://10.0.1.136:2380
ETCDCTL_API=3 etcdctl snapshot restore snapshot-155.db
--name master-2
--initial-cluster master-0=http://10.0.1.188:2380,master-1=http://10.0.1.136:2380,master-2=http://10.0.1.155:2380
--initial-cluster-token my-etcd-token
--initial-advertise-peer-urls http://10.0.1.155:2380The above three commands will give you three restored folders on three nodes named master:
0.etcd, master-1.etcd and master-2.etcd
Now, Stop all the etcd service on the nodes, replace the restored folder with the restored folders on all nodes and start the etcd service. Now you can see all the nodes, but in some time you will see that only master node is ready and other nodes went into the not ready state. You need to join those two nodes again with the existing ca.crt file (you should have a backup of that).
Run the following command on master node:
kubeadm token create --print-join-commandIt will give you kubeadm join command, add one –ignore-preflight-errors and run that command on other two nodes for them to come into the ready state.
Conclusion
One way to deal with master failure is to set up multi-master Kubernetes cluster, but even that does not allow you to completely eliminate the Kubernetes etcd backup and restore, and it is still possible that you may accidentally destroy data on the HA environment.
Need help with disaster recovery for your Kubernetes Cluster? Connect with the experts at Velotio!
For more insights into Kubernetes Disaster Recovery check out here.
