Published on

How to Backup and Restore an etcd Snapshot on a Kubernetes Control Plane Node

2 min read

Authors
banner

Introduction

This blog post details how to backup and restore the etcd database, which is critical for maintaining the state and operation of your Kubernetes cluster.

Backing up etcd

To create a backup of the etcd database, use the following command:

ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /opt/snapshot-pre-boot.

This command saves a snapshot of the etcd database to /opt/snapshot-pre-boot.db, ensuring you have a point-in-time copy of your data.

Restoring from a ETCD

To restore the etcd database from the snapshot you've taken, execute:

ETCDCTL_API=3 etcdctl --data-dir /var/lib/etcd-from-backup snapshot restore /opt/snapshot-pre-boot.db

This command restores the snapshot to a new directory /var/lib/etcd-from-backup, allowing you to recover the etcd state to the point when the snapshot was taken.

Updating Kubernetes to Use the Restored Data Next, you need to update the etcd.yaml file located in /etc/kubernetes/manifests to use the newly restored data directory:

volumes:
  - hostPath:
      path: /var/lib/etcd-from-backup
      type: DirectoryOrCreate
    name: etcd-data

This configuration change directs the etcd container to use the restored data at /var/lib/etcd-from-backup.

Monitoring and Troubleshooting Monitor the etcd pod's status to ensure it restarts correctly and troubleshoot if necessary:

watch "crictl ps | grep etcd"

If issues arise, such as the pod not reaching the Ready state (1/1), you may need to manually restart it:

kubectl delete pod -n kube-system etcd-

Wait about a minute after executing this command.

© 2024 Karan Pratap Singh