Script based Backup of OpenShift/OKD

3 min readFeb 3, 2021

The backup of etcd data from your OpenShift/OKD Cluster would be one of the most import tasks for your production environment in order to comply with compliance requirements Time to Recovery (TTR).

Backup of etcd data is described by OpenShift as a fairly easy 3 step process.

Open a shell terminal on one of the Masternodes with ‘oc debug node <masterNodeName>’
Run ‘chroot /host’ to use host binaries on the Masternode
Run the script a command and define where on the Masternode the backup should be saved. ‘/usr/local/bin/cluster-backup.sh /home/core/assets/backup’

Even though this is quickly done, it must be done manually. And as a DevOps Engineer the optimal goal is to automate this task, so we can control how often the Etcd data is backed up.

We will create a script, which can be triggered from a job scheduling service or from a CI/CD pipeline such as GitLab. Let’s see how we can automate this task with a script in order to avoid human interaction.

Prerequisites

OC command line must be installed
The path to the kubeconfig file with credentials to the cluster should be exported to the variable KUBECONFIG

Export the Masternode name

First we need to export the name of one of the Masternodes into an environment variable. We will query for nodes that has the node-role.kubernetes.io/master label, and then we will select the first item and output the metadata.name og the node with the use of jsonpath=''

export masterNode=$(oc get nodes -l node-role.kubernetes.io/master -o jsonpath='{.items[0].metadata.name}')

The name of the Masternode has now been saved as the environment variable $masterNode which we can now use in the script.

Run the Backup on a Masternode

Next up we will create the actual backup of the OpenShift/OKD cluster by opening a shell on the Masternode so we can run the backup script of Etcd data.

oc debug node/$masterNode --image rhel7/rhel-tools -- chroot /host /usr/local/bin/cluster-backup.sh /home/core/assets/backup

The Backup script will save to files in the /home/core/assets/backup directory.

A snapshot of the of the etcd data: snapshot_YYYY–MM–DD_TIME.db
Resources for the static pods: static_kuberesources_YYYY–MM–DD_TIME.tar.gz

Download the Backup from the Masternode

Currently the backup files are saved on the masternode. It is good practice to have the backup data separately of the currently running cluster — this is in order to avoid situations where the virtual machines running the masternodes cannot be accessed.

We want to download the backup files to where the script is running. Again we can use oc debug node for this task.

First we will download the snapshot file of the etcd database:

oc debug node/$masterNode --image rhel7/rhel-tools -- bash -c 'cat /host/home/core/assets/backup/snapshot_*' > backup/snapshot.db

Then we will download the static resources:

oc debug node/$masterNode --image rhel7/rhel-tools -- bash -c 'cat /host/home/core/assets/backup/static_kuberesources_*' > backup/static_kuberesources.tar.gz

Clean up

Lastly, we went to clean up the backup folder on the master node, so the folder is empty when the next backup will be made.

oc debug node/$masterNode --image rhel7/rhel-tools -- bash -c 'rm /host/home/core/assets/backup/snapshot_* /host/home/core/assets/backup/static_*'

Result

Generally what we have done is to use the “oc debug node” command in order to complete the automated backup task. The script will now perform the following five steps:

Export the Masternode name to an variable
Run the backup script
Download the etcd backup data to your local machine
Download the backup of the static resources to your local machine
Remove the backup data from the Masternode

The code used above can be found in the full script below, this code can be used in order to automate the backup task, so it can be done in a defined interval.

Full script

Upload backup data to Cloud provider

Now we have the backup files locally, and we can now upload these backup files to our favorite storage provider as AWS S3 & Azure Blob Storage, or any other cloud provider or private storage solution.

The backup data has been saved into the ‘backup’ folder on your local machine, therefore we can upload all the content of the folder by using a recursive flag in our CLI command to upload the files.

To upload the backup data to AWS S3 the following command can be used:

aws s3 cp backup/ s3://<bucketName>/ — recursive

To upload the backup data to Azure Blob Storage the following command can be used:

az storage copy -s backup/ -d https://<storageAccountName>.blob.core.windows.net/<containerName>/ — recursive