Kubernetes Application Consistency with PowerProtect Data Manager
Thu, 25 Apr 2024 15:58:12 -0000
|Read Time: 0 minutes
In this blog, let’s review application consistency for Kubernetes apps using PowerProtect Data Manager (PPDM).
PowerProtect Data Manager has been providing the ability to run pre/post K8s backup tasks (called hooks) for quite some time now. Now, these hooks can certainly quiesce the database running on K8s pods before the backup starts and end backup as a post-action for app-consistency, but these hooks can also be used to run pre/post actions on the pods as needed.
In this blog, I’ll also cover some use cases of app-consistency for K8s in PPDM and some advanced options. Hang tight, here we go.
Introduction and initial configuration
You can manage K8s application consistency in PowerProtect Data Manager by using the ppdmctl utility, which includes example application templates for the following databases:
- MySQL (standalone and cluster)
- PostgreSQL (standalone and cluster using helm chart)
- Cassandra (standalone)
- MongoDB (standalone and single shard cluster)
Obtain
You can obtain the ppdmctl utility through the PowerProtect Data Manager UI (under System Settings > Downloads > Kubernetes) or directly using the following URL:
https://<your-ppdm-host>/k8s-binaries-download?filename=/usr/local/brs/lib/cndm/misc/pptdmctl.tar.gz
Note that you need to login to the PPDM UI for this link to work.
You can also find the archive on the PPDM host itself at the following path: /usr/local/brs/lib/cndm/misc/ppdmctl.tar.gz
Run
In order to run ppdmctl, you’ll need to extract the archive, change directory, and make it executable:
tar zxvf ppdmctl.tar.gz cd ppdmctl chmod +x ppdmctl
Now, before we see how to apply an application template, let’s look at some useful commands:
ppdmctl help – shows the main help page
ppdmctl applicationtemplate --help – shows help for a specific command, applicationtemplate in this case
ppdmctl applicationtemplate apply --help – shows help for a specific flag, in this case applicationtemplate apply
ppdmctl completion bash | sudo tee -a /etc/bash_completion.d/ppdmctl – applies autocompletion. Note that in this case we’re applying the BASH flavor but Fish, PowerShell and Zsh are also available.
Applying Application Templates
You can apply an application template using the following command, for example by using one of the MySQL example templates:
ppdmctl applicationtemplate apply -i examples/mysqlapptemplate.yaml -n mysql
Applying an application template creates an applicationtemplates.powerprotect.dell.com CR:
kubectl get applicationtemplates -n mysql
NAME AGE mysql 1d
Application Templates Structure
Let’s have a look at the one of the included example application templates for MongoDB:
cat mongodbapptemplate1sts.yaml apiVersion: "powerprotect.dell.com/v1beta1" kind: ApplicationTemplate metadata: name: mongodbtemplate namespace: mongodb-helm spec: enable: true type: "MONGODB" appLabel: "app.kubernetes.io/name:mongodb" appActions: Pod: preHook: command: '["/bin/sh", "-c", "mongo -u root -p $MONGODB_ROOT_PASSWORD $MONGODB_PRIMARY_ROOT_PASSWORD --eval \"db.fsyncLock()\""]' postHook: command: '["/bin/sh", "-c", "mongo -u root -p $MONGODB_ROOT_PASSWORD $MONGODB_PRIMARY_ROOT_PASSWORD --eval \"db.fsyncUnlock()\""]' StatefulSet: selectors: - selectorTerms: - field: "Name" selectorExpression: ".*-[1-9][0-9]*$" - selectorTerms: - field: "Name" selectorExpression: ".*-0$"
Check out the following list for some guidance about the structure and format of app templates. Later in this blog we’ll explore more settings.
- The apiVersion, kind, metadata and spec fields are all mandatory.
- The app template can be provided in a YAML or JSON format.
- The namespace specified must exist and match the only one specified when creating or applying the app template with ppdmctl.
- The type field under spec must match the type specified when applying or creating the app template using ppdmctl.
- appLabel must be specified and should preferably match a single pod on the specified namespace. You can confirm the appropriate label with the following kubectl command:
kubectl get pods -n <your-ns> --show-labels
6. appActions is required and must include Pod and optionally the StatefulSet or Deployment parameters.
7. Either or both preHook and postHook are required.
8. Either the preHook or postHookcommand must be provided as a JSON array. Here are some examples:
- command: '[\"/usr/bin/backup_script\", \"--file\", \"/backups/backup.tgz\"]'
- command: '["/bin/sh", "-c", "mysql -uroot -p$(cat $MYSQL_ROOT_PASSWORD_FILE $MYSQL_MASTER_ROOT_PASSWORD_FILE) -e \"FLUSH TABLES WITH READ LOCK; FLUSH LOGS;SELECT SLEEP(100);\"\"]'
- command: '[\"/bin/sh\", \"-c\", \"BACKUPDIR=/db/backups ;SERVER=localhost; curl -XPOST http://$SERVER:9090/api/v1/admin/tsdb/snapshot;\"]'
9. If you need to exclude PVCs from protection, make sure that all PVCs that are being used by the backed-up pods are included in the backup. Inclusion/exclusion of PVCs can be configured as part of the protection policy, either when creating a new protection policy through the PPDM UI/ REST API or when editing an existing policy.
Using multiple labels
In some cases there are multiple pods that will be matched by a single label. For example, when multiple instances of the MySQL database are provisioned, app=mysql would result in many matched pods. In such cases, you can specify multiple values under appLabel as key-value pairs in a comma-separated list, as in the following examples:
- appLabel: "mariadb-master:yes,app:mariadbprod1"
- appLabel: "app.kubernetes.io/name:mongodb,component=mongo"
Running on a specific container
This use case is quite important for multi-container pods. If you need to run pre/post commands on a specific container on a given pod, you can specify that container under Pod, in the same block as the command. Note that by default the first container is being used unless the container parameter is specified. Check out the following example:
kubectl get pods -n mariadb NAME READY STATUS RESTARTS AGE mariadb-sts-0 2/2 Running 0 20d
kubectl get pods -n mariadb mariadb-sts-0 -o jsonpath='{.spec.containers[*].name}' mariadb maria-tools-sidecar
cat mariadbapptemplate.yaml apiVersion: "powerprotect.dell.com/v1beta1" kind: ApplicationTemplate metadata: name: mariadbtemplate namespace: mariadb spec: type: "MARIADB" enable: true appLabel: "mariadb-master:yes,app:mariadbprod1" appActions: Pod: preHook: command: "[\"/bin/sh\", \"-c\", \"export BACKUPDIR=/var/lib/mysql/backups; if ls $BACKUPDIR/*_full.bak 1> /dev/null 2>&1; then mariabackup --backup --stream=mbstream --extra-lsndir=$BACKUPDIR/backup_incr --incremental-basedir=$BACKUPDIR/backup_base --user=root --password=$MARIADB_ROOT_PASSWORD | gzip --rsyncable > $BACKUPDIR/backup.$(date +%F_%R:%S)_incr.bak; else mariabackup --backup --stream=mbstream --extra-lsndir=$BACKUPDIR/backup_base --user=root --password=$MARIADB_ROOT_PASSWORD | gzip --rsyncable > $BACKUPDIR/backup.$(date +%F_%R:%S)_full.bak; fi; exit $?\"]" container: "mariadb" …
Timeout
Another important capability is controlling the command timeout which can be performed by specifying the timeout parameter. By default, each command has a timeout of 30 seconds. For cases where there is a risk that the command might take longer, the timeout parameter can be specified. This especially relevant for DB dump / backup processes.
For example, let’s look at the first app template but this time with the timeout parameter:
apiVersion: "powerprotect.dell.com/v1beta1"
kind: ApplicationTemplate
metadata:
name: mongodbtemplate
namespace: mongodb-helm
spec:
enable: true
type: "MONGODB"
appLabel: "app.kubernetes.io/name:mongodb"
appActions:
Pod:
preHook:
command: '["/bin/sh", "-c", "mongo -u root -p $MONGODB_ROOT_PASSWORD $MONGODB_PRIMARY_ROOT_PASSWORD --eval \"db.fsyncLock()\""]'
timeout: 60
…
Behavior upon failure
The final advanced capability I want to talk about today is onError. OnError defaults to Fail but Continue is another possible value which means that if a certain pre or post hook fails then the backup flow carries on.
Here’s the last application template but with the OnError parameter this time:
apiVersion: "powerprotect.dell.com/v1beta1" kind: ApplicationTemplate metadata: name: mongodbtemplate namespace: mongodb-helm spec: enable: true type: "MONGODB" appLabel: "app.kubernetes.io/name:mongodb" appActions: Pod: preHook: command: '["/bin/sh", "-c", "mongo -u root -p $MONGODB_ROOT_PASSWORD $MONGODB_PRIMARY_ROOT_PASSWORD --eval \"db.fsyncLock()\""]' timeout: 60 onError: Continue …
Always remember – the documentation is your friend. Specifically, the PowerProtect Data Manager Kubernetes User Guide has some useful information for any PPDM with K8s deployment.
Feel free to reach out with any questions or comments.
Thanks for reading,
Idan
Author: Idan Kentor