Velero Backup to PowerScale S3 Bucket
Fri, 23 Dec 2022 21:50:39 -0000
|Read Time: 0 minutes
Velero is one of the most popular tools for backup and restore of Kubernetes resources.
You can use Velero for different backup options to protect your Kubernetes cluster. The three modes are:
- Protect the Kubernetes resource objects such as Pod, Namespace, and so on, with CRDs included
- Protect the PersistentVolume data with the help of VolumeSnapshot
- Protect the content of the PVs with the help of restic
In all cases, Velero syncs the information (YAML and restic data) to a storage object.
PowerScale is Dell Technologies’ leading scale-out NAS solution. It supports many different access protocols including NFS, SMB, HTTP, FTP, HDFS, and, in the case that interests us, S3!
Note: PowerScale is not 100% compatible with the AWS S3 protocol (for details, see the PowerScale OneFS S3 API Guide).
For a simple backup solution of a few terabytes of Kubernetes data, PowerScale and Velero are a perfect duo.
Deployment
To deploy this solution, you need to configure PowerScale and then install and configure Velero.
PowerScale S3 configuration
Prepare PowerScale to be a target for the backup as follows:
- Make sure the S3 protocol is enabled.
You can check that in the UI under Protocols > Object Storage (S3) > Global Settings or in the CLI.
In the UI:
In the CLI:
PS1-1% isi s3 settings global view HTTP Port: 9020 HTTPS Port: 9021 HTTPS only: No S3 Service Enabled: Yes
2. Create a bucket with the permission to write objects (at a minimum).
That action can also be done from the UI or CLI.
In the UI:
In the CLI:
See isi S3 buckets create in the PowerScale OneFS CLI Command Reference.
3. Create a key for the user that will be used to upload the objects.
Important notes:
- The username is the one indicated in the interface, not the one from the file system or provider (for example, here, the admin user is 1_admin_accid S3 user)
- The key is only displayed upon creation and cannot be retrieved later. Be sure to copy it right away.
Now that PowerScale is ready, we can proceed with the Velero deployment.
Velero installation and configuration
We assume that the Velero binary is installed and has access to the Kubernetes cluster. If not, see the Velero installation document for the deployment instructions.
Configure Velero:
- Create a file with the credentials previously obtained from PowerScale.
$ cat ~/credentials-velero [default] aws_access_key_id = 1_admin_accid aws_secret_access_key = 0**************************i …
- Optionally, obtain the PowerScale SSL certificate.
In our case, the HTTPS endpoint uses a self-signed certificate, so we have to get it and pass it to Velero. Note that we can use HTTP protocol, and that step can be skipped at the cost of plain text data transit. For more information on the self-signed certificates in the context of Velero, see https://velero.io/docs/v1.9/self-signed-certificates/. - Install Velero itself:
$ velero install \ --provider aws \ --plugins velero/velero-plugin-for-aws:v1.5.1 \ --bucket velero-backup \ --secret-file ./credentials-velero \ --use-volume-snapshots=false \ --cacert ./ps2-cacert.pem \ --backup-location-config region=powerscale,s3ForcePathStyle="true",s3Url=https://192.168.1.21:9021 …
The preceding command shows how to use Velero most simplistically and securely.
It is possible to add parameters to enable protection with snapshots. Every Dell CSI driver has snapshot support. To take advantage of that support, we use the install command with this addition:
velero install \ --features=EnableCSI \ --plugins=velero/velero-plugin-for-aws:v1.5.1,velero/velero-plugin-for-csi:v0.3.0 \ --use-volume-snapshots=true ...
Now that CSI snaps are enabled, we can enable restic to move data out of those snapshots into our backup target by adding:
--use-restic
As you can see, we are using the velero/velero-plugin-for-aws:v1.5.1 image, which is the latest available at the time of the publication of this article. You can obtain the current version from GitHub: https://github.com/vmware-tanzu/velero-plugin-for-aws
After the Velero installation is done, check that everything is correct:
kubectl logs -n velero deployment/velero
If you have an error with the certificates, you should see it quickly.
You can now back up and restore your Kubernetes resources with the usual Velero commands. For example, to protect the entire Kubernetes except kube-system, including the data with PV snapshots:
velero backup create backup-all --exclude-namespaces kube-system
You can check the actual content directly from PowerScale file system explorer:
Here is a demo:
Conclusion
For easy protection of small Kubernetes clusters, Velero combined with PowerScale S3 is a great solution. If you are looking for broader features (for a greater amount of data or more destinations that go beyond Kubernetes), look to Dell PowerProtect Data Manager, a next-generation, comprehensive data protection solution.
Interestingly, Dell PowerProtect Data Manager uses the Velero plug-in to protect Kubernetes resources!
Resources
Related Blog Posts
Looking Ahead: Dell Container Storage Modules 1.2
Mon, 21 Mar 2022 14:42:31 -0000
|Read Time: 0 minutes
The quarterly update for Dell CSI Drivers & Dell Container Storage Modules (CSM) is here! Here’s what we’re planning.
CSM Features
New CSM Operator!
Dell Container Storage Modules (CSM) add data services and features that are not in the scope of the CSI specification today. The new CSM Operator simplifies the deployment of CSMs. With an ever-growing ecosystem and added features, deploying a driver and its affiliated modules need to be carefully studied before beginning the deployment.
The new CSM Operator:
- Serves as a one-stop-shop for deploying all Dell CSI driver and Container Storage Modules
- Simplifies the install and upgrade operations
- Leverages the Operator framework to give a clear status of the deployment of the resources
- Is certified by Red Hat OpenShift
In the short/middle term, the CSM Operator will deprecate the experimental CSM Installer.
Replication support with PowerScale
For disaster recovery protection, PowerScale implements data replication between appliances by means of the the SyncIQ feature. SyncIQ replicates the data between two sites, where one is read-write while the other is read-only, similar to Dell storage backends with async or sync replication.
The role of the CSM replication module and underlying CSI driver is to provision the volume within Kubernetes clusters and prepare the export configurations, quotas, and so on.
CSM Replication for PowerScale has been designed and implemented in such a way that it won’t collide with your existing Superna Eyeglass DR utility.
A live-action demo will be posted in the coming weeks on our VP YouTube channel: https://www.youtube.com/user/itzikreich/.
CSI features
Across the portfolio
In this release, each CSI driver:
- Supports OpenShift 4.9
- Supports Kubernetes 1.23
- Supports the CSI Spec 1.5
- Updates the latest UBI-minimal image
- Supports fsGroupPolicy
fsGroupPolicy support
Kubernetes v1.19 introduced the fsGroupPolicy to give more control to the CSI driver over the permission sets in the securityContext.
There are three possible options:
- None -- which means that the fsGroup directive from the securityContext will be ignored
- File -- which means that the fsGroup directive will be applied on the volume. This is the default setting for NAS systems such as PowerScale or Unity-File.
- ReadWriteOnceWithFSType -- which means that the fsGroup directive will be applied on the volume if it has fsType defined and is ReadWriteOnce. This is the default setting for block systems such as PowerMax and PowerStore-Block.
In all cases, Dell CSI drivers let kubelet perform the change ownership operations and do not do it at the driver level.
Standalone Helm install
Drivers for PowerFlex and Unity can now be installed with the help of the install scripts we provide under the dell-csi-installer directory.
A standalone Helm chart helps to easily integrate the driver installation with the agent for Continuous Deployment like Flux or Argo CD.
Note: To ensure that you install the driver on a supported Kubernetes version, the Helm charts take advantage of the kubeVersion field. Some Kubernetes distributions use labels in kubectl version (such as v1.21.3-mirantis-1 and v1.20.7-eks-1-20-7) that require manual editing.
Volume Health Monitoring support
Drivers for PowerFlex and Unity implement Volume Health Monitoring.
This feature is currently in alpha in Kubernetes (in Q1-2022), and is disabled with a default installation.
Once enabled, the drivers will expose the standard storage metrics, such as capacity usage and inode usage through the Kubernetes /metrics endpoint. The metrics will flow natively in popular dashboards like the ones built-in OpenShift Monitoring:
Pave the way for full open source!
All Dell drivers and dependencies like gopowerstore, gobrick, and more are now on Github and will be fully open-sourced. The umbrella project is and remains https://github.com/dell/csm, from which you can open tickets and see the roadmap.
Google Anthos 1.9
The Dell partnership with Google continues, and the latest CSI drivers for PowerScale and PowerStore support Anthos v1.9.
NFSv4 POSIX and ACL support
Both CSI PowerScale and PowerStore now allow setting the default permissions for the newly created volume. To do this, you can use POSIX octal notation or ACL.
- In PowerScale, you can use plain ACL or built-in values such as private_read, private, public_read, public_read_write, public or custom ones;
- In PowerStore, you can use the custom ones such as A::OWNER@:RWX, A::GROUP@:RWX, and A::OWNER@:rxtncy.
Useful links
For more details you can:
- Watch these great CSM demos on our VP YouTube channel: https://www.youtube.com/user/itzikreich/
- Read the FAQs
- Subscribe to Github notification and be informed of the latest releases on: https://github.com/dell/csm
- Ask for help or chat with us on Slack
Author: Florian Coulombel
OneFS NDMP Backup Overview
Fri, 15 Dec 2023 15:00:00 -0000
|Read Time: 0 minutes
NDMP (Network Data Management Protocol) specifies a common architecture and data format for backups and restores of NAS (Network Attached Storage), allowing heterogeneous network file servers to directly communicate to tape devices for backup and restore operations. NDMP addresses the problems caused by the integrations of different backup software or DMA (Data Management Applications), file servers, and tape devices.
The NDMP architecture is a client/server model with the following characteristics:
- The NDMP host is a file server that is being protected with an NDMP backup solution.
- The NDMP server is a virtual state machine on the NDMP host that is controlled using NDMP.
- The backup software is considered as a client to the NDMP server.
OneFS supports the following two types of NDMP backups:
- NDMP two-way backup
- NDMP three-way backup
In both backup models, OneFS takes a snapshot of the backup directory to ensure consistency of data. The backup operates on the snapshot instead of the source directory, which allows users to continue read/write activities as normal. OneFS makes entries in the file history that are transferred from the PowerScale cluster to the backup server during the backup.
NDMP two-way backup
The NDMP two-way backup is also known as the local or direct NDMP backup, which is considered the most efficient model and usually provides the best performance. The backup moves the backup data directly from the PowerScale cluster to the tape devices without moving to the backup server over the network.
In this model, OneFS must detect the tape devices before you back up data to them. The PowerScale cluster provides the option for NDMP two-way backups as shown in the following figure. You can connect the PowerScale cluster to a Backup Accelerator node and connect tape devices to that node. The Backup Accelerator node is synonymous with a Fibre Attached Storage node without adding primary storage and offloads NDMP workloads from the primary storage nodes. You can directly connect tape devices to the Fibre Channel ports on the PowerScale cluster or Backup Accelerator node using Fibre Channel. Alternatively, you can connect Fibre Channel switches to the Fibre Channel ports that connect tape devices to the PowerScale cluster or Backup Accelerator node.
The following table shows details of the NDMP two-way backup supported by PowerScale:
NDMP two-way backup option | Generation 5 PowerScale nodes with an InfiniBand back end | Generation 6+ PowerScale nodes with an InfiniBand back end | Generation 6+ PowerScale nodes with an Ethernet back end |
B100 backup accelerator | Supported | Supported | Supported |
Note: The B100 backup accelerator requires OneFS 9.3.0.0 or later.
NDMP three-way backup
The NDMP three-way backup, also known as the remote NDMP backup, is shown in the following figure.
In this backup mode, the tape devices are connected to the backup media server. OneFS does not detect tape devices on the PowerScale cluster, and Fibre Channel ports are not required on the PowerScale cluster. The NDMP service runs on the NDMP server or the PowerScale cluster. The NDMP tape service runs on the backup media server. A DMA on the backup server instructs the PowerScale cluster to start backing up data from the PowerScale cluster to the backup media server over the network. The backup media server moves the backup data to tape devices. Both servers are connected to each other across the network boundary. Sometimes, the backup server and backup media server reside on the same physical machine.
For some specific DMA, DMA can write NDMP data to non-NDMP devices. For example, Dell NetWorker software writes NDMP data to non-NDMP devices, including tape, virtual tape, Advanced File Type Device (AFTD), and Dell PowerProtect DD series appliances. For more information on Data Protection with Dell NetWorker using NDMP, refer to this guide: Dell PowerScale: Data Protection with Dell NetWorker using NDMP.
Author: Jason He, Principal Engineering Technologist