Blog

How to Protect Your Statefulset Data with Backup for GKE

How-to-Protect-Your-Statefulset-Data-with-Backup-for-GKE

Introduction

Statefulsets are a powerful Kubernetes resource for managing stateful applications. However, StatefulSets can also be complex and difficult to back up. This can lead to data loss in the event of a failure.

Last year (2022), Google announced a new add-on feature for GKE called Backup For GKE. This is a simple and cloud-native way to protect, manage, and restore all containerized applications and data.

In this article we will go through the Backup For GKE components and how to backup volumes from a MySQL StatefulSet deployed in a GKE cluster.

Architecture

Backup for GKE consists of two main components:

  • A service that runs in Google Cloud and supports a resource-based REST API. This service serves as the control plane for Backup for GKE. The service includes Google Cloud console UI elements that interact with this API.
  • An agent that runs in every cluster where backups or restores are performed. The agent runs backup and restore operations in these clusters by interacting with the Backup for GKE API.

The Backup for GKE service integrates with the GKE UI, Google Cloud CLI, and REST APIs, providing consistent workflows for development and operations. Two forms of data are captured in a backup:

  • Config backup: a set of Kubernetes resource manifests extracted from the API server of the cluster undergoing backup, capturing the cluster state.
  • Volume backups: a set of volume backups that correspond to  PersistentVolumeClaim resources found in the config backup.

You can choose which workloads you want to back up or restore, or you can back up or restore all workloads. You can back up workloads from one cluster and restore them to another cluster. You can schedule your backups to automatically run so that you can respond quickly to recover your workloads in the event of an incident.

How To

In the next steps, we will create a zonal GKE cluster, deploy a MySQL stateful to the cluster and test the backup creation as well as the restore for our DB.

Define the variables

To make things easier, let’s create some environment variables:

export PROJECT_ID=felipe-playground-378415
export REGION=us-east1
export LOCATION=us-east1-c
export CLUSTER=backup-for-gke
export BK_PLAN_ALL_NAMESPACES=mysql-bk-plan-all-ns
export RESTORE_PLAN_ALL_NAMESPACES=mysql-bk-plan-all-ns
export BK_ALL_NAMESPACES=mysql-bk-all-namespaces
export RESTORE_ALL_NAMESPACES=mysql-restore-all-namespaces

Create GKE Cluster

Let’s set some variables to make the commands easier. Make sure you add your own  PROJECT_ID. For this exercise, we will be creating a zonal GKE cluster on the  us-east1-c zone. A zonal cluster has a single control plane in a single zone. I recommend using regional clusters for production workloads though as they offer higher availability running multiple replicas of the control plane in multiple zones within a given region. We are also using all default values for our cluster, such as default VPC & subnets, number of nodes, default service account, etc. I don’t recommend using the default values for production clusters.

Make sure you add the  --addons=BackupRestore flag in your command so the backup controllers and all the CRDs are deployed as well.

gcloud container clusters create ${CLUSTER} \
    --release-channel stable \
    --zone ${LOCATION} \
    --node-locations ${LOCATION} \
    --project ${PROJECT_ID} \
    --addons=BackupRestore

Now let’s connect to the cluster

gcloud container clusters get-credentials ${CLUSTER} \
--zone ${LOCATION} \
--project ${PROJECT_ID}

Install a StatefulSet

Now let’s create our StatefulSet. For this exercise, I chose the MySQL operator which is pretty easy and straightforward to install using Helm.

helm repo add mysql-operator https://mysql.github.io/mysql-operator/
helm repo update
helm install my-mysql-operator mysql-operator/mysql-operator \
   --namespace mysql-operator --create-namespace

Install Mysql

Install MySQL with help, please make sure to change your password!

helm install mycluster mysql-operator/mysql-innodbcluster \
   --set tls.useSelfSigned=true \
   --set credentials.root.user=safeuser \
   --set credentials.root.password=notsupersafepassword \
   --set credentials.root.host="%"

With the command,  kubectl get pods,pvc you should be able to see something around those lines

NAME                                    READY   STATUS    RESTARTS   AGE
pod/mycluster-0                         2/2     Running   0          23m
pod/mycluster-1                         2/2     Running   0          23m
pod/mycluster-2                         2/2     Running   0          23m
pod/mycluster-router-67585969f6-m9nd7   1/1     Running   0          22m

NAME                                        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/datadir-mycluster-0   Bound    pvc-d5214172-9b81-4d29-9bf0-9e0dcdc8f691   2Gi        RWO            standard-rwo   23m
persistentvolumeclaim/datadir-mycluster-1   Bound    pvc-df8e188a-51aa-4c4f-acb1-73daa72bc721   2Gi        RWO            standard-rwo   23m
persistentvolumeclaim/datadir-mycluster-2   Bound    pvc-defb5e6e-5946-4183-99d2-c168b05a3d6c   2Gi        RWO            standard-rwo   23m

Perfect! As the next step, we will set up the Backup for GKE resources for our cluster.

Backup for GKE includes two configuration and control resource types:

  •  BackupPlan : a parent resource for Backup resources that represent a chain of backups. This resource contains a backup configuration including the source cluster, the selection of which workloads to back up, and the region in which Backup artifacts produced under this plan are stored.
  •  RestorePlan : provides a reusable restore template. This resource contains a restore configuration including the target cluster in which you want to restore the backup, the source backup plan, the scope of the restore, conflict handling, and substitution rules.

Create a backup Plan for all namespaces

gcloud beta container backup-restore backup-plans create ${BK_PLAN_ALL_NAMESPACES} \
    --project=${PROJECT_ID} \
    --location=${REGION} \
    --cluster=projects/${PROJECT_ID}/locations/${LOCATION}/clusters/${CLUSTER} \
    --all-namespaces \
    --include-secrets \
    --include-volume-data

Create a restore plan for all namespaces

gcloud beta container backup-restore restore-plans create ${RESTORE_PLAN_ALL_NAMESPACES}\
  --all-namespaces \
  --project=${PROJECT_ID} \
  --location=${REGION} \
  --backup-plan=projects/${PROJECT_ID}/locations/${REGION}/backupPlans/${BK_PLAN_ALL_NAMESPACES} \
  --cluster=projects/${PROJECT_ID}/locations/${LOCATION}/clusters/${CLUSTER} \
  --cluster-resource-conflict-policy=use-existing-version \
  --namespaced-resource-restore-mode=delete-and-restore \
  --volume-data-restore-policy=restore-volume-data-from-backup \
  --cluster-resource-restore-scope="storage.k8s.io/StorageClass","scheduling.k8s.io/PriorityClass"

If the commands above went well, we do have all the plans in place.

Create a backup for all namespaces

gcloud beta container backup-restore backups create ${BK_ALL_NAMESPACES} \
--project=${PROJECT_ID} \
--location=${REGION} \
--backup-plan=${BK_PLAN_ALL_NAMESPACES} \
--wait-for-completion

This can take some time depending on the size of your Volumes, but you should see the message below as soon as this is done.

Backup completed. Backup state: SUCCEEDED

Test the backup restore

gcloud beta container backup-restore restores create ${RESTORE_ALL_NAMESPACES} \
  --project=${PROJECT_ID} \
  --location=${REGION} \
  --restore-plan=${RESTORE_PLAN_ALL_NAMESPACES} \
  --backup=projects/$PROJECT_ID/locations/${REGION}/backupPlans/${BK_PLAN_ALL_NAMESPACES}/backups/${BK_ALL_NAMESPACES} \
  --wait-for-completion

Restore Completed. Restore stat: SUCCEEDED

It just works!

Backup for GKE also enables you to back up a single application bundle created through the CRD ProtectedApplication, but this is for another post.

IaC

Today (Jun/2023) we only have one terraform resource to create GKE Backup Plan called  google_gke_backup_backup_plan added on the Google provider version 4.5.0 as you can see here. Hopefully, we will have more resources soon.

Pricing

Backup for GKE accrues fees along two dimensions: first, there is a GKE backup management fee, based on the number of GKE pods protected, and second, there is a backup storage fee, based on the amount of data (GB) stored. Both fees are calculated on a monthly basis, similar to other GKE feature billing.

As an example, a customer with a single backup plan in Iowa (us-central1) that backs up an average of 20 pods during a month, storing 200GB of backup storage data in Iowa, would be charged $25.60 in fees. This $25.60 would include $20 for the month for GKE backup management (20 x $1.00 / pod-month) and $5.60 for backup storage (200 * $0.028 / GB-month).

Starting June 26th 2023, new network egress charges will be introduced for backups that are stored in a different region from their source GKE cluster. These charges will be based on the source and destination region and the number of bytes transferred for each such “cross-region” backup operation

Clean UP

Delete cluster

gcloud container clusters delete ${CLUSTER} --project $PROJECT_ID --zone ${LOCATION}

Don’t forget to delete the backups and plans as well.

# Delete backup
gcloud beta container backup-restore backups delete ${BK_ALL_NAMESPACES} \
  --location=${REGION} \
  --backup-plan=${BK_PLAN_ALL_NAMESPACES}
# Delete backup plan
gcloud beta container backup-restore backup-plans delete ${BK_PLAN_ALL_NAMESPACES} \
  --location=${REGION}
# Delete restore plan
gcloud beta container backup-restore restore-plans delete ${RESTORE_PLAN_ALL_NAMESPACES} \
  --location=${REGION}

I hope you enjoy the journey and please let me know if you have any questions!

Conclusion

Overall, Backup for GKE is a powerful tool that can help you to protect your StatefulSets data. If you are using StatefulSets in Kubernetes, then you should consider using Backup for GKE to protect your data.

Resources:

MySQL :: MySQL Operator for Kubernetes Manual :: 2.1 Install using Helm Charts
Helm is an optional package manager for Kubernetes that helps manage Kubernetes applications; Helm uses charts to…
https://dev.mysql.com/doc/mysql-operator/en/mysql-operator-installation-helm.html

Release v4.50.0 Hashicorp/terraform-provider-google
FEATURES: New Data Source: google_compute_network_peering (#13476) New Data Source: google_compute_router_nat (#13475)…
https://github.com/hashicorp/terraform-provider-google/releases/tag/v4.50.0

Pricing | Google Kubernetes Engine (GKE) | Google Cloud
Review pricing for GKE
https://cloud.google.com/kubernetes-engine/pricing#backup-for-gke

Subscribe to updates, news and more.

Subscribe

Subscribe to updates, news and more.

Leave a Reply

Your email address will not be published. Required fields are marked *

Related blogs

View all blogs
Let’s do it

From cost optimization to cloud migration, machine learning and CloudOps,
we’re here to make the public cloud easy.

From cost optimization to cloud migration, machine learning and CloudOps, we’re here to make the public cloud easy.

Ready to get started?

Get in touch

Company

Offering

Support

Never miss an update.

Subscribe to updates, news and more.

Schedule a call with our team

You will receive a calendar invite to the email address provided below for a 15-minute call with one of our team members to discuss your needs.

You will be presented with date and time options on the next step