Minimizing Downtime during EBS Encryption in Kubernetes
In an era of increasing focus on data security and strict confidentiality requirements, encrypting disk storage in cloud environments, such as Amazon EBS (Elastic Block Store), is not just a recommendation but a necessity.
However, a challenge arises when existing EBS volumes in a Kubernetes environment are not initially encrypted. Administrators and engineers face the task of implementing encryption for these volumes, which is associated with the risk of temporary downtime and loss of service availability. This problem is particularly acute for systems where even short-term downtimes are unacceptable, as any interruption can have serious consequences.
In this article, we will focus on methods and strategies for encrypting EBS for StatefulSet in Kubernetes, minimizing or eliminating downtimes.
Encrypting EBS for StatefulSet with Nginx in Kubernetes
Scenario Overview
For our example, we will take a StatefulSet that manages several instances of Nginx. These instances use EBS in AWS for data storage. Our goal is to implement encryption for these EBS volumes without causing downtime in the operation of Nginx.
Step 1: Launching StatefulSet with Nginx
To begin with, we will create a StatefulSet that will manage several instances of Nginx. Here we assume that you have already configured a Kubernetes cluster and have access to its API via kubectl.
Create the following nginx-statefulset.yaml manifest:
Launch the StatefulSet using the command:
Step 2: Checking Encryption of PVC
Now that we have the PVC, it is necessary to check that it is not encrypted. To do this, first look at the StorageClass used by the PVC. Use the command:
Replace `nginx-storage-0` with the name of your PVC. This command will return the name of the StorageClass used by the PVC.
Next, check if this StorageClass supports encryption. Usually, this can be done by looking at the StorageClass documentation or its manifest. In AWS EKS, EBS-level encryption can be enabled in the StorageClass parameters.
Step 3: Creating an Encrypted Storage Class
To encrypt EBS in Kubernetes, you need to create a Storage Class that will use a KMS (Key Management Service) key for encryption. This will ensure that all new volumes created under this Storage Class will be automatically encrypted.
Creating a KMS Key
Before creating the Storage Class, make sure you have a KMS key that you want to use for encryption. If you don’t have a key yet, it can be created in the AWS Management Console or using AWS CLI.
Encrypted Storage Class Manifest
Create a manifest for the Storage Class with the KMS key specified. Below is an example of the manifest:
Replace `<region>`, `<account-id>`, and `<key-id>` with the appropriate values of your KMS key.
Applying the Manifest
Apply the manifest using kubectl:
Now that the Storage Class has been created, it can be used to create encrypted EBS volumes. All PVCs that are created using this Storage Class will automatically be encrypted using the specified KMS key.
Important Points
- Ensure that you have the necessary access rights to use the KMS key in Kubernetes.
- Check if your version of Kubernetes supports EBS encryption using KMS.
- After creating encrypted EBS volumes, you can proceed to migrate data from existing PVCs to the new encrypted volumes, which will be described in the following steps of the article.
Step 4: Creating VolumeSnapshotClass
To create snapshots of your current EBS volumes, it’s necessary to set up a VolumeSnapshotClass. This will enable taking snapshots of the volume states, which can then be used to create encrypted copies of EBS. We will use the instructions provided on the GitHub page of AWS EBS CSI Driver — https://github.com/kubernetes-sigs/aws-ebs-csi-driver/tree/master
Step 5: Saving the StatefulSet Manifest
Before beginning the encryption procedure, it’s essential to save the current StatefulSet manifest for possible recovery or debugging. To do this, use the following command:
Replace `[stsname]` with the name of your StatefulSet.
Step 6: Deleting StatefulSet Without Removing Pods
To delete the StatefulSet while keeping the pods running, use the command:
This will preserve the current state of the pods and prevent their automatic deletion.
Step 7: Deleting a Single Pod
Next, manually delete one of the pods to free up the PVC associated with it. This can be done using the command:
Replace `[podname]` with the name of the pod you want to delete.
Step 8: Recreating PVC Using the New Storage Class
Now, recreate the PVC using the new encrypted Storage Class. For this, you can use a script that automates this process. Here is an example of such a script:
Save this script to a file and execute it, passing the name of the PVC that needs to be recreated. This script will create a snapshot of the current PVC, delete the original PVC, and create a new PVC using the created snapshot and the encrypted Storage Class.
Step 9: Restoring StatefulSet
After you have recreated the PVC for one of the pods, the next step will be to restore the original StatefulSet. This will allow Kubernetes to manage the pods and ensure their correct operation. Use the previously saved StatefulSet manifest for this:
This command will restore the StatefulSet with its original configuration. Kubernetes will automatically connect the new encrypted PVCs to the corresponding pods.
Step 10: Repeating the Process for the Remaining Pods
Now, it is necessary to repeat steps 5 to 9 for each remaining pod in the StatefulSet. This process includes:
Repeat this process for each pod in the StatefulSet until all PVCs have been transitioned to use encrypted EBS volumes. This method minimizes downtimes and ensures a safe migration to encrypted data storage.
Thus, this approach ensures data encryption for StatefulSet in Kubernetes with minimal downtimes, which is critically important for maintaining continuous operation of applications and services.
Nikolay Jusev
QuantumSoft
The guide above was prepared and accomplished by Nikolay Jusev. The opinions expressed here are the author’s own and do not necessarily represent the views of QuantumSoft.
