Store API keys, passwords, certificates, and other sensitive data in cloud-native secret management services and access them from K8s clusters.

Introduction

Secrets are essential for the operation of many production systems. Unintended secrets exposure is one of the top risks that should be properly addressed. Developers should do their best to protect application secrets.

The problem becomes even harder, once a company moves to a microservice architecture and multiple services require access to different secrets in order to properly work. This leads to new challenges: how to distribute, manage, monitor, and rotate application secrets while avoiding unintended exposure?

In the previous post (Part I), I showed a way of integrating AWS and Google Cloud secrets management services (AWS Secrets Manager, AWS SSM Parameter Store, and Google Cloud Secret Manager) with Kubernetes, using doitintl/secrets-init initContainer manually added into a target Pod.

In this post, I’m going to present a Kubernetes-native approach to integrating cloud secrets management services mentioned above.

Automatic Cloud Secret Injection

While it is possible to manually modify Kubernetes Deployment YAML files to use secret-init as a container init system, the better option would be if someone could do it for you and do it only for Kubernetes Pods that are referencing cloud secrets. Fortunately for us, Kubernetes allows us to inspect and modify any Pod before the container is created with a mechanism known as a mutating admission webhook.

The doitintl/kube-secrets-init is an open-source project from DoiT International that implements a Kubernetes mutating admission webhook for cloud secrets injection, supporting both AWS and Google Cloud managed secrets.

The kube-secrets-init monitors Kubernetes clusters for newly created or updated Pods, adding an initContainer with doitintl/secrets-init utility to the Pods that are referencing cloud secrets directly (through environment variables) and/or indirectly (through Kubernetes Secret and ConfigMap).

Integration with AWS Secrets Manager

Users can put an AWS secrets ARN reference as an environment variable value. The secrets-init will resolve an environment value, using the specified ARN, to a referenced secret value.

Integration with AWS Systems Manager Parameter Store

It is possible to use the AWS Systems Manager Parameter Store to store application parameters and secrets.

Users can put an AWS Parameter Store ARN reference as an environment variable. The secrets-init will resolve an environment value, using the specified ARN, to a referenced parameter store value.

Integration with Google Secret Manager

Users can put a Google Secret name (prefixed with gcp:secretmanager:) as environment variable value. The secrets-init will resolve an environment value, using the specified name, to a referenced secret value. The secret name can include a secret version, to reference a specific version of the secret.

Requirements

AWS

In order to resolve AWS secrets from AWS Secrets Manager and Parameter Store, the secrets-init application should run under AWS IAM Role with one of the following IAM policies attached.

For AWS Secrets Manager:

For AWS Systems Manager Parameter Store:

When running in the EKS cluster, it’s recommended to use AWS IAM Roles for Service Account. It’s also possible to assign an IAM Role to an EC2 instance, where the container is running. This option is considered to be less secure.

Google Cloud

In order to resolve Google secrets from Google Secret Manager, the secrets-init application should run under an IAM role with sufficient permissions to access desired secrets. For example, you can assign the following two predefined Google IAM roles to a Google Service Account: Secret Manager Viewer and Secret Manager Secret Accessor role.

In the GKE cluster, it is possible to assign IAM Role to a Kubernetes Pod with Workload Identity. It’s also possible to assign an IAM Role to a GCE instance, where the container is running. This option is considered to be less secure.

Uncomment --provider=google flag in the deployment.yaml file.

The Webhook Deployment

1. To deploy the kube-secrets-init webhook server, we need to create a webhook service and a deployment in our Kubernetes cluster. It’s pretty straightforward, except for one thing; the server’s TLS configuration. If you’d care to examine the deployment.yaml file, you’ll find that the certificate and corresponding private key files are read from command line arguments and that the path to these files comes from a volume mount that points to a Kubernetes secret:

The most important thing to remember is to set the corresponding CA certificate later in the webhook configuration, so the apiserver will know that it should be accepted. For now, we’ll reuse the script originally written by the Istio team to generate a certificate signing request. Then we’ll send the request to the Kubernetes API, fetch the certificate, and create the required secret from the result.

First, run webhook-create-signed-cert.sh script and check if the secret holding the certificate and key has been created:

Once the secret is created, we can create deployment and service. These are standard Kubernetes deployment and service resources. Up until this point we’ve produced nothing but an HTTP server that’s accepting requests through service on port 443:

Configure Mutating Admission Webhook

Now that our webhook server is running, it can accept requests from the apiserver. However, we should create some configuration resources in Kubernetes first. Let’s start with our validating webhook, then we’ll configure the mutating webhook later. If you take a look at the webhook configuration, you’ll notice that it contains a placeholder for CA_BUNDLE:

There is a small script that substitutes the CA_BUNDLE placeholder in the configuration with this CA. Run this command before creating the validating webhook configuration:

Create a mutating webhook configuration:

Configure RBAC for secrets-init-webhook

Create a Kubernetes Service Account to be used with secrets-init-webhook:

Define RBAC permissions for webhook service account:

Summary

I hope you find this post useful. I look forward to your comments and any questions you have.

You are also invited to contribute (Issues, Features, PRs) to the doitintl/kube-secrets-init GitHub Project.

Want more stories from Alexei? Check our blog, or follow Alexei on Twitter.