Secrets are essential for the operation of many production systems. Unintended secrets exposure is one of the top risks that should be properly addressed. Developers should do their best to protect application secrets.
The problem becomes even harder, once a company moves to a microservice architecture and multiple services require access to different secrets in order to properly work. And this leads to a new challenge: how to distribute, manage, monitor and rotate application secrets, avoiding unintended exposure?
Kubernetes Secrets
Kubernetes provides an object called Secret, which you can use to store application sensitive data, like passwords, SSH keys, API keys, tokens, and others. Kubernetes Secret can be injected into a Pod container either as an environment variable or mounted as a file. Using Kubernetes Secrets allows us to abstract sensitive data and configuration from application deployment.
For example, a Kubernetes secret can be created with the command kubectl create secret:
or Kubernetes db-credentials.yaml file, that describes the same secret:
Please note, that storing sensitive data in a Kubernetes Secret does not make it secure. By default, all data in Kubernetes Secrets is stored as a plaintext encoded with base64.
Starting with version 1.13, Kubernetes supports encrypting Secrets data at rest, using EncryptionConfiguration object with a built-in or external encryption provider.
List of currently supported encryption providers:
Built-in providers: aescbc, aesgsm, secretbox
However, Secrets encryption at rest is not enforced by default. And even when enabled, it is not sufficient and cannot be considered a complete secrets management solution.
A complete secrets management solution must also support; secrets distribution, rotation, fine-grained access control, audit log, usage monitoring, versioning, strongly encrypted storage, convenient API and client SDK/s and probably some other useful features.
Cloud Secrets Management
Multiple cloud vendors provide secret management services, as part of their cloud platform offering, helping you to protect secrets needed to access applications, services, and APIs. Using these services eliminates the need to hardcode sensitive information in plaintext and develop home-grown secrets management lifecycle. Secrets management services enable you to control access to application secrets using fine-grained permissions and auditing.
Integrating Kubernetes with Secrets Management services
In general, any application can use vendor-specific SDK/API to access secrets stored in a secrets management service. Usually, this requires modifying application code or using some kind of bootstrap scripts or Init containers, using client CLI tools or web APIs.
Make it easy with secrets-init tool
Today we are releasing doitintl/secrets-init - open source tool (Apache License 2.0) that simplifies the integration of cloud-native Secrets Management services with containerized workload running on cloud-managed or self-managed Kubernetes clusters.
In its essence, the secrets-init is a minimalistic init system, designed to run as PID 1 inside a container environment, similarly to dumb-init, and provides a fluent integration with multiple cloud-native Secrets Management services, like:
Why you need an init system in Docker container?
Please read Yelp dumb-init repo explanation
Summary:
- Proper signal forwarding
- Orphaned zombies reaping
What secrets-init does
The secrets-init runs as PID 1, acting like a simple init system serving as ENTRYPOINT or first container command, which is responsible to launch a child process, proxying all systems signals to a session rooted at that child process. This is the essence of init process. On the other hand, the secrets-init also passes almost all environment variables without modification, replacing special secret variables with values from Secret Management services.
Integration with Docker
The secrets-init is a statically compiled binary file (without external dependencies) and can be easily embedded included in any Docker image. Download secrets-init binary and use it as the Docker container ENTRYPOINT.
For example:
Integration with Kubernetes
In order to use secrets-init with Kubernetes object (Pod/Deployment/Job/etc) without modifying Docker image, consider injecting secrets-init into a target Pod through initContainer. You can use the doitint/secrets-init Docker image or create your own. Copy secrets-init binary from init container to a common shared volume and change Pod command to run secrets-init as a first command.
Integration with AWS Secrets Manager
To start using secrets-init with AWS Secrets Manager, a user should put an AWS secret ARN as environment variable value. The secrets-init will resolve any environment value, using specified ARN, to a referenced secret value.
Example, using secrets-init with Kubernetes Job:
Integration with AWS Systems Manager Parameter Store
It is possible to use AWS Systems Manager Parameter Store to store application parameters as plaintext or encrypted (kind of secrets).
As with the previous example, the user can put AWS Parameter Store ARN as environment variable value. The secrets-init will resolve any environment value, using specified ARN, to a referenced parameter value.
AWS Systems Manager Parameter Store format example:
In order to resolve AWS secrets from AWS Secrets Manager and Parameter Store, secrets-init should run under the IAM role that has permission to access desired secrets.
This can be achieved by assigning IAM Role to Kubernetes Pod or ECS Task. See Introducing fine-grained IAM roles for service accounts EKS blog post.
It’s possible to assign IAM Role to EC2 instance, where the container is running, but this option is considerably less secure and not recommended.
Integration with Google Secret Manager
Google Cloud released recently a new service for managing secrets in the cloud: Google Secret Manager
User can put Google secret name, using secrets-init recognizable prefix (gcp:secretmanager:), following secret name (projects/{PROJECT_ID}/secrets/{SECRET_NAME} or projects/{PROJECT_ID}/secrets/{SECRET_NAME}/versions/{VERSION}) as environment variable value. The secrets-init will resolve any environment value, using the specified name, to a referenced secret value.
In order to resolve Google secrets from Google Secret Manager, secrets-init should run under the IAM role that has permission to access desired secrets.
This can be achieved by assigning IAM Role to Kubernetes Pod with Workload Identity. It’s possible to assign IAM Role to GCE instance, where the container is running, but this option is less secure.
(15–03–2020) Update: Kubernetes Admission Webhook
The kube-secrets-init project implements a Kubernetes admission webhook that injects initContainer into any Pod with explicit (environment variable) or implicit (Kubernetes Secrets and ConfigMaps) references to cloud secrets.
Useful links
- Kubernetes GKE Workload Identity DoiT blog post
- doitintl/secrets-init GitHub repository
- kube-secrets-init Kubernetes admission webhook GitHub repository
Summary
I hope, you find this post useful. I look forward to your comments and any questions you have.
Want more stories? Check our blog, or follow Alexei on Twitter.
