It is not a rare case when an application running on Google Kubernetes Engine (GKE) needs to access Amazon Web Services (AWS) APIs. Any application has needs. Maybe it needs to run an analytics query on Amazon Redshift, access data stored in Amazon S3 bucket, convert text to speech with Amazon Polly or use any other AWS service. This multi-cloud scenario is common nowadays, as companies are working with multiple cloud providers.

Cross-cloud access introduces a new challenge; how to manage cloud credentials, required to access from one cloud provider to services running in the other. The naive approach, distributing and saving cloud provider secrets is not the most secure approach; distributing long-term credentials to each service, that needs to access AWS services, is challenging to manage and a potential security risk.

Current Solutions

Each cloud provides it’s own unique solution to overcome this challenge, and if you are working with a single cloud provider, it’s more than enough.

Google Cloud announced a Workload Identity, the recommended way for GKE applications to authenticate to and consume other Google Cloud services. Workload Identity works by binding Kubernetes service accounts and Cloud IAM service accounts, so you can use Kubernetes-native concepts to define which workloads run as which identities, and permit your workloads to automatically access other Google Cloud services, all without having to manage Kubernetes secrets or IAM service account keys! Read DoiT Kubernetes GKE Workload Identity blog post.

Amazon Web Services supports a similar functionality with IAM Roles for Service Accounts feature. With IAM roles for service accounts on Amazon EKS clusters, you can associate an IAM role with a Kubernetes service account. This service account can then provide AWS permissions to the containers in any pod that uses that service account. With this feature, you no longer need to provide extended permissions to the worker node IAM role so that pods on that node can call AWS APIs.

But what if you are running your application workload on GKE cluster and would like to access AWS services without compromising on security?

Use Case Definition

Let’s assume that you already have an AWS account, and a GKE cluster and your company has decided to run a microservice-based application on GKE cluster, but still wants to use resources in the AWS account (Amazon S3 and SNS services) to integrate with other systems deployed on AWS.

For example, the orchestration job (deployed as a Kubernetes Job) is running inside a GKE cluster and needs to upload a data file into a S3 bucket and send a message to an Amazon SNS topic. The equivalent command-line might be:

Pretty simple example. In order for these commands to succeed, the orchestration job must have AWS credentials available to it, and those credentials must be able to make the relevant API calls.

The Naive (and non-secure) Approach: IAM long-term credentials

Export AWS Access Key and Secret Key for some AWS IAM User, and inject AWS credentials into the orchestration job, either as a credentials file or environment variables. Probably not doing this directly, but using Kubernetes Secrets resource protected with RBAC authorization policy.

The risk here is that these credentials never expire. They have to be transferred somehow from the AWS environment to the GCP environment, and in most cases, people want them to be stored somewhere so that they can be used to re-create the orchestration job later if required.

When using long-term AWS credentials, there are multiple ways that your AWS account can be compromised; unintentionally committing AWS credentials into a GitHub repository, keeping them in a Wiki system, reusing credentials for different services and applications, allowing non-restricted access and, so on.

While it’s possible to design a proper credentials management solution for issued IAM User credentials, it won’t be required if you will never create these long-term credentials in the first place.

The Proposed Approach

The basic idea is to assign AWS IAM Role to GKE Pod, similarly to Workload Identity and EKS IAM Roles for Service Accounts cloud-specific features.

Luckily for us, AWS allows to create an IAM role for OpenID Connect Federation OIDC identity providers instead of IAM users. On the other hand, Google implements OIDC provider and integrates it tightly with GKE through Workload Identity feature. Providing a valid OIDC token to GKE pod, running under Kubernetes Service Account linked to a Google Cloud Service Account. All these may come in handy to implement GKE-to-AWS secure access.

Exchanging OIDC access token to ID token

There is one thing missing, required to complete the puzzle. With properly setup Workflow Identity GKE Pod gets an OIDC access token that allows access to Google Cloud services. In order to get temporary AWS credentials from AWS Security Token Service (STS), you need to provide a valid OIDC ID token.

AWS SDK (and aws-cli tool) will automatically request temporary AWS credentials from STS service, when the following environment variables are properly setup:

• AWS_WEB_IDENTITY_TOKEN_FILE - the path to the web identity token file (OIDC ID token)
• AWS_ROLE_ARN - the ARN of the role to assume by Pod containers
• AWS_ROLE_SESSION_NAME - the name applied to this assume-role session

This may sound a bit complex, but I will provide a step-by-step guide and supporting open source project dointl/gtoken to simplify the setup.

gtoken-webhook Kubernetes Mutating Admission webhook

The gtoken-webhook is a Kubernetes mutating admission webhook, that mutates any K8s Pod running under specially annotated Kubernetes Service Account (see details below).

gtoken-webhook mutation flow

The gtoken-webhook injects a gtoken initContainer into a target Pod and an additional gtoken sidekick container (to refresh an OIDC ID token a moment before expiration), mounts token volume and injects three AWS-specific environment variables. The gtoken container generates a valid GCP OIDC ID Token and writes it to the token volume. It also injects required AWS environment variables.

The AWS SDK will automatically make the corresponding AssumeRoleWithWebIdentity calls to AWS STS on your behalf. It will handle in-memory caching as well as refreshing credentials as needed.

The Configuration Flow Guide

Deploy gtoken-webhook

1. To deploy the gtoken-webhook server, we need to create a webhook service and a deployment in our Kubernetes cluster. It’s pretty straightforward except one thing, which is the server’s TLS configuration. If you’d care to examine the deployment.yaml file, you’ll find that the certificate and corresponding private key files are read from command line arguments and that the path to these files comes from a volume mount that points to a Kubernetes secret:

The most important thing to remember is to set the corresponding CA certificate later in the webhook configuration, so the apiserver will know that it should be accepted. For now, we’ll reuse the script originally written by the Istio team to generate a certificate signing request. Then we’ll send the request to the Kubernetes API, fetch the certificate, and create the required secret from the result.

First, run the webhook-create-signed-cert.sh script and check if the secret holding the certificate and key has been created:

Once the secret is created, we can create a deployment and service. These are standard Kubernetes deployment and service resources. Up until this point we’ve produced nothing but an HTTP server that’s accepting requests through service on port 443:

Configure Mutating Admission webhook

Now that our webhook server is running, it can accept requests from the apiserver. However, we should create some configuration resources in Kubernetes first. Let’s start with our validating webhook, then we’ll configure the mutating webhook later. If you take a look at the webhook configuration, you’ll notice that it contains a placeholder for CA_BUNDLE:

There is a small script that substitutes the CA_BUNDLE placeholder in the configuration with this CA. Run this command before creating the validating webhook configuration:

Create a mutating webhook configuration:

Configure RBAC for gtoken-webhook

Create Kubernetes Service Account to be used with gtoken-webhook:

Define RBAC permission for webhook service account:

Flow Variables

Some of the following variables should be provided by the user, others will be automatically generated and reused in the following steps.

• PROJECT_ID - GCP project ID (provided by the user)
• CLUSTER_NAME - GKE cluster name (provided by the user)
• GSA_NAME - Google Cloud Service Account name (provided by the user)
• GSA_ID - Google Cloud Service Account unique ID (generated by Google)
• KSA_NAME - Kubernetes Service Account name (provided by the user)
• KSA_NAMESPACE - Kubernetes namespace (provided by the user)
• AWS_ROLE_NAME - AWS IAM role name (provided by the user)
• AWS_POLICY_NAME - an AWS IAM policy to assign to IAM role (provided by the user)
• AWS_ROLE_ARN - AWS IAM Role ARN identifier (generated by AWS)

Google Cloud: Enable GKE Workload Identity

Create a new GKE cluster with Workload Identity enabled:

or update an existing cluster:

Google Cloud: Create a Google Cloud Service Account

Create a Google Cloud Service Account:

Update GSA_NAME Google Service Account with following roles:

• roles/iam.workloadIdentityUser - impersonate service accounts from GKE Workloads
• roles/iam.serviceAccountTokenCreator - impersonate service accounts to create OAuth2 access tokens, sign blobs, or sign JWT tokens

AWS: Create AWS IAM Role with Google OIDC Federation

Prepare a role trust policy document for Google OIDC provider:

Create AWS IAM Role with Google Web Identity:

Assign AWS Role desired policies:

Get AWS Role ARN to be used in K8s SA annotation:

GKE: Create a Kubernetes Service Account

Create K8s namespace:

Create K8s Service Account:

Annotate K8s Service Account with GKE Workload Identity (GCP Service Account email):

Annotate K8s Service Account with AWS Role ARN:

Run Demo

Run a new K8s Pod with K8s ${KSA_NAME}` Service Account:

External References

• GitHub: Securely access AWS services from GKE cluster with doitintl/gtoken
• AWS Docs: Creating a Role for Web Identity or OpenID Connect Federation
• Blog: Kubernetes GKE Workload Identity link
• AWS Blog: Introducing fine-grained IAM roles for service accounts link
• GitHub: AWS Auth using Web Identity Federation from Google Cloud shrikant0013/gcp-aws-webidentityfederation GitHub project
• Blog: Using GCP Service Accounts to access AWS IAM Roles blog post by Colin Panisset

---

Summary

I hope, you find this post useful. I look forward to your comments and any questions you have.

Want more stories? Check our blog, or follow Alexei on Twitter.