The GKE Workload Identity Analyzer ensures that Workload Identity is configured properly

If you’re using Google Kubernetes Engine (GKE) and your workloads are communicating to other services within Google Cloud, you’re probably using Workload Identity. If you’re not using it, you should be. Google recommends Workload Identity for workloads running on Google Kubernetes Engine (GKE) to access Google Cloud services in a secure and manageable way.

Alternatives to Workload Identity

Alternative (and somewhat legacy) options for authentication between GKE and Google Cloud include:

• using the Compute Engine default service account, which normally provides too broad of a scope and cannot be customized per workload
• supplying service account secrets to your workloads, which is a security risk because these secret keys are long-lived credentials that require care and occasional rotation to align with security best practices

How Workload Identity works

Workload Identity comes with none of the above concerns as it is configured on a per-workload basis and requires no management nor injection of secrets to your workload.

Essentially, it works by creating a relationship between a Kubernetes Service Account (KSA) and a Google Service Account (GSA). The KSA is used to identify your workload running in GKE as the user of the GSA, and the GSA in turn is configured with the necessary access to Google Cloud resources. Access can be as broad or as narrow as your workload requires, e.g,  a workload that requires access to all PubSub topics in a project, compared to a workload that needs to access only a single one (or more).

The advantages of Workload Identity are obvious, yet using it requires some extra set up work, which can be somewhat confusing at first.

Misconfiguration can be present in many different links in the chain of authentication between a Pod running in GKE and a service running in a GCP project. That can make troubleshooting quite tedious.

What the GKE Workload Identity Analyzer does

DoiT has developed the GKE Workload Identity Analyzer for this purpose. The Workload Identity (WI) Analyzer command line tool takes a pod name as an argument and performs configuration checks to make sure that WI is set up correctly for your workload. It checks the following:

• Workload Identity is enabled on the GKE cluster
• Pod has .spec.serviceAccountName configured
• KSA (configured in previous step) exists
• KSA is annotated correctly with a GSA
• GSA (configured in previous step) exists in the project
• KSA has roles/iam.workloadIdentityUser on the GSA

Eventually, the tool will check and list any IAM roles the GSA has in the project (note that it does not check for IAM roles granted on a narrower scope, such as specific GCP resource; PubSub Topic, CloudSQL instance, etc.).

It takes less than 10 seconds to run and can save a lot of time inspecting resources in the console or running commands in the terminal.

The tool is written in Python and published to PyPI. It can be easily installed with PIP:

$ pip install wi-analyzer

Running GKE Workload Identity Analyzer

The prerequisites to running it are:

• gcloud cli installed and configured
• Application Default Credentials generated using gcloud
• kubectl installed and configured with cluster access

Note that the tool determines some information from your current kubeconfig context. It also expects the default generated name for a GKE cluster when fetching credentials (in the format of: gke_<GCP_PROJECT_ID>_<LOCATION>_<CLUSTER_NAME>).

Alternatively, pass arguments to configure the GCP project id, cluster location and cluster name.

Once everything is set up, you can run the tool from the terminal:

$ wi-analyzer --help
usage: wi-analyzer [-h] [-n NAMESPACE] [-d] pod

GKE Workload Identity Analyzer

positional arguments:
  pod                   Kubernetes Pod name to check

options:
  -h, --help            show this help message and exit
  -n NAMESPACE, --namespace NAMESPACE
                        Kubernetes Namespace to run in
  -p PROJECT, --project PROJECT
                        GCP Project holding the cluster
  -l LOCATION, --location LOCATION
                        The GCP location of the cluster
  -c CLUSTER, --cluster CLUSTER
                        The name of the cluster

  -d, --debug           Enable debug logging

For example, let’s say I was following the documentation on configuring my workload to use WI and accidentally missed a trivial step. The output from the tool might help us with that:

$ wi-analyzer demo-pod
Check results
---------------------------
V=Passed, X=Failed, -=Skipped

[V] GCP project and GKE info determined from current context
[V] Namespace passed as argument,or determined from current context
[V] Workload Identity enabled on GKE Cluster
[V] Pod found in current context
[V] GKE Node found in the cluster
[V] Workload Identity enabled on Node Pool
[V] KSA found in the cluster
[X] KSA Workload Identity annotation set correctly
[-] GSA found in GCP project
[-] GSA is enabled
[-] GSA has Workload Identity users configured
[-] GSA does not have KSA as a Workload Identity user

GKE cluster info
---------------------------
Cluster: "projects/test-eyal/locations/europe-west1-b/clusters/playground"
Workload: "demo/demo-pod" running on Node: "gke-playground-nodepool-1f1ace09-96kr"
KSA name: "demo-ksa"

Google Service Account info
---------------------------
Google Service Account information could not be determined, fix previous issues

As the output suggests, the WI annotation was missing from the KSA. After fixing it, running the tool again should show us only successful checks and the full overview:

$ wi-analyzer demo-pod
Check results
---------------------------
V=Passed, X=Failed, -=Skipped

[V] GCP project and GKE info determined from current context
[V] Namespace passed as argument,or determined from current context
[V] Workload Identity enabled on GKE Cluster
[V] Pod found in current context
[V] GKE Node found in the cluster
[V] Workload Identity enabled on Node Pool
[V] KSA found in the cluster
[V] KSA Workload Identity annotation set correctly
[V] GSA found in GCP project
[V] GSA is enabled
[V] GSA has Workload Identity users configured
[V] GSA does not have KSA as a Workload Identity user

GKE cluster info
---------------------------
Cluster: "projects/test-eyal/locations/europe-west1-b/clusters/playground"
Workload: "demo/demo-pod" running on Node: "gke-playground-nodepool-1f1ace09-96kr"
KSA name: "demo-ksa"

Google Service Account info
---------------------------
Google Service Account: "projects/test-eyal/serviceAccounts/[email protected]"
Has the following Workload Identity Users:
serviceAccount:test-eyal.svc.id.goog[demo/demo-ksa]

GSA: "[email protected]" has the following roles in project "test-eyal":
roles/cloudsql.client
Workload Identity configured properly - check if any IAM roles are missing from the list above

The tool currently supports only GKE Standard, but support for Fleet Workload Identity (GKE WI for Anthos) is on the roadmap. Testing on multilcloud Anthos clusters is already underway but will require some fundamental changes to the tool. Stay tuned!

Found an issue with the tool? Do you have any additional feature ideas?

Don’t hesitate to reach out, either here in the comments, or by creating an issue on the GitHub repository.