When using Amazon Elastic Kubernetes Service (EKS) have you ever come across an error message stating, “Your current user or role does not have access to Kubernetes objects on this EKS cluster”? This error can be quite disconcerting, especially when you’re a global super admin being told you don’t have rights. In this article, we’ll break down what this error means and provide a step-by-step guide on how to resolve it.

Example sourced from the AWS console UI

Example from kubectl CLI against an EKS cluster

Understanding the Problem

In order to view Kubernetes resources on the AWS Management Console, the AWS IAM user or role must map to the aws-auth  ConfigMapin the EKS cluster. This mapping is essential because when an individual creates an EKS cluster, their IAM user or role is automatically granted system:masterspermissions in the cluster's RBAC (Role-Based Access Control) configuration. This lets them view Kubernetes resources through the EKS console and also edit the aws-auth  ConfigMap within Kubernetes, granting additional AWS users or roles the ability to interact with the cluster.

The AWS IAM user/role that creates the cluster has exclusive access to both the AWS Console and kubectl tool by default. Contrary to expectations, the reason behind not being able to view items in the AWS Console is the absence of proper mapping configuration between Kubernetes RBAC rights and IAM in the aws-auth  ConfigMap.

How to Resolve the Problem

Identifying the Cluster Creator

You need to identify who the EKS cluster creator is. One way is by looking at AWS CloudTrail logs. AWS CloudTrail records all AWS API calls, including those made by the AWS Management Console, SDKs, and CLI. Therefore, if you have CloudTrail enabled in your account, you should be able to find the identity of the user who created the EKS cluster.

Here’s a step-by-step guide on how you can do this:

1. Log into AWS Management Console
2. Navigate to CloudTrail from the AWS Management Console, you can find CloudTrail by typing ‘CloudTrail’ into the service search bar and selecting it from the dropdown.
3. In the CloudTrail dashboard, select ‘Event history’ from the sidebar.
4. On the Event history page, you will see filters at the top. Set the ‘Event name’ filter to ‘CreateCluster’, which is the API call that creates a new EKS cluster.
5. If you know the name of the EKS cluster, you can also filter by ‘Resource name’ and provide the cluster name.
6. After applying the filters, you should see a list of ‘CreateCluster’ events. Click on the event to view the details. The ‘User name’ field in the event details is the identity of the user who created the EKS cluster.

Please note that this approach will only be effective if CloudTrail was activated in the AWS account during the creation of the EKS cluster, which is generally enabled by default. However, if CloudTrail was not activated at that time, it will not be possible to determine the creator of the EKS cluster. In such a scenario, it would be advisable to seek assistance from an AWS administrator, DevOps, SRE, or an expert in this field within your organization who can guide you towards locating the cluster creator.

Identifying the IAM Identity ARN

Identify the IAM user or role that you’re using to access the console. You can get the IAM identity ARN by running the following AWS CLI command:

aws sts get-caller-identity --query Arn

Adding an IAM user or role to aws-auth ConfigMap

To grant access to your EKS cluster, your cluster creator can add an IAM user or role to the  aws-auth  ConfigMap . This enables the user or role to interact with the cluster seamlessly. The following are examples of how this can be done using either  kubectl or  eksctl tool:

kubectl

1. Retrieve the current ConfigMap:

kubectl get -n kube-system configmap/aws-auth -o yaml

This command will output the current  aws-auth ConfigMap. It should look something like this:

apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system
data:
  mapRoles: |
    - rolearn: arn:aws:iam::11122223333:role/EKS-Worker-NodeInstanceRole-1I00GBC9U4U7B
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes

2, To add a new IAM user or role, modify this YAML to include the new user or role under  mapRoles or  mapUsers . Here's an example of what this might look like:

apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system
data:
  mapRoles: |
    - rolearn: arn:aws:iam::11122223333:role/EKS-Worker-NodeInstanceRole-1I00GBC9U4U7B
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes
    - rolearn: arn:aws:iam::11122223333:role/YourNewRole
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:masters
  mapUsers: |
    - userarn: arn:aws:iam::11122223333:user/YourNewUser
      username: YourNewUser
      groups:
        - system:masters

In this example, replace  arn:aws:iam::11122223333:role/YourNewRole with the ARN of your new IAM role, and  arn:aws:iam::11122223333:user/YourNewUser with the ARN of your new IAM user. Also, replace  YourNewUser with the username of the new IAM user.

3. Once the modifications have been made, apply the new ConfigMap with the following command:

kubectl apply -f aws-auth-configmap.yaml

In the above command, replace  aws-auth-configmap.yaml with the filename of your modified ConfigMap.

eksctl

To add an IAM User:

eksctl create iamidentitymapping --region region-code --cluster cluster-name --arn arn:aws:iam::11122223333:user/YourNewUser --group system:masters

To add an IAM Role:

eksctl create iamidentitymapping --region region-code --cluster cluster-name --arn arn:aws:iam::11122223333:role/YourNewRole --group system:masters

In the above examples:

• Replace  region-code with the AWS region code (e.g.,  us-west-2 ).
• Replace  cluster-name with the name of the EKS cluster.
• Replace  arn:aws:iam::11122223333:role/YourNewRole with the ARN of the new IAM role, and  arn:aws:iam::11122223333:user/YourNewUser with the ARN of the new IAM user.
• Replace  system:masters with the appropriate Kubernetes group. Refer to the Kubernetes Groups section below for further details.

Kubernetes Groups

In Kubernetes, a group is a set of users that have the same permissions. The following are some of the default groups:

•  system:authenticated:  This includes all authenticated users.
•  system:unauthenticated:  This includes all unauthenticated users.
•  system:masters:  This group has full access to the cluster and can perform any action within the cluster. It's typically associated with cluster creator roles.
•  system:serviceaccounts:<namespace>:  This includes all service accounts in the  <namespace> namespace.

The  system:masters group gives full access to the cluster, which includes the ability to create, read, update, and delete any resource in the cluster. Because of this, you should be careful when adding users or roles to this group. Only trusted users or roles should be added to this group, and always apply the principle of least privilege. This concept is the idea of granting a user or role only the necessary levels of access or permissions to carry out tasks. The principle is used to decrease the risk of malicious activity or breaches by reducing the attack surface.

Ongoing Access Problems

If you’re still experiencing issues after the  aws-auth ConfigMap has been updated, it could be due to the IAM policy not being updated or the Kubernetes RBAC not recognizing the changes. As mentioned earlier, the IAM permissions are separate from Kubernetes permissions, and both need to be set correctly for the user to be able to interact with the Kubernetes API.

Here’s a list of troubleshooting steps to follow:

1. Make sure that the aws-auth ConfigMap in the kube-system namespace has been updated correctly:

kubectl -n kube-system describe configmap aws-auth

The output should show the IAM user or role under  mapRoles or  mapUsers . If it's not there, it means that the  ConfigMap has not been updated correctly or is possibly malformed. If your  ConfigMap in Kubernetes is malformed, it’s as though it’s been deleted, potentially locking out everyone except the cluster creator. Therefore, it’s important to be cautious and perform yaml linting by using tools such as YAML Lint in order to check your yaml configuration.

2. Verify the IAM user or role to ensure that it’s the one that has been added to the  aws-auth  ConfigMap . Mistyping the IAM user or role ARN is a common issue.

3. Ensure the IAM user or role has the necessary AWS permissions. This can be done by checking the IAM policies attached to the user or role. If you need to manage the EKS cluster and nodes, then the  AmazonEKSClusterPolicy and  AmazonEKSWorkerNodePolicy policies will grant the necessary permissions. Please refer to the guide AWS managed policies for Amazon Elastic Kubernetes Service for further details.

You can also create a custom IAM policy that includes read-only permissions for the  eks:Describe* and  eks:List* actions, which allows the user or role to view information about EKS resources.

4. Make sure that the Kubernetes user and groups mapped to the IAM entity have the necessary permissions in Kubernetes. This can be done by referencing the Kubernetes RBAC roles and bindings user guide.

Remember to always follow the principle of least privilege when granting permissions, both in AWS and in Kubernetes. Only grant the permissions that are necessary for the user or role to perform their tasks.

Also, it’s important to highlight that if the  aws-auth ConfigMap is removed, contains syntax errors or is incorrectly formatted, all users except the cluster creator will lose access. The cluster creator is the sole individual with the ability to rectify this situation.

Conclusion

Navigating the complex landscape of AWS EKS errors can undoubtedly be challenging, but with the right combination of knowledge, tools, and practical insight, these obstacles can be overcome. Understanding the intricate interplay between IAM and Kubernetes identities is a critical piece of the puzzle, as is mastering the nuances of handling the  aws-auth ConfigMap . I hope this guide will serve as a useful resource for you, whether you’re a seasoned EKS veteran or a newcomer to the platform.