Many of our customers who have tried to create a Kubernetes cluster encountered this error message without specific logs that may help them understand what went wrong. This is how we usually solve it.
Often when using `eksctl` for creating Kubernetes clusters using AWS EKS, the process may get stuck waiting for the nodes to join the cluster the the following error appears:
> nodegroup “my-cluster” has 0 node(s) > waiting for at least 3 node(s) to become ready in “my-cluster” > timed out (after 25m0s) waiting for at least 3 nodes to join the cluster and become ready in “my-cluster”
The message is displayed, without any additional logs, and the process eventually times out, while the newly created nodes are unable to communicate with the Kubernetes cluster. From our experience in DoiT International this issue has two leading roots: misconfigured VPC or missing policies.
Misconfigured VPC
When using an existing VPC network, the user has to make sure that the VPC conforms with all EKS-specific requirements [1, 2]. The blog post by logz.io provides detailed guidance on setting up a VPC network, as well as an example AWS Cloud Formation template that you can use as the basis [3].
Missing IAM Policies
Another leading cause is related to AmazonEKSWorkerNodePolicy and AmazonEKS_CNI_Policy policies [4] that are required by the EKS worker nodes to be able to communicate with the cluster. By default, eksctl automatically generates a role containing these policies. However, when you use “attachPolicyARNs” property to attach specific policies by ARN, you have to include these policies explicitly [5]:
nodeGroups:
- name: my-special-nodegroup
iam:
attachPolicyARNs:
- arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
- arn:aws:iam::aws:policy/ElasticLoadBalancingFullAccess
Still getting these errors? Post your situation under this content.
[1] https://docs.aws.amazon.com/eks/latest/userguide/create-public-private-vpc.html
[2] https://eksctl.io/usage/vpc-networking/
[3] https://logz.io/blog/amazon-eks/
[4] https://docs.aws.amazon.com/eks/latest/userguide/worker_node_IAM_role.html
[5] https://eksctl.io/usage/iam-policies/
