Skip to content

Karpenter Blueprint: Working with Graviton Instances

Purpose

You might be wondering how to use Graviton instances with Karpenter. Well, first you need to make sure that your application can run on different CPUs such as arm64 or x86-64. The programming language you’re using and its ecosystem needs to be multi-arch aware, as you'll need to container images for both arm64 and x86-64 architectures. AWS Graviton processors are custom built by AWS using 64-bit Arm Neoverse. They power Amazon EC2 instances such as: M6g, M6gd, T4g, C6g, C6gd, C6gn, R6g, R6gd, X2gd, and more. Graviton instances provide up to 40% better price performance over comparable current generation x86-based instances for a wide variety of workloads.

Karpenter set the default architecture constraint on your NodePool that supports most common user workloads, which today will be amd64 (or x86-64 architecture). However, if you're flexible to support either arm64 or x86-64, when working with AWS, you defer the decision of which architecture to use depending on purchase model: On-Demand or Spot.

If it’s an On-Demand Instance, Karpenter uses the lowest-price (LP) allocation strategy to launch the cheapest instance type that has available capacity. If it’s a Spot Instance, Karpenter uses the price-capacity-optimized (PCO) allocation strategy. PCO looks at both price and capacity availability to launch from the Spot Instance pools that are the least likely to be interrupted and have the lowest possible price.

Requirements

  • A Kubernetes cluster with Karpenter installed. You can use the blueprint we've used to test this pattern at the cluster folder in the root of this repository.
  • A default Karpenter NodePool as that's the one we'll use in this blueprint. You did this already in the "Deploy a Karpenter Default EC2NodeClass and NodePool" section from this repository.
  • A container image built for arm64 architecture hosted in a container image registry such as ECR.

NOTE: To build a multi-arch container image, you can use Docker‘s buildx or, equally possible, a remote build. In this context, you want to check the multi-arch readiness of your automated build and test pipeline, for example, [support in Travis. Next, you need to push your container images to a registry such as ECR.

NOTE: The sample workload in this repository already supports arm64.

Deploy

You're going to use the default NodePool as there's no need to create a separate NodePool to launch Graviton instances.

Results

You can inspect the pods from the workload-flexible deployment, but they don't have something in particular for Graviton instances other than asking for On-Demand capacity (karpenter.sh/capacity-type: on-demand) as a node selector. So, let's deploy the following assets:

kubectl apply -f workload-flexible.yaml

Wait for about one minute, and you'll see a new Graviton instance coming up:

$> kubectl get nodeclaims
NAME            TYPE         ZONE         NODE                                        READY   AGE
default-sgmkw   c6g.xlarge   eu-west-1b   ip-10-0-66-182.eu-west-1.compute.internal   True    42s

NOTE: All pods should be running now, and you didn't have to say anything special to Karpenter about which container image to use. Why? In Kubernetes, and by extension in Amazon EKS, the worker node-local supervisor called kubelet instructs the container runtime via a standardized interface to pull container images from a registry such as Amazon ECR and launch them, accordingly. All of which is multi-arch enabled and automated.

Now, let's suppose that you've make the decision to go all-in with Graviton. Instead of creating a new NodePool, you can control that behavior within the Deployment by using a nodeSelector of kubernetes.io/arch: arm64 and without limiting to On-Demand only. This means that now chances are that Karpenter will launch a Spot instance as it's the one with a better price offering. Let's see, deploy the other workload:

kubectl apply -f workload-graviton.yaml

Wait for about one minute, and run the following command to see which nodes Karpenter has launched and see if it's On-Demand or Spot:

kubectl get nodes -L karpenter.sh/capacity-type,beta.kubernetes.io/instance-type,karpenter.sh/nodepool,topology.kubernetes.io/zone -l karpenter.sh/initialized=true

You should see something similar to this:

NAME                                        STATUS   ROLES    AGE    VERSION               CAPACITY-TYPE
ip-10-0-87-181.eu-west-2.compute.internal   Ready    <none>   114s   v1.34.1-eks-473151a   on-demand       c6g.xlarge      default    eu-west-2b

Notice that now Karpenter decided to launch a c6g.2xlarge Spot instance because the workload and the NodePool support both pricing models, and the one that has a better price at this moment was a Graviton Spot instance.

EKS Auto Mode **Prerequisite:** an EKS cluster with Auto Mode enabled, and an EKS Access Entry granting `AmazonEKSAutoNodePolicy` to the node IAM role used by Auto Mode. > If you're using the Terraform template under [`cluster/automode/`](https://github.com/aws-samples/karpenter-blueprints/tree/main/cluster/automode) in this repo, the cluster, node IAM role, and Access Entry are all created for you — you can skip the manual access entry steps below. This blueprint uses the default NodePool and requires no custom NodePool or EC2NodeClass manifests. The workload runs as-is on an EKS Auto Mode cluster with the default `NodeClass` (`eks.amazonaws.com/v1`). 
kubectl apply -f .
If you are **not** using the `cluster/automode/` Terraform template, configure the Access Entry manually: 
aws eks create-access-entry \
  --cluster-name $CLUSTER_NAME \
  --principal-arn <node-role-arn> \
  --type EC2

aws eks associate-access-policy \
  --cluster-name $CLUSTER_NAME \
  --principal-arn <node-role-arn> \
  --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSAutoNodePolicy \
  --access-scope type=cluster

Cleanup

kubectl delete -f .