AWS recently announced Karpenter – An Open-Source High-Performance Kubernetes Cluster Autoscaler - ReInvent-2021

Before getting into the discussion of Karpenter, let's discuss k8s native cluster auto-scaler.

According to the documentation below is the definition:

Cluster Autoscaler is a tool that automatically adjusts the size of the Kubernetes cluster when one of the following conditions is true:

• there are pods that failed to run in the cluster due to insufficient resources.
• there are nodes in the cluster that have been underutilized for an extended period of time and their pods can be placed on other existing nodes.

And if we prefer to adjust the CA for various use cases we have to create each group for them, as explained below.

Traditional way:

Creating autoscaling groups different types of node groups based on our needs.

Example:

ASG node group for GPU use ASG node group for general use and based on instance family etc..

This creates more overhead in maintenance and operation costs. :(

Why do we need a cloud-native cluster auto-scaler?

Cloud-native CA (cluster-autoscaler) explores the full capability of the native tools and technologies which helps to effectively and efficiently use the resources in all aspects.

This creates a debate about Kubernetes native vs AWS native.

Karpenter:

Karpenter solves the following problem making effective decisions and scheduling.

What does the pod need? Where is the pod best fit-in? What can I do to best fit that pod in a node?

it provides complete control over ec2 instances

cat <<EOF |kubectl apply -f -apiVersion: karpenter.sh/v1alpha5
 kind: Provisioner
 metadata:
 name: default
 spec:
 #Requirements that constrain the parameters of provisioned nodes. 
 #Operators { In, NotIn } are supported to enable including or excluding values
   requirements:
     - key: node.k8s.aws/instance-type #If not included, all instance types are considered
       operator: In
       values: ["m5.large", "m5.2xlarge"]
     - key: "topology.kubernetes.io/zone" #If not included, all zones are considered
       operator: In
       values: ["us-east-1a", "us-east-1b"]
     - key: "kubernetes.io/arch" #If not included, all architectures are considered
       values: ["arm64", "amd64"]
     - key: " karpenter.sh/capacity-type" #If not included, the webhook for the AWS cloud provider will default to on-demand
       operator: In
       values: ["spot", "on-demand"]
   provider:
     instanceProfile: KarpenterNodeInstanceProfile-eks-karpenter-demo
   ttlSecondsAfterEmpty: 30  
 EOF

From the above provisioner config file, we can see we can control instance type, Availability zone, architecture and capacity type (on-demand vs Spot)

Highlights:

• Faster Scheduling - Karpenter directly manages the nodes.

• No Node group provisioning - Karpenter takes care of instance provisioning with node groups, it can schedule pods in instances based on the configuration.

• Cost-effective than k8s native cluster auto-scaler.

Karpenter: Right now it only supports AWS native, anyone can contribute to other cloud tools.

Wonderfull self-explained video from JustinGarrison about Karpenter https://www.youtube.com/watch?v=3QsVRHVdOnM&ab_channel=JustinGarrison

--- End-of-Blog ---

Reference: github.com/kubernetes/autoscaler/blob/maste..