Autoscaling

Objective

Deploy the necessary metrics and autoscaling components to automatically rightsize the cluster.

Background

In Kubernetes, there are three distinct flavors of autoscaling:

  • Horizontal Pod Autoscaling (HPA): A built-in controller that adjusts the number of pods for a Deployment or StatefulSet. This can be extended to incorporate event-driven autoscaling with tools like KEDA.

  • Vertical Pod Autoscaling (VPA): A specific controller that can be installed to automatically adjust the resource requests and limits of pods based on historical usage.

  • Cluster Autoscaling: A category of controllers that will adjust the number of nodes running in the cluster. The two most popular projects are the Cluster Autoscaler and Karpenter.

The Panfactum stack makes use of all three types of autoscaling. To learn more about autoscaling and resource optimization, see our concept docs.

Deploy Metrics Server

For autoscaling to work, the Kubernetes API server must provide realtime metrics about individual container CPU and memory usage. Interestingly, this is not built-in to Kubernetes by default but rather powered by an API extension provided by the metrics-server project.

We provide a module to deploy the server: kube_metrics_server.

Let’s deploy it now:

  1. Create a new directory adjacent to your kube_linkerd module called kube_metrics_server.

  2. Add a terragrunt.hcl to that directory that looks like this.

  3. Run pf-tf-init to enable the required providers.

  4. Run terragrunt apply.

Let’s test to ensure it is working as intended:

  1. Open k9s (or restart if it is already open).

  2. Notice that k9s is now reporting your total cluster resource utilization:

    Cluster capacity displayed in k9s
  3. Navigate to the pods view. Notice that all pods are now reporting CPU and memory metrics:

    Pod metrics displayed in k9s

    The N/A fields for the /R and /L columns indicate that many pods have not had their resource requests (R) or limits (L) set. When we install the VPA, these will automatically be set.

  4. To diagnose utilization issues in the cluster, we bundle a CLI utility called kube-capacity for consolidating granular metrics across the entire cluster. Run kube-capacity -uc now:

    Terminal window
    NODE NAMESPACE POD CONTAINER CPU REQUESTS CPU LIMITS CPU UTIL MEMORY REQUESTS MEMORY LIMITS MEMORY UTIL
    * * * * 1240m (21%) 3700m (63%) 514m (8%) 1740Mi (9%) 5334Mi (29%) 4596Mi (25%)
    ....
    ip-10-0-213-182.us-east-2.compute.internal * * * 360m (18%) 1200m (62%) 161m (8%) 530Mi (8%) 1238Mi (20%) 1420Mi (23%)
    ip-10-0-213-182.us-east-2.compute.internal cilium cilium-4d4gf * 100m (5%) 0m (0%) 25m (1%) 10Mi (0%) 0Mi (0%) 126Mi (2%)
    ip-10-0-213-182.us-east-2.compute.internal cilium cilium-4d4gf cilium-agent 0m (0%) 0m (0%) 25m (1%) 0Mi (0%) 0Mi (0%) 126Mi (2%)
    ip-10-0-213-182.us-east-2.compute.internal aws-ebs-csi-driver ebs-csi-node-ph26m * 30m (1%) 100m (5%) 2m (0%) 120Mi (1%) 768Mi (12%) 26Mi (0%)
    ip-10-0-213-182.us-east-2.compute.internal aws-ebs-csi-driver ebs-csi-node-ph26m ebs-plugin 10m (0%) 0m (0%) 1m (0%) 40Mi (0%) 256Mi (4%) 11Mi (0%)
    ip-10-0-213-182.us-east-2.compute.internal aws-ebs-csi-driver ebs-csi-node-ph26m linkerd-proxy 0m (0%) 0m (0%) 1m (0%) 0Mi (0%) 0Mi (0%) 4Mi (0%)
    ip-10-0-213-182.us-east-2.compute.internal aws-ebs-csi-driver ebs-csi-node-ph26m liveness-probe 10m (0%) 0m (0%) 1m (0%) 40Mi (0%) 256Mi (4%) 8Mi (0%)
    ip-10-0-213-182.us-east-2.compute.internal aws-ebs-csi-driver ebs-csi-node-ph26m node-driver-registrar 10m (0%) 0m (0%) 1m (0%) 40Mi (0%) 256Mi (4%) 4Mi (0%)
    ip-10-0-213-182.us-east-2.compute.internal cert-manager jetstack-cert-manager-7b467c7747-xhbrx * 10m (0%) 100m (5%) 2m (0%) 10Mi (0%) 10Mi (0%) 16Mi (0%)
    ip-10-0-213-182.us-east-2.compute.internal cert-manager jetstack-cert-manager-7b467c7747-xhbrx cert-manager-controller 0m (0%) 0m (0%) 1m (0%) 0Mi (0%) 0Mi (0%) 13Mi (0%)
    ip-10-0-213-182.us-east-2.compute.internal cert-manager jetstack-cert-manager-7b467c7747-xhbrx linkerd-proxy 0m (0%) 0m (0%) 1m (0%) 0Mi (0%) 0Mi (0%) 4Mi (0%)
    ip-10-0-213-182.us-east-2.compute.internal linkerd linkerd-destination-96c96755b-8thzg * 10m (0%) 100m (5%) 3m (0%) 10Mi (0%) 10Mi (0%) 45Mi (0%)
    ip-10-0-213-182.us-east-2.compute.internal linkerd linkerd-destination-96c96755b-8thzg destination 0m (0%) 0m (0%) 2m (0%) 0Mi (0%) 0Mi (0%) 22Mi (0%)
    ip-10-0-213-182.us-east-2.compute.internal linkerd linkerd-destination-96c96755b-8thzg linkerd-proxy 0m (0%) 0m (0%) 1m (0%) 0Mi (0%) 0Mi (0%) 8Mi (0%)
    ip-10-0-213-182.us-east-2.compute.internal linkerd linkerd-destination-96c96755b-8thzg policy 0m (0%) 0m (0%) 1m (0%) 0Mi (0%) 0Mi (0%) 6Mi (0%)
    ip-10-0-213-182.us-east-2.compute.internal linkerd linkerd-destination-96c96755b-8thzg sp-validator 0m (0%) 0m (0%) 1m (0%) 0Mi (0%) 0Mi (0%) 10Mi (0%)
    ip-10-0-213-182.us-east-2.compute.internal linkerd linkerd-identity-cc6dffdf-tt9jm * 10m (0%) 100m (5%) 1m (0%) 10Mi (0%) 10Mi (0%) 15Mi (0%)
    ip-10-0-213-182.us-east-2.compute.internal linkerd linkerd-identity-cc6dffdf-tt9jm identity 0m (0%) 0m (0%) 1m (0%) 0Mi (0%) 0Mi (0%) 11Mi (0%)
    ip-10-0-213-182.us-east-2.compute.internal linkerd linkerd-identity-cc6dffdf-tt9jm linkerd-proxy 0m (0%) 0m (0%) 1m (0%) 0Mi (0%) 0Mi (0%) 4Mi (0%)
    ip-10-0-213-182.us-east-2.compute.internal linkerd linkerd-proxy-injector-8497c6bd8-dwv6j * 10m (0%) 100m (5%) 1m (0%) 10Mi (0%) 10Mi (0%) 19Mi (0%)
    ip-10-0-213-182.us-east-2.compute.internal linkerd linkerd-proxy-injector-8497c6bd8-dwv6j linkerd-proxy 0m (0%) 0m (0%) 1m (0%) 0Mi (0%) 0Mi (0%) 4Mi (0%)
    ip-10-0-213-182.us-east-2.compute.internal linkerd linkerd-proxy-injector-8497c6bd8-dwv6j proxy-injector 0m (0%) 0m (0%) 1m (0%) 0Mi (0%) 0Mi (0%) 16Mi (0%)
    ip-10-0-213-182.us-east-2.compute.internal metrics-server metrics-server-6df8ffd998-65mkb * 100m (5%) 100m (5%) 4m (0%) 200Mi (3%) 10Mi (0%) 24Mi (0%)
    ip-10-0-213-182.us-east-2.compute.internal metrics-server metrics-server-6df8ffd998-65mkb linkerd-proxy 0m (0%) 0m (0%) 1m (0%) 0Mi (0%) 0Mi (0%) 4Mi (0%)
    ip-10-0-213-182.us-east-2.compute.internal metrics-server metrics-server-6df8ffd998-65mkb metrics-server 100m (5%) 0m (0%) 3m (0%) 200Mi (3%) 0Mi (0%) 20Mi (0%)
    ip-10-0-213-182.us-east-2.compute.internal secrets-csi secrets-csi-secrets-store-csi-driver-zch5w * 70m (3%) 400m (20%) 1m (0%) 140Mi (2%) 400Mi (6%) 26Mi (0%)
    ip-10-0-213-182.us-east-2.compute.internal secrets-csi secrets-csi-secrets-store-csi-driver-zch5w linkerd-proxy 0m (0%) 0m (0%) 1m (0%) 0Mi (0%) 0Mi (0%) 4Mi (0%)
    ip-10-0-213-182.us-east-2.compute.internal secrets-csi secrets-csi-secrets-store-csi-driver-zch5w liveness-probe 10m (0%) 100m (5%) 1m (0%) 20Mi (0%) 100Mi (1%) 8Mi (0%)
    ip-10-0-213-182.us-east-2.compute.internal secrets-csi secrets-csi-secrets-store-csi-driver-zch5w node-driver-registrar 10m (0%) 100m (5%) 1m (0%) 20Mi (0%) 100Mi (1%) 4Mi (0%)
    ip-10-0-213-182.us-east-2.compute.internal secrets-csi secrets-csi-secrets-store-csi-driver-zch5w secrets-store 50m (2%) 200m (10%) 1m (0%) 100Mi (1%) 200Mi (3%) 12Mi (0%)
    ip-10-0-213-182.us-east-2.compute.internal vault vault-1 * 10m (0%) 100m (5%) 41m (2%) 10Mi (0%) 10Mi (0%) 52Mi (0%)
    ip-10-0-213-182.us-east-2.compute.internal vault vault-1 linkerd-proxy 0m (0%) 0m (0%) 4m (0%) 0Mi (0%) 0Mi (0%) 4Mi (0%)
    ip-10-0-213-182.us-east-2.compute.internal vault vault-1 vault 0m (0%) 0m (0%) 38m (1%) 0Mi (0%) 0Mi (0%) 49Mi (0%)
    ip-10-0-213-182.us-east-2.compute.internal vault vault-csi-provider-4cndp * 10m (0%) 100m (5%) 3m (0%) 10Mi (0%) 10Mi (0%) 34Mi (0%)
    ip-10-0-213-182.us-east-2.compute.internal vault vault-csi-provider-4cndp linkerd-proxy 0m (0%) 0m (0%) 1m (0%) 0Mi (0%) 0Mi (0%) 4Mi (0%)
    ip-10-0-213-182.us-east-2.compute.internal vault vault-csi-provider-4cndp vault-agent 0m (0%) 0m (0%) 2m (0%) 0Mi (0%) 0Mi (0%) 25Mi (0%)
    ip-10-0-213-182.us-east-2.compute.internal vault vault-csi-provider-4cndp vault-csi-provider 0m (0%) 0m (0%) 1m (0%) 0Mi (0%) 0Mi (0%) 6Mi (0%)

Deploy the Vertical Pod Autoscaler

Now that we are capturing resource utilization data, we can use the vertical pod autoscaler to automatically rightsize your pod’s CPU and memory resource requests and limits based on historical usage.

We provide a module to deploy the VPA: kube_vpa.

Let’s deploy it now:

  1. Create a new directory adjacent to your kube_metrics_server module called kube_vpa.

  2. Add a terragrunt.hcl to that directory that looks like this.

  3. Run pf-tf-init to enable the required providers.

  4. Run terragrunt apply.

  5. In your region.yaml, set extra_inputs.vpa_enabled to true like this. This will turn on the VPA for all modules in this cluster in the next apply.

  6. Navigate up to the directory containing all your modules (this should be the parent of kube_vpa). Run terragrunt run-all apply to re-apply all the modules at once. Note that this might cause your Vault proxy to disconnect as pods are restarted and thus result in an incomplete apply. Simply reconnect the proxy and try the command again.

  7. After all these modules are updated, you should now see many VPA resources in k9s (:vpa). After a minute or two, they should begin to provide resource estimates:

    VPA resources in k9s
  8. Return to the pod view. Notice that all request columns are populated as well as the memory limit column:

    Resource requests and limits are set
    • We set requests for both CPU and memory so that the Kubernetes scheduler can accurately decide which nodes have enough resources available for pods during the pod-to-node assignment phase.

    • We set memory limits so that memory leaks in any application will not consume unbounded memory on the node and thus cause all other pods on the node to crash with OOM errors.

    • We do not set cpu limits because if a node’s CPU is constrained, CPU will automatically be shared across all pods proportional to each pod’s CPU request. 1

Deploy Karpenter

Historically, cluster autoscaling has almost always been done by the cluster-autoscaler addon. However, this controller had several limitations and required lots of configuration to provide the optimal setup for your organization.

Fortunately, a new tool has emerged, Karpenter, which provides both more flexibility, better performance, and better cost optimization. Specifically:

  • You do not need to choose instance types in advance as it will query your cloud providers API for the entire list of available instance types and prices and then spin up the instance types that are the best fit for your workloads at the lowest cost

  • If you allow it, it will prioritize utilizing spot instances for around a 70% discount over list price. Moreover, it will periodically check if existing workloads can be migrated to cheaper instances as prices change over time.

  • It incorporates the functionality provided by the aws-node-termination-handler removing the need to run it altogether.

  • It scales extremely well with the ability to spin up and down 100s of nodes at a time.

We provide an infrastructure module to deploy it: kube_karpenter.

Let’s deploy it now:

  1. Create a new directory adjacent to your kube_vpa module called kube_karpenter.

  2. Add a terragrunt.hcl to that directory that looks like this.

  3. Adjust the wait input to false. This will allow the installation to proceed even though there will be temporary errors as Karpenter sets up the required resources.

  4. Run pf-tf-init to enable the required providers.

  5. Run terragrunt apply.

  6. Update wait to true.

Deploy NodePools

Karpenter requires some instructions about how to perform autoscaling such as what AMI to use for the underlying nodes. It looks for those instructions in NodePool custom resources.

The Panfactum stack comes with several classes of NodePools which are defined in the kube_karpenter_node_pools module:

  • burstable
  • spot
    • general purpose spot nodes
    • highest scheduling preference
    • assigned the following taints:
      • { key = "spot", value = "true", effect = "NoSchedule" }
  • worker
    • standard on-demand instance types
    • base scheduling preference

All nodes are labeled with their class via panfactum.com/class. 2

Let’s deploy the NodePools now:

  1. Create a new directory adjacent to your kube_karpenter module called kube_karpenter_node_pools.

  2. Add a terragrunt.hcl to that directory that looks like this.

  3. For node_subnets, we strongly recommend using the same subnets you used in the aws_eks module for node_subnets unless you have a specific reason not to.

  4. Run pf-tf-init to enable the required providers.

  5. Run terragrunt apply.

Test Cluster Autoscaling

Let’s verify that autoscaling now works as expected.

All deployments in the Panfactum stack are configured to disallow multiple replicas of the same pod to run on the same node. 3 As a result, we can increase the number of replicas in one of our pods and ensure that Karpenter recognizes that additional nodes need to be schedule.

  1. In K9s, navigate to :deployments.

  2. Highlight the cert-manager-webhook deployment and press s to trigger the scale dialogue.

  3. Increase this to 10.

  4. Immediately, Karpenter should recognize that new nodes are required to run these pods and begin to provision them. After 30-60 seconds, the nodes will be registered with Kubernetes and you should see them under :nodes.

  5. Describe some of the new nodes by pressing d when highlighting one. Notice that many of these new nodes are spot instances.

    Labels indicating that Karpenter spun up spot instances
  6. Scale cert-manager-webhook back down to 2. Karpenter will automatically take care of the node cleanup and de-provisioning. This will happen slower than the scale-up as scaling down happens in batches to avoid cluster thrash or service disruption. However, this should complete within 15 minutes.

Adjust EKS Node Pools

Now that Karpenter is successfully running, we can disable bootstrap mode on the EKS cluster by setting bootstrap_mode_enabled to false in the aws_eks module. This will reduce costs by allowing Karpenter to autoscale most of the cluster’s capacity.

Make that change and re-apply the module.

Deploy Bin-Packing Kubernetes Scheduler

The Kubernetes Scheduler (kube-scheduler) is the control plane component that is in charge of assigning new Pods to Nodes. AWS provides a built-in one (default-scheduler) via EKS, but it comes with a few limitations. 4 Most importantly, this scheduler cannot be configured and does not allow for resource bin-packing.

Instead, the default scheduler will try to spread pods evenly across nodes rather than consolidate them on as few nodes as possible. This hinders Karpenter’s ability to efficiently de-provision nodes, and can result in significant wasted resources. 5

We provide an alternative scheduler in kube_scheduler (panfactum) that implements the bin-packing scheduling algorithm. Typically, this results in a 25-33% cost reduction.

Let’s deploy it now:

  1. Create a new directory adjacent to your kube_karpenter module called kube_scheduler.

  2. Add a terragrunt.hcl to that directory that looks like this.

  3. Run pf-tf-init to enable the required providers.

  4. Run terragrunt apply.

  5. In your region.yaml, set extra_inputs.panfactum_scheduler_enabled to true like this. This will instruct all modules in this cluster to use this new scheduler instead of the default EKS one.

  6. Re-deploy all Kubernetes modules.

Next Steps

Now that autoscaling is active, we can proceed to set up inbound networking for our cluster.

Panfactum Bootstrapping Guide:
Step 15 of 21

Footnotes

  1. In general, it is best to avoid setting CPU limits in order to allow CPU to occasionally spike as needed. If the CPU utilization remains high, cluster autoscaling will kick in to provision net new nodes.

  2. The EKS-managed nodes created by aws_eks have a class of controller.

  3. To prevent service disruption if a node goes offline.

  4. The EKS scheduler does not run inside your cluster, but rather in the private AWS system where the Kubernetes API server is hosted.

  5. This article provides a great real world example of how this impacted a company running Kubernetes clusters in AWS.