Skip to main content
Source

This page is generated from skills/eks-operation-review/references/autoscaling.md. Edit the source, not this page.

Vendored skill

This skill is sourced from eks-operation-review, also maintained by the APEX team.

Autoscaling

Purpose​

Assess cluster-level autoscaling (nodes) and workload-level autoscaling (pods), plus AZ resilience.

Checks to Execute​

7.1 — Cluster Autoscaling Strategy​

What to check:

  • Karpenter NodePools and EC2NodeClasses
  • Cluster Autoscaler deployment in kube-system
  • EKS Auto Mode (cluster computeConfig)
  • Node group scaling config (min/max/desired)
  • Spot vs On-Demand nodes
  • Currently pending pods

How to check:

  1. List resources nodepools.karpenter.sh → check limits and consolidation policy. If 404/NotFound (CRD not installed) → Karpenter is not deployed, proceed to check for Cluster Autoscaler or Auto Mode. If 403/Forbidden → mark Karpenter status UNKNOWN. Note: EKS Auto Mode is built on Karpenter and also serves the karpenter.sh CRDs (including nodepools.karpenter.sh), so CRD presence alone does not prove open-source/self-managed Karpenter is deployed — it only means the karpenter.sh API is served. Distinguish Auto Mode (via computeConfig.enabled == true in step 3, or the Auto-Mode node compute-type label eks.amazonaws.com/compute-type) from self-managed Karpenter before scoring.
  2. List Deployments in kube-system → check for cluster-autoscaler
  3. Describe cluster → computeConfig for Auto Mode
  4. List node groups → describe each for scalingConfig and capacityType
  5. List nodes → check labels for capacity type (karpenter.sh/capacity-type or eks.amazonaws.com/capacityType)
  6. List pods in Pending state (field selector status.phase=Pending); for each, read its age (.metadata.creationTimestamp) and scheduling events (look for FailedScheduling with Insufficient cpu/memory or no nodes available) to separate capacity-blocked pods from normal transient scheduling latency. Also list nodeclaims.karpenter.sh and recently-created nodes to detect in-flight scale-up activity. If listing the Pending-pods list returns 403/Forbidden → mark the Pending-pods signal UNKNOWN; do NOT read a forbidden pod list as "no Pending pods" (pod is a core API so 404 is not expected; an empty successful list means no Pending pods), so GREEN's "no Pending pods with FailedScheduling" precondition cannot be affirmed from an unreadable list — apply the 403 Pending-pods-signal rule in the Rating block below. If listing NodeClaims/nodes returns 403/Forbidden → mark scale-up activity UNKNOWN.

Rating:

  • 🟢 GREEN: Karpenter or EKS Auto Mode with consolidation enabled (AWS-preferred path), and the Pending-pods list SUCCEEDED and showed no Pending pods with FailedScheduling/insufficient-capacity events that lack corresponding scale-up activity (see AMBER for the precise discriminator). An unverifiable Pending-pods signal (list 403/Forbidden) cannot award GREEN's "no such Pending pods" precondition — see the 403 Pending-pods-signal rule.
  • 🟡 AMBER: Cluster Autoscaler present (legacy — consider migration to Karpenter), or Karpenter without consolidation, or the autoscaler is not responding to demand — defined as one or more pods Pending longer than a threshold (skill-defined heuristic: >2 minutes) that carry FailedScheduling events citing Insufficient cpu/memory or no nodes available AND show no corresponding scale-up activity (no new NodeClaims or recently-created nodes from step 6)
  • 🔴 RED: No cluster autoscaling — manual node management. Only reachable when the autoscaler reads (nodepools.karpenter.sh, cluster-autoscaler Deployment, and cluster computeConfig) all SUCCEEDED and genuinely found no autoscaler present.
  • ⬜ UNKNOWN: A 403/Forbidden on ANY autoscaler-presence read (the nodepools.karpenter.sh / Karpenter CRD list in step 1, the kube-system cluster-autoscaler Deployment list in step 2, OR the cluster computeConfig describe in step 3) leaves that presence signal undeterminable AND no autoscaler is confirmed present via a successful read (whether the other presence reads returned 404/NotFound, i.e. confirmed-absent, or also 403) — the absence of autoscaling is unconfirmed, so do NOT rate RED; rate UNKNOWN and note the forbidden presence read(s) under Investigate Manually. This covers the mixed case where one presence read is a confirmed 404 (e.g. Karpenter CRD 404 = Karpenter absent) while another is 403 (e.g. cluster-autoscaler Deployment list 403 = presence unknown) and none confirms an autoscaler — RED is blocked (it requires ALL three presence reads SUCCEEDED) and UNKNOWN catches it, so no state falls through to a gap. OR the scale-up / Pending-pods read is 403/Forbidden and that read is the SOLE remaining discriminator (see the 403 Pending-pods-signal rule) with no RED or AMBER arm independently confirmed.
  • 403 Pending-pods-signal rule (sole-discriminator only): the Pending-pods / scale-up read is a GREEN-vs-AMBER discriminator, not a floor. A 403 on this read routes to UNKNOWN ONLY when it is the SOLE discriminator — i.e. the autoscaler-presence reads (Karpenter/NodePool, cluster-autoscaler Deployment, Auto Mode) SUCCEEDED and would otherwise land GREEN, and the only thing separating GREEN from AMBER is whether capacity-blocked Pending pods exist. In that sole-discriminator case: (a) if there ARE Pending pods with FailedScheduling/insufficient-capacity events past the >2-minute threshold but the scale-up-activity read (NodeClaims/nodes, step 6) returned 403, the AMBER discriminator cannot be evaluated and GREEN's "no such pods" cannot be affirmed → UNKNOWN (or AMBER-with-note); (b) if the Pending-pods list itself returned 403, GREEN's "no Pending pods" precondition cannot be affirmed → cap GREEN at AMBER-with-note "autoscaler present but Pending-pod backlog could not be verified", or UNKNOWN if it is the sole signal. This rule NEVER lowers a confirmed floor: if an AMBER arm is INDEPENDENTLY confirmed (cluster-autoscaler Deployment present, Karpenter without consolidation, or a confirmed >2-min capacity-blocked pod with confirmed no scale-up), that confirmed AMBER stands and a 403 on the Pending-pods/scale-up read does NOT override it to UNKNOWN — the check rates AMBER (CONFIRMED FLOOR). Record the forbidden read under Investigate Manually.
  • Evaluation order: assess RED first; if not RED, assess AMBER; otherwise GREEN. Keeps the bands exhaustive and non-overlapping. Before rating RED, confirm the Karpenter/NodePool read did not return 403 with no other autoscaler confirmed — if it did, rate UNKNOWN instead (mirrors 7.2's "do not rate RED on a forbidden list"). A confirmed RED/AMBER survives a 403 on a different signal (Pending-pods/scale-up 403 does not demote a confirmed RED/AMBER to UNKNOWN).
  • Non-scoring note: true scale-up latency cannot be measured without load-testing; record any such latency question under Investigate Manually rather than as a band determinant.
  • Note: capacity-type mix (Spot vs On-Demand) is collected as supporting context for the autoscaling rating, not itself a band determinant.

7.2 — Horizontal Pod Autoscaler (HPA)​

What to check:

  • HPAs across all namespaces (targets, min/max, current replicas)
  • Multi-replica deployments without HPA
  • HPAs with minReplicas=1 (single point of failure)
  • KEDA ScaledObjects
  • VPA resources

How to check:

  1. List HPAs across all namespaces → check minReplicas, maxReplicas, current metrics. If 403/Forbidden when listing HPAs → mark the HPA signal UNKNOWN; do not rate RED on a forbidden list. (HPA is a core autoscaling/v2 API, so 404 is not expected; treat an empty successful list as no HPAs.)
  2. List Deployments with replicas > 1 → cross-reference with HPA targets. If 403/Forbidden when listing Deployments → mark the workload-enumeration signal UNKNOWN; do NOT read a forbidden list as "no workloads" (→ false N/A) or as "full coverage, no gaps" (→ false GREEN). apps/v1 Deployment is a core API so 404 is not expected; an empty successful list means no scalable application workloads. Apply the 403 Deployment-signal rule in the Rating block below. (Siblings 5.3, 7.3, and 8.1 guard the same Deployment list.)
  3. List HPAs where minReplicas == 1 (single point of failure for production workloads; acceptable for dev/staging)
  4. List ScaledObjects (KEDA CRD). If 404/NotFound → KEDA not installed, skip. If 403/Forbidden → mark UNKNOWN.
  5. List VPA resources. If 404/NotFound → VPA not installed, skip. If 403/Forbidden → mark UNKNOWN.

Rating:

  • N/A: No scalable application workloads present — a successful Deployment (and HPA) enumeration shows the cluster runs only kube-system add-ons / system workloads and zero scalable application workloads, so there is nothing to autoscale. Excluded from scoring (no finding; no scalable application workloads present — nothing to autoscale). Requires a SUCCESSFUL, empty application-workload enumeration; if the Deployment list returned 403/Forbidden, rate UNKNOWN, not N/A (a forbidden read must not produce a false-favorable N/A). N/A takes precedence over the colored bands.
  • 🟢 GREEN: Full coverage — the Deployment list SUCCEEDED and all stateless application/non-system workloads are autoscaled by HPA/KEDA/VPA, or a managed add-on's built-in autoscaling (e.g. CoreDNS add-on autoScaling) (min >= 2 where applicable), no gaps. Exclude kube-system add-ons that have their own managed autoscaling from the gap count. An unenumerable workload set (Deployment list 403/Forbidden) cannot award "full coverage, no gaps" — see the 403 Deployment-signal rule. A 403 on ONE autoscaler-mechanism list does NOT block GREEN when a different mechanism's successful read confirms full coverage: if the HPA list 403s but a successful KEDA (or VPA) read shows ScaledObjects/VPAs covering ALL application workloads, that confirmed-good coverage is GREEN regardless of the HPA-list 403 (the HPA list is not the sole discriminator here).
  • 🟡 AMBER: Autoscaling exists but with gaps — HPAs with min=1, or some application workloads missing HPA/KEDA/VPA coverage (partial adoption)
  • 🔴 RED: Scalable application workloads exist (a successful Deployment list found one or more scalable application workloads) but NONE are autoscaled — no HPA, KEDA, or VPA and all such workloads are at a fixed replica count. RED is NOT reachable when there are no scalable application workloads (that is N/A) nor from a forbidden Deployment/HPA list (that is UNKNOWN).
  • ⬜ UNKNOWN: The HPA (or KEDA/VPA) list returned 403/Forbidden so autoscaling presence cannot be determined from cluster state AND that 403'd list is the SOLE discriminator — i.e. no other autoscaler mechanism was successfully confirmed to cover the workloads. A 403 on ONE list does NOT force UNKNOWN when another mechanism read succeeded and settles the band: if the HPA list 403s BUT a successful KEDA (or VPA) read confirms ScaledObjects/VPAs covering ALL application workloads, that is GREEN via the confirmed-good path (see GREEN), not UNKNOWN. OR the Deployment list returned 403/Forbidden so the workload set cannot be enumerated (blocks both the N/A carve-out and the RED "workloads exist but unautoscaled" finding).
  • 403 Deployment-signal rule: the Deployment list is the workload-enumeration signal. When it returns 403/Forbidden the workload set is unenumerable: (a) do NOT award GREEN "full coverage, no gaps" on an unenumerable set — cap GREEN at AMBER-with-note "HPA/KEDA present but full workload coverage could not be verified" (or UNKNOWN if the HPA list also failed); (b) do NOT rate RED "all workloads at fixed replica count" on an unenumerable set; (c) do NOT award the N/A carve-out (a false "no workloads present"). A confirmed AMBER (a real HPA with min=1, or a confirmed uncovered workload seen before the failure) survives a 403 on a different list as a floor. Record the forbidden read under Investigate Manually.
  • Evaluation order: assess N/A first (no scalable application workloads → N/A, precedence over all colored bands); else assess RED first; if not RED, assess AMBER; otherwise GREEN. Keeps the bands exhaustive and non-overlapping.

7.3 — Pod Topology Spread & AZ Resilience​

What to check:

  • Node distribution across AZs
  • Deployments with topology spread constraints
  • Deployments with pod anti-affinity
  • Multi-replica deployments with neither (vulnerable to AZ failure)

How to check:

  1. List nodes → group by label topology.kubernetes.io/zone to count distinct AZs. If 403/Forbidden when listing nodes → mark the node/AZ signal UNKNOWN; do NOT infer an AZ count (single-AZ RED, 2-AZ AMBER, or >=3-AZ GREEN) from an unreadable node list. Node is a core API so 404 is not expected; an empty successful list means zero nodes running (0 AZs) — a Karpenter/Auto-Mode cluster scaled to zero at review time, which is the N/A zero-node case below, NOT a single-AZ RED. The AZ count is the sole discriminator for the RED (single-AZ) and AMBER (2-AZ) arms, so when the node list is 403 those arms cannot be evaluated — apply the 403 node/AZ-signal rule in the Rating block below.
  2. List Deployments → check spec.template.spec.topologySpreadConstraints. If 403/Forbidden when listing Deployments → mark the topology-spread / anti-affinity signal UNKNOWN; do NOT infer that deployments "lack topology spread / anti-affinity" from a forbidden list. (Deployment is a core apps/v1 API, so 404 is not expected; an empty successful list means no multi-replica deployments to assess.)
  3. List Deployments → check spec.template.spec.affinity.podAntiAffinity (same 403 guard as step 2 — a forbidden list leaves the topology-spread / anti-affinity signal UNKNOWN, not "absent").
  4. List multi-replica Deployments with neither topology spread nor anti-affinity — only meaningful when the step 2-3 Deployment list SUCCEEDED. If that list was 403/Forbidden, this comparison cannot be made; the topology-spread signal is UNKNOWN (the AZ-count signal from step 1 is independent and still usable for RED/AMBER on node distribution).

Rating:

  • N/A: A successful node list returned zero nodes (0 AZs) — a Karpenter/Auto-Mode cluster scaled to zero at review time. No nodes are running, so topology spread across AZs is not assessable ("no nodes running at review time — topology spread not assessable"). Excluded from scoring (no finding; mirrors the successful-empty → N/A pattern in 7.2 and the Fargate/Windows N/A pattern in 10.2). Requires a SUCCESSFUL, empty node list; if the node list returned 403/Forbidden, rate UNKNOWN, not N/A (a forbidden read must not produce a false-favorable N/A). N/A takes precedence over the colored bands.
  • 🟢 GREEN: A successful node read shows nodes in >= 3 AZs AND topology spread (or pod anti-affinity) confirmed (via a successful Deployment list) on production multi-replica deployments. A 403 node list cannot affirm the >=3-AZ precondition — see the 403 node/AZ-signal rule. (A successful zero-node list is N/A, not GREEN — the >=3-AZ text is not satisfied by 0 AZs.)
  • 🟡 AMBER: A successful node read shows nodes in exactly 2 AZs (regardless of spread config), OR a successful node read shows nodes in >= 3 AZs AND the Deployment list SUCCEEDED AND production multi-replica deployments were observed to lack topology spread / anti-affinity. The "lacks topology spread / anti-affinity" arm requires a successful Deployment list — it is NOT reachable from a forbidden (403) Deployment list, which routes to UNKNOWN instead. The 2-AZ arm requires a successful node read — a 403 node list is not "2 AZs" and routes to UNKNOWN.
  • 🔴 RED: Single-AZ — a successful step-1 node/AZ read shows nodes in exactly one AZ (>=1 node, one distinct zone), so no AZ redundancy is possible (decided from the node/AZ read alone, independent of the Deployment list). Not reachable from a 403/Forbidden node list (that is UNKNOWN — see the 403 node/AZ-signal rule) nor from a successful zero-node list (that is N/A — single-AZ RED requires at least one node in exactly one AZ). A confirmed single-AZ RED survives a 403 on the Deployment list (floor).
  • ⬜ UNKNOWN: The node list (step 1) returned 403/Forbidden so the AZ count cannot be read — the AZ count is the sole discriminator for the single-AZ RED and 2-AZ AMBER arms and a precondition for >=3-AZ GREEN, so none can be evaluated → UNKNOWN (do NOT award a false single-AZ RED, 2-AZ AMBER, or >=3-AZ GREEN from an unreadable node list); OR the node read SUCCEEDED with nodes in >= 3 AZs but the Deployment list (steps 2-4) returned 403/Forbidden so topology-spread / anti-affinity presence cannot be determined — do NOT rate AMBER "lacks topology spread" from a forbidden list. (A confirmed single-AZ RED still applies regardless, since it needs only a successful node/AZ read.)
  • 403 node/AZ-signal rule: the node/AZ read is the sole discriminator for the single-AZ RED and 2-AZ AMBER arms and a precondition for >=3-AZ GREEN. A 403 on the node list routes the AZ-count-dependent rating to UNKNOWN — never a false RED/AMBER/GREEN, and never N/A (the zero-node N/A carve-out requires a successful empty list; a forbidden node list is UNKNOWN, kept distinct from the zero-node case). A confirmed single-AZ RED or 2-AZ AMBER (successful node read) survives a 403 on the Deployment list (floor); conversely a confirmed AMBER "lacks topology spread" (successful reads) survives a later read failure elsewhere. True pod-to-node distribution cannot be verified without checking live pod placement; record any such question under Investigate Manually rather than as a band determinant. Record any forbidden read under Investigate Manually.
  • Zero-node vs 403 note: the two "no AZ count" cases are distinct and must not be conflated — a successful empty node list = zero nodes running = N/A (excluded from scoring); a 403/Forbidden node list = AZ count hidden by permissions = UNKNOWN. Only the successful-empty case is N/A.
  • Production-namespace decision rule: "production" is not directly observable from cluster state. When namespace criticality cannot be determined, treat application namespaces (exclude kube-system, kube-public, kube-node-lease, and default) as production — apply the GREEN/AMBER topology-spread / anti-affinity expectation to multi-replica deployments in those assumed-production namespaces, and record the assumption under Investigate Manually.
  • Evaluation order: assess N/A first (successful zero-node list → N/A, precedence over all colored bands); else assess RED first; if not RED, assess AMBER; otherwise GREEN. Keeps the bands exhaustive and non-overlapping.

Key talking point: Having nodes in 3 AZs doesn't mean pods are spread. Without topology spread constraints, the scheduler may pack all pods into one AZ.