Skip to main content
Source

This page is generated from devops-agent/eks-operation-review/references/deployment-practices.md. Edit the source, not this page.

Deployment Practices

Purpose​

Assess deployment strategies, CI/CD integration, and graceful shutdown configuration.

Automation Note​

CI/CD pipeline details (approval gates, post-deployment tests) are not fully detectable from cluster state. The skill checks for tool presence and configuration; process maturity items are marked UNKNOWN.

Checks to Execute​

8.1 — Deployment Strategy & Rollback​

What to check:

  • Deployment strategies in use (RollingUpdate vs Recreate)
  • maxUnavailable and maxSurge settings — evaluate the RESOLVED integer, not the raw percentage. The default 25% maxUnavailable rounds DOWN for small replica counts (25% of 2 = 0, 25% of 3 = 0), which is already safe; the risk is when it resolves to >= 1 (25% of 4 = 1), allowing capacity to drop mid-rollout. Recommend an explicit maxUnavailable: 0, maxSurge: 1 for latency-sensitive workloads so a rollout never reduces ready replicas.
  • Argo Rollouts resources
  • Flagger Canary resources
  • terminationGracePeriodSeconds and preStop hooks

How to check:

  1. List Deployments → inspect spec.strategy.type, rollingUpdate.maxUnavailable, rollingUpdate.maxSurge. If 403/Forbidden when listing Deployments → mark BOTH the deployment-strategy signal AND the graceful-shutdown signal (steps 1–2 and step 5 all read this same Deployment list) UNKNOWN (do not conclude Recreate, and do not conclude "no preStop / no non-default terminationGracePeriodSeconds"); apps/v1 Deployment is a core API so 404 is not expected; an empty successful list means no Deployments. A forbidden Deployment list can never yield a RED — route to UNKNOWN (mirrors 5.2's identical guard on the same list).
  2. For each deployment, resolve maxUnavailable against the replica count (round the percentage DOWN to an integer). Flag deployments where the resolved maxUnavailable is >= 1 (capacity can drop during a rollout); a resolved value of 0 is safe.
  3. List Rollouts (Argo Rollouts CRD, if exists). If the CRD list returns 404/NotFound → the tool is not installed; if 403/Forbidden → mark that progressive-delivery signal UNKNOWN rather than assuming absence.
  4. List Canaries (Flagger CRD, if exists). If the CRD list returns 404/NotFound → the tool is not installed; if 403/Forbidden → mark that progressive-delivery signal UNKNOWN rather than assuming absence.
  5. Inspect Deployments for terminationGracePeriodSeconds and lifecycle.preStop (same Deployment list as steps 1–2; if that list was 403/Forbidden the graceful-shutdown signal is UNKNOWN per step 1, not "absent")

Rating:

  • 🟢 GREEN: Resolved maxUnavailable is 0 (zero-downtime rollout), graceful shutdown configured
  • 🟡 AMBER: Rolling update where resolved maxUnavailable is >= 1
  • 🔴 RED: Recreate strategy on a Deployment (causes downtime on every rollout, regardless of whether the workload is user-facing), OR no deliberately-configured graceful shutdown (neither a preStop hook nor an explicitly-set non-default terminationGracePeriodSeconds) on a long-lived workload — both RED triggers require a SUCCESSFUL Deployment read; a 403/Forbidden Deployment list is UNKNOWN, never RED (a forbidden read cannot confirm Recreate or confirm a missing preStop/non-default tGPS)
  • ⬜ UNKNOWN: A 403/Forbidden on the Deployment list (step 1) — the deployment-strategy and graceful-shutdown signals both become UNKNOWN and the RED band is unreachable from a forbidden Deployment list. (Rollback speed and CI/CD rollback process are never observable from cluster state — that is an always-true unobservable, so it is NOT a band trigger and does NOT by itself force the check to UNKNOWN; it is recorded under Investigate Manually, below.)
  • ◻️ N/A: no Deployments present — a SUCCESSFUL, empty Deployment list means there is no rollout strategy or graceful-shutdown target to assess, so 8.1 is excluded from scoring (dropped from the score denominator, consistent with the report-generation rubric and with 7.2/8.3) rather than awarded a vacuous GREEN that would inflate the maturity score. This N/A requires a SUCCESSFUL empty list; a Deployment-list 403 is UNKNOWN (per step 1 and the UNKNOWN band above), never N/A — a forbidden list cannot confirm "no Deployments." (StatefulSet-only clusters with no Deployments therefore rate N/A here; their rollout/graceful-shutdown posture, if any, is not assessed on 8.1's axis.)
  • Confirmed floor (403 on a different signal): a RED or AMBER established from a SUCCESSFUL Deployment read is NOT downgraded by a 403 on a different read. A confirmed Recreate-strategy RED (or a resolved-maxUnavailable >= 1 AMBER) survives a 403/Forbidden on the Rollouts or Canaries CRD list (steps 3–4) — progressive-delivery tooling is a GREEN/AMBER discriminator only and can never lift a confirmed RED/AMBER back to UNKNOWN. Only the Deployment list itself (steps 1–2/5) returning 403 removes the strategy/graceful-shutdown signals; a 403 elsewhere leaves a confirmed color standing.
  • Graceful shutdown configured = the workload has EITHER a preStop hook OR an explicitly-set (non-default) terminationGracePeriodSeconds. Kubernetes always defaults terminationGracePeriodSeconds to 30s, so it is never observably missing; the default 30s alone (with no preStop hook) does NOT count as deliberately-configured graceful shutdown.
  • Evaluation order: if the Deployment list returned 403/Forbidden → UNKNOWN (strategy and graceful-shutdown signals unreadable). Else, if the Deployment list SUCCEEDED but is empty → N/A (no rollout strategy to assess; excluded from scoring). Else (≥1 Deployment read successfully) assess RED first; if not RED, assess AMBER; otherwise GREEN. Keeps the bands exhaustive and non-overlapping.
  • Key talking point: Progressive delivery (Argo Rollouts / Flagger) is a bonus that reduces blast radius, not a GREEN gate — a cluster with resolved maxUnavailable 0 and graceful shutdown is GREEN whether or not progressive-delivery tooling is present.
  • Scoring authority: 8.1 OWNS graceful-shutdown scoring — the existence of deliberately-configured graceful shutdown (a preStop hook OR an explicitly-set non-default terminationGracePeriodSeconds) on long-lived workloads is rated here, and here only. 8.3 OWNS a different axis — connection-draining / deregistration-delay alignment (does terminationGracePeriodSeconds cover the explicitly-configured LB target-group deregistration delay) — and DEFERS the graceful-shutdown-existence signal to 8.1: it may gather preStop presence only as evidence for its draining assessment but does NOT rate whether graceful shutdown exists, nor penalize a workload for a missing preStop alone. This prevents the same preStop/terminationGracePeriodSeconds state from being scored (and rated differently) by both checks.

Investigate manually: rollback speed and the CI/CD rollback process (how fast a bad release can be reverted, whether rollbacks are automated or gated) are not observable from cluster state — record for the user to confirm; these do not move the band.

8.2 — CI/CD Pipeline Integration​

What to check:

  • ECR repositories: scan-on-push, tag immutability
  • Admission webhooks enforcing image policies
  • Image registries in use (ECR vs public)

How to check:

  1. Describe ECR repositories → scanOnPush, imageTagMutability
  2. List ValidatingWebhookConfigurations → filter for image/policy-related names. If 403/Forbidden when listing ValidatingWebhookConfigurations/MutatingWebhookConfigurations → mark the admission-enforcement signal UNKNOWN (route to the UNKNOWN band); do not rate RED on a forbidden list. (Admission webhook configs are core admissionregistration.k8s.io APIs, so 404 is not expected; an empty successful list means no admission webhooks.)
  3. List running pods → aggregate image registries

Rating: The rated axis is admission enforcement, which cluster inspection can actually observe. GitOps/deployment-automation CRDs only distinguish GREEN from AMBER when enforcement is present; their absence never drives RED (mirroring 2.2/2.3, external CI/CD is undetectable from cluster state).

  • 🟢 GREEN: Admission enforcement present AND GitOps / deployment automation observable in-cluster (e.g. Argo/Flux CRDs) — image scan-on-push / tag immutability / registry trust are rated under 5.4
  • 🟡 AMBER: Admission enforcement present but deployment automation / CI/CD provenance is not determinable from cluster state (no GitOps CRDs observable) — the CI/CD-maturity question is routed to Investigate-Manually, not RED
  • 🔴 RED: No admission enforcement (regardless of whether GitOps/deployment-automation CRDs are present) — absent enforcement is itself the observable gap. A cluster is NOT rated RED purely for absent GitOps CRDs; GitOps-present-but-no-enforcement still lands here because enforcement is missing.
  • ⬜ UNKNOWN: Cannot determine whether admission enforcement is present from cluster state — suggest user investigate
  • Evaluation order: assess RED first (no admission enforcement → RED, regardless of GitOps). If enforcement is present, rate GREEN when GitOps/deployment automation is observable, otherwise AMBER. The three bands are jointly exhaustive: every cluster is either (enforcement present + automation observable → GREEN) / (enforcement present + provenance undetermined → AMBER) / (no enforcement → RED). CI/CD process maturity itself is unobservable from cluster state and belongs to Investigate-Manually, not the bands.
  • Confirmed floor (403 on a different signal): the RED trigger is a SUCCESSFUL, empty admission-webhook list confirming no enforcement. Once that no-enforcement RED is confirmed, it is NOT downgraded by a 403 on a different read — a forbidden ECR DescribeRepositories (step 1) or a forbidden running-pod list (step 3) leaves the confirmed no-enforcement RED standing (those reads feed 5.4's registry/scan signals, not 8.2's rated axis). Only a 403 on the webhook list itself (step 2) removes the enforcement signal and routes to UNKNOWN. No unearned GREEN: when enforcement is confirmed present but the GitOps/deployment-automation CRD reads returned 403/Forbidden (so automation can be neither confirmed nor refuted), do NOT award GREEN — cap at AMBER-with-note (enforcement confirmed, automation unconfirmable) and record the automation uncertainty under Investigate Manually; GREEN requires enforcement present AND automation observably confirmed.
  • Scoring authority: ECR scan-on-push and image tag immutability are rated under check 5.4 (Image Tag Hygiene); 8.2 assesses admission enforcement and observable deployment automation.

Investigate manually: For AMBER clusters (enforcement present, no GitOps CRDs), how are workloads deployed and promoted — external CI/CD (Jenkins, GitHub Actions), approval gates, post-deployment tests? These are not observable from cluster state.

8.3 — Graceful Shutdown & Connection Draining​

What to check:

  • Deployments with preStop hooks vs without
  • terminationGracePeriodSeconds (default 30s vs customized)
  • Services with AWS Load Balancer annotations (deregistration delay matters)
  • Ingress resources with target group attributes

How to check:

  1. List Deployments → count those with lifecycle.preStop vs without. If 403/Forbidden when listing Deployments → mark the draining-alignment signal UNKNOWN for any externally-facing workload (steps 1–2 both read this same Deployment list; terminationGracePeriodSeconds becomes unreadable, so alignment against an explicitly-configured deregistration delay cannot be assessed). apps/v1 Deployment is a core API so 404 is not expected; an empty successful list means no Deployments. A forbidden Deployment list can never yield a RED — and must never yield a silent GREEN pass — on 8.3's axis when an externally-facing workload exists: route to UNKNOWN (mirrors 8.1's identical guard on the same list).
  2. List Deployments → check terminationGracePeriodSeconds (null = default 30s) — same Deployment list as step 1; if that list was 403/Forbidden, tGPS is unreadable and the draining-alignment signal is UNKNOWN per step 1, not silently treated as aligned/GREEN.
  3. List Services → check for service.beta.kubernetes.io/aws-load-balancer-type annotation (identifies a LoadBalancer-type/NLB-exposed workload). Also read service.beta.kubernetes.io/aws-load-balancer-target-group-attributes on the same Service — a deregistration_delay.timeout_seconds set here is an EXPLICITLY-configured NLB deregistration delay and MUST be collected as a delay source for the alignment comparison (a Service-exposed workload can carry an explicit delay with no Ingress at all). If 403/Forbidden → mark the externally-facing-workload signal UNKNOWN (do not conclude 'no externally-facing workloads'); Service is core v1, so 404 is not expected; an empty successful list means none exist.
  4. List Ingresses → check for alb.ingress.kubernetes.io/target-group-attributes annotation. If 403/Forbidden → mark the externally-facing-workload signal UNKNOWN (do not conclude 'no externally-facing workloads'); Ingress is core networking.k8s.io, so 404 is not expected; an empty successful list means none exist. The explicitly-configured deregistration delay is gathered from BOTH sources — the Ingress ALB alb.ingress.kubernetes.io/target-group-attributes annotation (step 4) AND the NLB Service service.beta.kubernetes.io/aws-load-balancer-target-group-attributes annotation (step 3) — plus any observable TargetGroupBinding; a delay set via EITHER counts as explicitly configured for the comparison below. (terminationGracePeriodSeconds should be greater than or equal to the target-group deregistration delay — the pod must stay alive until in-flight connections finish draining). This comparison applies ONLY when the deregistration delay is explicitly set (via the ALB Ingress annotation, the NLB Service annotation, or an observable TargetGroupBinding). When the delay is not explicitly configured (neither annotation nor TargetGroupBinding sets it) AND the Deployment/tGPS read succeeded, do NOT infer it from the ALB default of 300s — with no explicit delay there is nothing to misalign against, so draining alignment is not a defect and rates GREEN on 8.3's axis (this is NOT an UNKNOWN case). (If the Deployment/tGPS read returned 403, the no-explicit-delay state does NOT rate GREEN — it routes to UNKNOWN per the UNKNOWN band trigger (b) below.)

Rating:

  • 🟢 GREEN: externally-facing workloads exist, the Deployment/tGPS read SUCCEEDED, AND either (i) terminationGracePeriodSeconds is aligned with (≥) the explicitly-configured LB target-group deregistration delay so in-flight connections drain before pod termination, OR (ii) no explicit deregistration delay is configured (from BOTH the ALB Ingress and NLB Service annotations, plus TargetGroupBinding) so there is nothing to misalign against. Both GREEN arms require a SUCCESSFUL Deployment/tGPS read — the no-explicit-delay arm applies ONLY when the Deployment read succeeded; if the Deployment read 403'd it is UNKNOWN via trigger (b), never GREEN. GREEN requires at least one externally-facing workload to actually assess; a cluster with none is N/A (nothing to drain), not a free GREEN.
  • 🔴 RED: externally-facing workloads exist AND terminationGracePeriodSeconds is shorter than an explicitly-configured target-group deregistration delay for AT LEAST ONE of them (guaranteed connection drops on rollout — only when the delay is explicitly configured, never inferred from the 300s ALB default). The explicit delay may come from EITHER the ALB Ingress alb.ingress.kubernetes.io/target-group-attributes annotation OR the NLB Service service.beta.kubernetes.io/aws-load-balancer-target-group-attributes annotation (or a TargetGroupBinding) — so a Service-exposed (NLB) workload whose explicit delay exceeds its tGPS is reachable RED here, not a false GREEN. Requires a SUCCESSFUL Deployment read of the offending workload's tGPS AND a SUCCESSFUL read of the exposing Service (or Ingress); a 403/Forbidden on the Deployment list is UNKNOWN, never RED. Confirmed floor (R1): once a misalignment is confirmed this way, a 403 on the other externally-facing list (e.g. the offending workload is confirmed via a successful Service read but the Ingress list 403s) does NOT demote the RED to UNKNOWN — the Ingress 403 only hides ADDITIONAL externally-facing workloads, it cannot un-confirm the one already read as misaligned. A confirmed misalignment RED is the floor.
  • ⬜ UNKNOWN: the draining-alignment signal cannot be observed. Two triggers: (a) the Service or Ingress list returned 403/Forbidden so the externally-facing-workload set cannot be fully observed, AND no externally-facing workload has already been CONFIRMED misaligned from the successful reads — do NOT emit GREEN or N/A on the strength of an unconfirmed set. Floor exception (R1): if a Service (or Ingress) list SUCCEEDED and, together with a SUCCESSFUL Deployment read, already confirms at least one misaligned externally-facing workload, that is a confirmed RED and it SURVIVES a 403 on the other externally-facing list (the 403 only hides ADDITIONAL externally-facing workloads, it cannot un-confirm the one already observed misaligned) — rate RED, not UNKNOWN. Trigger (a) fires only when the 403 is the SOLE discriminator (no confirmed RED from the successful reads). (b) an externally-facing workload IS confirmed (Service/Ingress read succeeded) but the Deployment list returned 403/Forbidden, so its terminationGracePeriodSeconds is unreadable and alignment cannot be assessed — this applies REGARDLESS of whether an explicit deregistration delay was observed: even when the successful Service/Ingress reads show NO explicit delay, a delay could exist on a Deployment-level field that was not read, so route to UNKNOWN, never let a Deployment-403 fall through to GREEN. Route UNKNOWN to Investigate Manually. Note: the "no explicit deregistration delay configured" case rates GREEN ONLY when the Deployment read SUCCEEDED (externally-facing workloads present, tGPS readable, delay not set from any source) — default LB behavior with no explicit delay is not a defect, so it rates GREEN (nothing misaligned), never UNKNOWN and never RED (do not infer RED from the 300s ALB default). When the Deployment read 403'd, the no-explicit-delay case is UNKNOWN via trigger (b), not GREEN — the two paths are mutually exclusive on the Deployment-read outcome, so no state matches both.
  • ◻️ N/A: there are provably (successful Service AND Ingress lists) zero externally-facing workloads — the connection-draining axis does not apply to this cluster (nothing to drain), so 8.3 is excluded from scoring (consistent with the report-generation rubric: N/A checks are dropped from the score denominator) rather than awarded a free GREEN that would inflate the maturity score. This requires SUCCESSFUL, empty Service and Ingress reads; an UNCONFIRMED absence (403 on either list) is NOT N/A — it is UNKNOWN.
  • Externally-facing = a deployment exposed via a LoadBalancer-type Service or an Ingress. A provably-empty (successful) externally-facing set is NOT RED and NOT GREEN — 8.3's axis is connection draining, and with nothing to drain there is nothing to assess, so the check is N/A (excluded from scoring); whether those (nonexistent-here) workloads have deliberately-configured graceful shutdown is rated by 8.1, not here. An UNCONFIRMED absence (403 on the Service/Ingress list) is NOT treated as zero — it is UNKNOWN.
  • Scoring authority: graceful-shutdown existence (whether a preStop hook or non-default terminationGracePeriodSeconds is deliberately configured at all) is OWNED by 8.1. 8.3 OWNS the connection-draining / deregistration-alignment axis — whether terminationGracePeriodSeconds covers the explicitly-configured LB target-group deregistration delay so in-flight connections drain before pod termination. 8.3 may GATHER preStop presence as evidence for that draining assessment but does NOT rate any workload RED for a missing preStop alone (that is 8.1's signal; 8.3 has no AMBER band). A workload with no externally-facing exposure and no preStop is therefore scored by 8.1, and never penalized on 8.3's axis for the missing preStop. This guarantees the same preStop/terminationGracePeriodSeconds state is not rated by both checks.
  • Bands: 8.3 rates GREEN / RED / UNKNOWN / N/A only — there is NO AMBER band on this axis, and every one of the four listed bands is reachable (see the evaluation-order partition below: N/A ← provably-zero externally-facing workloads (successful empty Service AND Ingress lists); UNKNOWN ← 403 on Service/Ingress with NO confirmed misalignment from the successful reads, OR a 403 on the Deployment tGPS read of a confirmed workload (the latter regardless of whether an explicit delay was seen); RED ← ≥1 confirmed misaligned workload (its exposing Service/Ingress read AND its Deployment/tGPS read both succeeded) — this RED holds even if the other externally-facing list 403'd, since the confirmed misalignment is the floor; GREEN ← ≥1 externally-facing workload with the Deployment read succeeded, all aligned or no explicit delay). The draining-alignment axis is binary per workload (for each externally-facing workload with an explicitly-configured deregistration delay, tGPS is either ≥ the delay = aligned, or < the delay = misaligned), so any "partial alignment" state necessarily contains a misaligned workload, which is exactly the RED trigger. A separate AMBER band would therefore be unreachable under RED-first evaluation and is omitted rather than left as a dead band with a false "non-overlapping" claim. (Progressive-delivery / preStop-margin nuance is not scored here: missing-preStop existence is owned solely by 8.1.)
  • Evaluation order: (0) Confirmed-RED floor (assessed FIRST, R1): if the successful reads already confirm ≥1 externally-facing workload misaligned (a Service- or Ingress-exposed workload read successfully AND its tGPS read successfully from the Deployment list AND tGPS < its explicitly-configured deregistration delay) → RED, regardless of any 403 on the other externally-facing list — a confirmed misalignment is the floor and an Ingress-list (or Service-list) 403 only hides ADDITIONAL workloads, it cannot demote the one already confirmed. (1) Else, if the Service or Ingress list returned 403/Forbidden → UNKNOWN (externally-facing set unconfirmed and the 403 is the SOLE discriminator — no confirmed RED exists). (2) Else the externally-facing set is fully confirmed: if it is empty → N/A (nothing to drain; excluded from scoring). (3) Else ≥1 externally-facing workload is confirmed present — if the Deployment list needed to read their terminationGracePeriodSeconds returned 403/Forbidden → UNKNOWN (tGPS unreadable, alignment cannot be assessed; do NOT fall through to GREEN). (4) Else (Deployment read succeeded) assess RED: ≥1 externally-facing workload with tGPS shorter than its explicitly-configured deregistration delay → RED. (5) Otherwise → GREEN (≥1 externally-facing workload, all aligned or no explicit delay configured — nothing misaligned). The "no explicit deregistration delay configured" case is NOT undeterminable WHEN the Deployment read succeeded — it rates GREEN (step 5), so it never reaches UNKNOWN; but when the Deployment read 403'd it is caught first at step 3 → UNKNOWN (a delay could exist on an unread Deployment field), so the no-delay+Deployment-403 state resolves to exactly one band (UNKNOWN) and never also to GREEN. Every observable state maps to exactly one band: ≥1 CONFIRMED-misaligned externally-facing workload (its Service/Ingress read AND its Deployment/tGPS read both succeeded) → RED even if the other externally-facing list 403'd (step 0); else a Service-or-Ingress 403 with no confirmed misalignment → UNKNOWN (step 1); provably-zero externally-facing workloads → N/A (step 2); ≥1 confirmed workload with a 403 Deployment/tGPS read (whether or not an explicit delay was seen) → UNKNOWN (step 3); ≥1 misaligned externally-facing workload (all reads succeeded) → RED (step 4); ≥1 externally-facing workload all aligned or no explicit delay (successful reads) → GREEN (step 5). This partition is exhaustive and non-overlapping — steps 0→5 are evaluated in order, so each state is claimed by the first matching step only.

Key talking point: There's a race condition during pod termination. The LB still sends traffic for a few seconds after SIGTERM. A preStop sleep of 5-10s fixes it.