Skip to main content
Source

This page is generated from devops-agent/eks-operation-review/references/cluster-lifecycle.md. Edit the source, not this page.

Cluster Lifecycle & Upgrades

Purpose​

Assess EKS cluster version currency, data plane alignment, upgrade readiness, add-on compatibility, and upgrade process maturity.

EKS Version Support Status​

Last verified: 2026-08-09. Support status is determined primarily from the live EKS API; the dated table below is a fallback used only when the API is unavailable.

Determine support status from the live API first; fall back to the dated table only when the API cannot be reached. Do NOT guess or use training data.

Primary (live) method — preferred: Call the EKS DescribeClusterVersions API to get the authoritative real-time list of EKS versions and each version's versionStatus (falling back to the deprecated status field) — one of STANDARD_SUPPORT / EXTENDED_SUPPORT / UNSUPPORTED. Define latest = the highest version whose status is STANDARD_SUPPORT in the API response.

Fallback method — only when the live API is unavailable: Use the dated table below. In fallback mode, latest = the highest STANDARD_SUPPORT version in this table (not the true current latest), so ratings are relative to the table and may lag reality — note fallback mode in the finding.

This fallback table was sourced from the EKS upgrade skill's version data on the "as of" date shown in the table header; it goes stale as new EKS versions ship and support windows advance — refresh it periodically against the official EKS version calendar. Prefer the DescribeClusterVersions API with includeAll set (which also surfaces UNSUPPORTED versions) whenever the live API is reachable. If the API is unavailable and the cluster runs a version not present in this fallback table, do not guess from training data — classify by the table's bounds instead (this applies in fallback mode only; if the live API is reachable, the API's own support status governs and this fallback logic does not apply). A version below the lowest table entry (older than 1.30, the lowest version in this fallback table — note that 1.30 is itself now UNSUPPORTED, its extended support having ended July 23, 2026, so 1.31 is the oldest version still in extended support) is past extended-support end — rate 🔴 RED, since it is out of support, and note that it is below the fallback table and out of support. A version newer than the highest table entry (1.36) may be a brand-new release that the stale fallback table simply predates — do not rate it out of support. Rate it 🟢 GREEN via the out-of-range guard in check 1.1 and note that the version is newer than the fallback table (last verified 2026-08-09).

VersionStandard Support UntilExtended Support UntilStatus (as of 2026-08-09)
1.36August 2, 2027August 2, 2028✅ STANDARD_SUPPORT (latest in table)
1.35March 27, 2027March 27, 2028✅ STANDARD_SUPPORT
1.34December 2, 2026December 2, 2027✅ STANDARD_SUPPORT
1.33July 29, 2026July 29, 2027⚠️ EXTENDED_SUPPORT (standard ended)
1.32March 23, 2026March 23, 2027⚠️ EXTENDED_SUPPORT (standard ended)
1.31November 26, 2025November 26, 2026⚠️ EXTENDED_SUPPORT (standard ended)
1.30July 23, 2025July 23, 2026⛔ UNSUPPORTED (extended support ended July 23, 2026)

CRITICAL: The upgradePolicy.supportType field from the API is a CONFIGURATION PREFERENCE, not the current billing status. A cluster on a standard-support version with supportType: EXTENDED is still on standard support and NOT paying the extended premium. Always determine actual support status from the DescribeClusterVersions API (or the fallback table), never from supportType.

Checks to Execute​

1.1 — Control Plane Version Currency​

What to check:

  • The cluster's current Kubernetes version and platform version
  • The version's actual support status (STANDARD_SUPPORT / EXTENDED_SUPPORT / UNSUPPORTED)
  • Report the actual support status, NOT the supportType API field

How to check:

  1. Describe the cluster → get version and platformVersion
  2. Call the EKS DescribeClusterVersions API → find the cluster's version and its versionStatus (falling back to the deprecated status field), and identify latest = the highest STANDARD_SUPPORT version. If the live API is unavailable, fall back to the dated table above and note fallback mode in the finding.
  3. Report: version, standard/extended/unsupported status, and when the current support period ends

Rating (evaluate RED first, then AMBER, then GREEN):

  • 🔴 RED: Version is on extended support, or unsupported (past its extended-support end date) — extended support incurs the cost premium below; unsupported means running with no support
  • 🟡 AMBER: Version is on standard support but older — a standard version more than one minor behind the latest standard version
  • 🟢 GREEN: Version is the latest standard-support version or N-1 (one minor behind the latest standard version)
  • ⬜ UNKNOWN: Cannot determine the version (should not happen with live access); OR the DescribeClusterVersions call succeeded but returned no versions (an empty list) — this is NOT a 403 (do not treat an empty-but-successful response as access-denied, and do not read empty data as "version unsupported"). Prefer falling back to the dated table above (treat empty-but-successful as "live version data unavailable") and note fallback mode; if the table is also unavailable, rate this dimension UNKNOWN with the note "version support data unexpectedly empty from a successful DescribeClusterVersions response." Floor-preserving: an empty-but-successful response never yields GREEN on its own.

Version out-of-range guard (fallback mode only): If using the fallback table and the cluster version is higher than the highest version in the table, rate GREEN and note: "Version 1.X is newer than the fallback table (last verified 2026-08-09); the live API was unavailable. Rated GREEN as latest. Refresh the table when convenient."

Extended-support cost impact: Extended support has historically cost ~$0.60/hr vs ~$0.10/hr for standard support. These rates are indicative and subject to change — verify against the current Amazon EKS pricing page before quoting figures. Compute, do not estimate:

extra_cost_per_month = (extended_rate - standard_rate) × 730
total_extended_cost  = extended_rate × 730
# With the indicative rates above: extra = (0.60 - 0.10) × 730 = ~$365/month per cluster
# 730 = average hours per month (365 days × 24 hours ÷ 12 months)

Key talking point: At indicative rates, extended support adds roughly $365/month per cluster — verify against the EKS pricing page before quoting a figure.

1.2 — Data Plane Version Alignment​

What to check:

  • List all node groups and their Kubernetes versions
  • Compare each node group version against the control plane version
  • Check AMI type (AL2 vs AL2023 vs Bottlerocket)
  • Check for Karpenter NodePools or EKS Auto Mode

How to check:

  1. List node groups → describe each for version, AMI type, capacity type. 403 guard: if listing or describing node groups returns 403/Forbidden → mark the node-group and AMI-type signals UNKNOWN; do NOT read a forbidden result as "no managed node groups" or "not AL2." An unconfirmed AMI type must not be assumed non-AL2, so the AL2 end-of-life AMBER cap cannot be silently skipped — report AMI type UNKNOWN rather than granting GREEN on the AMI dimension.
  2. List nodes via Kubernetes API → get kubelet versions. 403 guard: if this read returns 403/Forbidden → mark the version-skew signal UNKNOWN (a forbidden read is the auditor identity's own permission restriction, not a cluster state); do NOT route it to RED. A forbidden node-list marks ONLY the version-skew signal UNKNOWN — it does NOT by itself force the whole check UNKNOWN: a confirmed end-of-life AMI type from a successful describe-nodegroup (step 1) still forces at-least-AMBER (confirmed-floor), and the whole check goes UNKNOWN only when BOTH the version-skew signal AND the AMI-type signal are unavailable (see UNKNOWN band and the combo map). The RED "no visibility into node versions" arm applies ONLY when the node-list read SUCCEEDS but returns no parseable/usable version data (a genuine no-visibility state) — never when the read was forbidden.
  3. Check for Karpenter NodePools (nodepools.karpenter.sh). If 404/NotFound (CRD not installed) → Karpenter is not deployed, rate based on other node management. If 403/Forbidden → mark Karpenter status UNKNOWN. Note: EKS Auto Mode is built on Karpenter and serves the karpenter.sh CRDs too, so the nodepools.karpenter.sh CRD being present does NOT by itself mean self-managed (open-source) Karpenter is deployed — distinguish Auto Mode via step 4.
  4. Describe cluster → check computeConfig.enabled for Auto Mode (cluster-level signal). Auto Mode nodes also carry the label eks.amazonaws.com/compute-type=auto; use these to tell Auto Mode apart from self-managed Karpenter even though both serve the nodepools.karpenter.sh CRD.

Rating:

  • 🟢 GREEN: All nodes within N-1 of control plane, using managed node groups/Karpenter/Auto Mode, AND the AMI-type signal was SUCCESSFULLY read and confirms no node runs an end-of-life AMI type (see AMBER). GREEN requires BOTH preconditions confirmed — an unconfirmable good signal never earns GREEN (see the AMI-UNKNOWN cap and the combo map)
  • 🟡 AMBER: Nodes within two minors of the control plane but mixed versions, or self-managed nodes (exactly 2 minors behind lands here — still within the upstream kubelet N-3 skew policy); OR nodes running an end-of-life AMI type — AL2, whose EKS-optimized AMIs ended at Kubernetes 1.32 and whose OS support ended 2026-06-30 — which caps the rating at AMBER regardless of version skew (migrate to AL2023 or Bottlerocket), so an otherwise-GREEN cluster on AL2 AMIs lands here; OR the version-skew signal is GREEN-worthy (nodes successfully read, all within N-1) but the AMI-type signal is UNKNOWN because describe-nodegroup returned 403 — GREEN's AMI precondition is unconfirmed, so cap at AMBER-with-note ("versions current but AMI type could not be verified") and record the AMI uncertainty under Investigate Manually; this never raises a band (a confirmed >N-2 RED still requires a successful version read), it only caps an otherwise-GREEN down to AMBER
  • 🔴 RED: Any node more than 2 minors behind the control plane, or a successful node-list read that returns no parseable version data for the nodes it lists (a genuine no-visibility state) (this >N-2 bound is a skill-defined operational standard — stricter than the upstream kubelet version-skew policy, which permits up to N-3). A 403/Forbidden on the node-list read does NOT satisfy this arm — that is UNKNOWN, not RED.
  • ⬜ UNKNOWN: No nodes found (possible if cluster is new or uses Fargate only); OR both the version-skew signal AND the AMI-type signal are unavailable — i.e. the node-list read was forbidden (403) AND the AMI type could not be confirmed from a successful describe-nodegroup. A forbidden node-list alone does NOT force whole-check UNKNOWN when a successful describe-nodegroup confirmed the AMI type (that path caps at AMBER-with-note if AMI is clean, or AMBER if AMI is AL2/EOL — confirmed-floor). A permission restriction is never a RED. (The node-list-403 arm is retained as defense-in-depth even though pre-flight typically hard-stops on a failed node list; when it does fire it marks only the version-skew signal UNKNOWN, not the whole check.)
  • Evaluation order: assess RED first, then AMBER, otherwise GREEN. Version-skew RED (any node >N-2, or a successful read that yields no parseable node versions) dominates and is evaluated before the AMI-type check, so an AL2 node that is also >N-2 behind stays RED; the AL2 end-of-life signal only caps at AMBER a cluster that would otherwise be GREEN. A forbidden (403) node-list read is not a RED — it marks only the version-skew signal UNKNOWN per the 403 guard in step 2. Keeps the bands exhaustive and non-overlapping.
  • Signal-combination map (version-skew × AMI-type → exactly one band; confirmed RED/AMBER survives a 403 on the other signal):
    • version >N-2 (confirmed RED), AMI = any (AL2 / AL2023 / unreadable) → RED (version-skew RED dominates and survives an AMI-read 403 — confirmed-floor)
    • version N-1/current (confirmed), AMI = AL2/EOL (confirmed) → AMBER (AL2 end-of-life cap)
    • version N-1/current (confirmed), AMI = AL2023/Bottlerocket (confirmed clean) → GREEN (both preconditions confirmed)
    • version N-1/current (confirmed), AMI = unreadable (describe-nodegroup 403) → AMBER-with-note ("versions current but AMI type could not be verified"; GREEN's AMI precondition unconfirmed)
    • version unreadable (node-list 403), AMI = AL2/EOL (confirmed) → AMBER (confirmed EOL AMI forces at-least-AMBER even though version-skew is UNKNOWN — confirmed-floor)
    • version unreadable (node-list 403), AMI = AL2023/Bottlerocket (confirmed clean) → AMBER-with-note ("AMI type verified clean but node version skew could not be verified"; no unearned GREEN)
    • version unreadable (node-list 403), AMI = unreadable (describe-nodegroup 403) → UNKNOWN (both discriminators unavailable, nothing confirmed yields a color)

Red flags: AL2 OS is past EOL (2026-06-30) and EKS AL2 AMIs ended with Kubernetes 1.32 — migrate to AL2023 or Bottlerocket; self-managed nodes with no automated upgrade path.

1.3 — Upgrade Readiness & Deprecated API Detection​

What to check:

  • EKS Cluster Insights for upgrade blockers
  • Presence of deprecated API usage
  • PodSecurityPolicy resources (removed in K8s 1.25)

How to check:

  1. Get EKS Insights → filter for UPGRADE_READINESS category
  2. List any PodSecurityPolicy resources via Kubernetes API. If 404/NotFound or the API group is not served (expected on Kubernetes ≥1.25, where PSP was removed) → no PSPs present, not a finding; if 403/Forbidden → mark the PSP signal UNKNOWN rather than assuming absence.
  3. Check for Helm releases in kube-system as supporting evidence for deprecated-API risk — Helm-managed workloads there may ship manifests targeting deprecated/removed API versions; treat any found as supporting evidence feeding the deprecated-API RED determination above (evidence only; the insight/deprecated-API signals in the bands govern the rating)

Rating:

  • 🟢 GREEN: No critical insights AND the deprecated-API/PSP signals were CONFIRMED via successful reads (Insights read succeeded clean AND the PSP list was successfully read — an empty/404 result confirming no PSPs). GREEN requires the GREEN-precondition signals confirmed — if the PSP list returned 403/Forbidden (PSP signal UNKNOWN), GREEN is not awardable even when insights read clean (see the AMBER-with-note cap)
  • 🟡 AMBER: WARNING-level insights present, OR AMBER-with-note: insights read clean but the PSP list returned 403/Forbidden so deprecated-API/PSP usage could not be fully verified — cap at AMBER with the note "no critical insights, but deprecated-API/PSP usage could not be fully verified" (GREEN's PSP precondition is unconfirmed) and record the uncertainty under Investigate Manually. This never raises a band — a confirmed ERROR-insight / deprecated-API RED still stands; it only caps an otherwise-GREEN down to AMBER
  • 🔴 RED: ERROR insights, deprecated APIs actively in use. Confirmed-floor: a confirmed ERROR insight (or confirmed deprecated-API usage) keeps this RED even if a different signal read is forbidden — e.g. a 403 on the PSP list marks only the PSP signal UNKNOWN and never downgrades a confirmed ERROR-insight RED to UNKNOWN.
  • ⬜ UNKNOWN: Insights API not accessible (the Insights read is the sole discriminator and it returned 403/Forbidden or is otherwise unreachable), and no other successfully-read signal (e.g. confirmed deprecated-API usage) already yields a color
  • Evaluation order: assess RED first; if not RED, assess AMBER; otherwise GREEN. Keeps the bands exhaustive and non-overlapping.
  • Scoring authority: this check owns the EKS Cluster Insights / upgrade-readiness signal; check 10.3 defers here and is evidence-only. Note: step 1 filters to the UPGRADE_READINESS category, but 10.3 may defer insight evidence spanning ALL categories (PASSING/WARNING/ERROR); score any such deferred evidence with this check's generic bands (ERROR insight → RED, WARNING insight → AMBER) regardless of category, so no ERROR insight is dropped.

1.4 — Add-on Version Compatibility​

What to check:

  • List all EKS managed add-ons with versions and health
  • Compare installed versions against latest compatible for the cluster version
  • Check for self-managed add-ons in kube-system (Helm releases)

How to check:

  1. List addons → describe each for version, status, health. 403 guard: if listing add-ons returns 403/Forbidden → UNKNOWN (see band). If a per-add-on describe-addon returns 403/Forbidden → mark that add-on's health signal UNKNOWN and do NOT assume healthy — a forbidden health read must not silently skip the health-issue RED.
  2. For each core add-on (vpc-cni, coredns, kube-proxy, aws-ebs-csi-driver), compare installed vs latest compatible
  3. Determine EBS usage (required before aws-ebs-csi-driver can be classified): list PersistentVolumes and PersistentVolumeClaims via the Kubernetes API and check for EBS-backed volumes — CSI provisioner ebs.csi.aws.com, in-tree provisioner kubernetes.io/aws-ebs, in-tree volume source awsElasticBlockStore, or a gp2/gp3 StorageClass. 403 guard: if this PV/PVC read returns 403/Forbidden (or otherwise cannot complete) → mark the EBS-usage signal UNKNOWN; do NOT read a forbidden/failed list as "no EBS PVs found." The favorable absent-and-not-needed → N/A classification for aws-ebs-csi-driver requires a SUCCESSFUL, empty PV/PVC read confirming no EBS-backed volumes exist. When EBS usage is UNKNOWN, an absent aws-ebs-csi-driver may NOT be classified absent-and-not-needed — treat it as absent-and-needed (→ AMBER) or report the aws-ebs-csi-driver state UNKNOWN, never N/A.

Core add-on set: vpc-cni, coredns, kube-proxy, aws-ebs-csi-driver. The first three (vpc-cni, coredns, kube-proxy) are REQUIRED for normal cluster function; aws-ebs-csi-driver is CONDITIONALLY required — needed only when EBS-backed PersistentVolumes/PersistentVolumeClaims are in use. For each core add-on, classify its state as one of: managed-current (EKS Managed, latest or N-1 compatible), managed-behind (EKS Managed but older than N-1), self-managed (present as a Helm/self-managed release rather than an EKS Managed add-on), absent-and-needed (neither managed nor self-managed and the add-on is required — any of the three required add-ons, or aws-ebs-csi-driver when EBS PVs/PVCs exist), absent-and-not-needed (aws-ebs-csi-driver only, when a SUCCESSFUL PV/PVC read confirms no EBS-backed PVs/PVCs exist → N/A, not a finding; if the EBS-usage read was forbidden/failed and is therefore UNKNOWN, this favorable state does NOT apply), health-issue (DEGRADED/failed add-on status via describe-addon), or health-unknown (the add-on is present per ListAddons but its per-add-on describe-addon returned 403/Forbidden, so its version/health could not be read — the add-on's contribution is UNKNOWN; do NOT assume healthy).

Rating:

  • 🟢 GREEN: Every core add-on is either managed-current or absent-and-not-needed (the latter applies only to aws-ebs-csi-driver with no EBS PVs/PVCs) AND every core add-on's health was CONFIRMED via a successful describe-addon (no health-unknown add-on). Equivalently: all required core add-ons are EKS Managed and on latest or N-1 with health confirmed, and any not-needed conditional add-on is legitimately absent. A health-unknown add-on (describe-addon 403) blocks overall GREEN — see the AMBER-with-note cap and the aggregation rule.
  • 🟡 AMBER: At least one core add-on is managed-behind, self-managed, or absent-and-needed (a required add-on — vpc-cni/coredns/kube-proxy, or aws-ebs-csi-driver while EBS volumes are in use — is genuinely not installed), and no add-on has a health issue. A genuinely-absent required CNI add-on (vpc-cni) may also indicate a non-standard/third-party CNI — note that possibility rather than assuming misconfiguration. OR AMBER-with-note (describe-addon 403): every determinable core add-on is managed-current/absent-and-not-needed (would otherwise be GREEN) but at least one core add-on is health-unknown (its per-add-on describe-addon returned 403) so its version/health could not be verified — cap at AMBER with the note "core add-ons look current but 's version/health could not be verified" (can't award overall GREEN with an unverified core add-on) and record the uncertainty under Investigate Manually. This never raises a band — a confirmed health-issue RED, or a confirmed managed-behind/self-managed/absent-and-needed AMBER, on another add-on still sets the band via worst-of; the cap only lowers an otherwise-GREEN to AMBER.
  • 🔴 RED: Any core add-on is in the health-issue state (DEGRADED/failed add-on status via describe-addon). RED is reserved for health issues. Confirmed-floor: a confirmed health-issue RED on one add-on is not downgraded by a 403 on a different add-on's describe/health read (that 403 marks only the other add-on's signal UNKNOWN) — the confirmed RED stands.
  • ⬜ UNKNOWN: The ListAddons call itself returned 403/Forbidden (cannot enumerate add-ons at all), OR every core add-on resolved to health-unknown (each present add-on's describe-addon returned 403) with no determinable add-on left — i.e. the forbidden read is the sole discriminator and no successfully-read add-on already yields a color. A single add-on's describe-addon 403 does NOT force whole-check UNKNOWN when other add-ons are determinable (that path caps at AMBER-with-note per the health-unknown rule — confirmed floor).
  • State→band map (each state → exactly one outcome): managed-current → GREEN · managed-behind → AMBER · self-managed → AMBER · absent-and-needed → AMBER · absent-and-not-needed → N/A for that add-on (no finding; does not block GREEN — requires a successful empty PV/PVC read) · health-issue → RED · health-unknown → per-add-on UNKNOWN (describe-addon 403; contribution undetermined — blocks overall GREEN, does not itself set a worse band). EBS-usage UNKNOWN (forbidden/failed PV/PVC read) with aws-ebs-csi-driver absent → NOT N/A: treat as absent-and-needed (AMBER) or mark that add-on UNKNOWN, so a forbidden read can never silently grant GREEN.
  • Aggregation rule (worst-of over DETERMINABLE add-ons, mirrors 10.2's UNKNOWN-dimension-blocks-GREEN): rank per-add-on bands RED > AMBER > GREEN; managed-current/managed-behind/self-managed/absent-and-needed/health-issue are determinable, absent-and-not-needed is N/A (excluded), and health-unknown is UNKNOWN (a scored add-on whose 403-hidden health could be anything, including RED). (1) If at least one add-on is determinable, the overall band = the worst determinable band, with one guard: a health-unknown add-on blocks overall GREEN — so if the worst determinable band is GREEN but any core add-on is health-unknown, cap overall at AMBER-with-note (name the unverified add-on under Investigate Manually); a confirmed RED (health-issue) or AMBER (managed-behind/self-managed/absent-and-needed) determinable add-on still sets the overall via worst-of (a health-unknown never lowers an already-worse band — confirmed floor). Overall GREEN therefore requires every core add-on determinable AND (managed-current or absent-and-not-needed). (2) If NO add-on is determinable (every present add-on is health-unknown, others N/A), the overall check is UNKNOWN. This yields exactly one overall result for every {ListAddons-outcome × per-add-on-describe-outcome} combination.
  • Evaluation order: assess RED first (any health issue); if not RED, assess AMBER (any managed-behind / self-managed / absent-and-needed); otherwise GREEN. Keeps the bands exhaustive and non-overlapping.
  • Scoring authority: this check owns add-on version compatibility scoring; check 10.1 defers here and is evidence-only.

Key talking point: EKS does NOT auto-update add-ons when you upgrade the control plane. This is the #1 thing customers forget.

1.5 — Upgrade Process Maturity​

What to check (target cluster only):

  • Cluster tags for environment classification (dev, staging, prod)
  • Evidence of IaC-managed upgrades (eksctl, CloudFormation, Terraform tags)

How to check:

  1. Describe the target cluster → check tags for environment indicators (dev/staging/prod) AND IaC-provenance tags (same set as check 2.1: terraform / managed-by / aws:cloudformation:stack-name / eksctl.cluster.k8s.io/* / aws:cdk:*). 403 guard: if the describe/tags read returns 403/Forbidden → UNKNOWN; do NOT treat a forbidden tag read as "no environment tag" (that would be a false AMBER). The "no environment tag" AMBER requires a SUCCESSFUL tags read that returns no environment-classification tag.

Do NOT list or describe other clusters in the account. Stay within the scope of the target cluster.

Rating:

  • 🟢 GREEN: Cluster has an environment tag whose value is a clear dev/staging/prod classification
  • 🟡 AMBER: An environment tag is present but its value is not a clear dev/staging/prod classification (ambiguous or non-standard value), OR environment intent is split across conflicting tag keys, OR no environment-classification tag is present at all. An absent environment tag is a governance/hygiene improvement opportunity — not a critical availability or security gap — so it rates AMBER, not RED.
  • ⬜ UNKNOWN: The cluster tags cannot be read (describe/tags returns 403/Forbidden) — this is the sole decidable UNKNOWN trigger for 1.5
  • Evaluation order: assess AMBER first (any of: ambiguous/non-standard environment value, conflicting environment tag keys, or no environment tag at all); otherwise GREEN. (1.5 has no RED band — no environment-tagging state rises to Critical; a forbidden tags read is UNKNOWN.)
  • Scoring authority: this check owns environment-classification and upgrade-process-maturity scoring; the IaC-provenance tag signal (terraform/CloudFormation/eksctl/CDK) is owned by check 2.1 — 1.5 gathers those tags as evidence but defers their scoring there, so an untagged cluster is not double-penalized.

Investigate manually: Upgrade history cannot be determined from the API alone — confirm with the user. Do you have a documented upgrade runbook? Do you test upgrades on a non-prod cluster first? Can more than one person execute an upgrade?