Skip to main content
Source

This page is generated from skills/ecs-operation-review/references/scoring-rubric.md. Edit the source, not this page.

Scoring Rubric & Rating Rules

Ratings​

RatingEmojiMeaning
GREEN🟢Fully implemented — matches Amazon ECS best practices
AMBER🟡Partial or inconsistent — improvement opportunity
RED🔴Not implemented or significant gap — action needed
UNKNOWN⬜Cannot be determined from estate data — investigate manually
N/A⚪Check does not apply to this estate's compute model / architecture — see below

N/A deliberately uses a different emoji (⚪) from UNKNOWN (⬜) so the two states stay visually distinct in every table.

N/A vs UNKNOWN (distinct states)​

N/A and UNKNOWN are different and must not be conflated:

  • UNKNOWN = the check applies but the data could not be obtained (permission denied, timeout, ambiguous evidence). UNKNOWN items go on the "Investigate Manually" list.
  • N/A = the check does not apply to this estate by design — e.g., an EC2-ASG-only check on a Fargate-only estate, GuardDuty Runtime Monitoring on a Managed-Instances-only estate (7.4), or a Windows/ECS-Anywhere/Express-Mode workload where a Linux/Fargate assumption doesn't hold. State the reason inline.

N/A items are excluded from BOTH the maturity-score table AND the "Investigate Manually" list — there is nothing to investigate and nothing to score. They may be listed in a short "Not applicable (with reason)" note if useful, but they never affect counts or percentages. Checks with a dedicated "⚪ N/A:" branch in their reference files: 1.2, 1.3, 1.4, 1.5, 1.6, 2.5, 2.6, 4.1, 5.1, 5.2, 5.3, 6.5, 7.4, 8.3. Any other check may still be marked N/A when a standing scope limitation applies (Express Mode, Windows, ECS Anywhere — see SKILL.md, Scope & limitations); state the reason inline.

Rules​

  • Rate only on observed evidence. If a check returns no data, times out, or is denied by permissions, mark UNKNOWN — never assume a GREEN or a RED.
  • One item, one rating. Each check produces exactly one rating; do not average a section into a single score.
  • Blast-radius priority. When ordering findings, rank by category: security > availability > cost. Within a category, cluster-wide/estate-wide issues rank above single-service issues.
  • Every RED needs an action. A RED finding must have a specific, actionable recommendation with a cited AWS doc URL from report-generation.md.
  • Estate scope. When assessing multiple clusters/services, rate per-resource and roll up: if any production service in a domain is RED, the domain's headline for that cluster is RED. Note which resource drives the rating.
  • Production vs non-production. If tags (Environment, env) or naming indicate non-production, an item that would be RED in production may be AMBER — state the assumption explicitly and list it under "Investigate manually" if the environment class is uncertain.

Maturity score​

  • Total check count (anchor): the full assessment comprises 44 checks across Sections 01–08, of which 43 are scorable (3.3 is a non-scored pointer to 6.2). Check 8.5 is UNKNOWN by design. If a future revision adds or removes a check, update this line so drift is detectable.
  • Count GREEN, AMBER, RED, UNKNOWN across all rated items. N/A items are not counted at all (see the N/A vs UNKNOWN section).
  • Percentages exclude UNKNOWN (and N/A) from the denominator.
  • Always report coverage alongside every percentage. A percentage computed over a handful of assessable items while most are UNKNOWN is misleading (e.g., "5 GREEN + 30 UNKNOWN" is not "100% GREEN"). Next to each maturity percentage, state "N of M items assessable (X% coverage)", where N = GREEN+AMBER+RED and M = all items except N/A.
  • Do NOT quote a headline maturity percentage when UNKNOWN exceeds ~25% of assessable-plus-unknown items. Instead lead with the coverage figure and state the actual, observed cause(s) of the UNKNOWNs — e.g., denied API permissions, sections that were not run, or evidence the APIs cannot expose — rather than assuming a permissions or scope problem. This prevents a low-coverage run from reporting a flattering score, and prevents blaming the customer's credentials for gaps that have another cause. (Note: check 8.5 — runbooks/on-call/PIR — is UNKNOWN by design, so it always contributes to the UNKNOWN count; factor that in when judging the 25% threshold. Also confirm no not-applicable items were mistakenly booked as UNKNOWN — they belong under N/A and never count here.)

Cross-domain duplicate checks (score once)​

Some best practices are visible from two domains. To avoid inflating the maturity denominator by counting one control twice, each is scored in exactly one section and cross-referenced (not re-rated) from the other:

ControlScored inReferenced (not scored) from
Container has no log driver / log routing / stream-prefix / delivery mode6.23.3
Task-role least privilege, over-broad policy, execution-role-reused-as-task-role7.13.4
Deployment-failure alerting (SERVICE_DEPLOYMENT_FAILED)4.56.4
RESOURCE:ENI placement failures / ENI density2.61.3
Fargate task-retirement exposure (platform-version currency + retirement absorption)8.4— (replica count, AZ spread, and deploy bounds are scored in 4.4 / 5.4 / 5.5; 8.4 cross-references those ratings as inputs and scores only the retirement-specific angle)

When building the consolidated finding list, include each of these once, under its scoring section only.

Consistency contract (MANDATORY — canonical)​

This section is the canonical statement of the consistency contract. The "Consistency Checks" block in report-generation.md is an operational checklist that implements these rules; if the two ever appear to differ, this section wins.

  1. Ratings are consistent everywhere. If 04.1 is RED in the findings table, it is RED in the executive summary, prioritized actions, and quick wins — no drift.
  2. Prioritized Actions reference the finding ID. Write "04.1 — Deployment Circuit Breaker 🔴", not just "Enable circuit breaker".
  3. Every RED appears in Critical or Important. Every AMBER appears in Important or Quick Wins. Nothing rated RED/AMBER is missing from Prioritized Actions.
  4. Executive Summary matches the findings tables. Do not call an AMBER a "critical gap", and do not omit a RED.
  5. One row per finding, with shared root causes annotated. Each finding keeps its own row, context, action, and references — do not merge two findings into one row. But when several rows trace to one root cause (e.g., a single-replica, single-AZ service with no autoscaling legitimately surfaces at 4.4, 5.4, 5.5, and 8.4), annotate each row with the shared root cause and the group size (e.g., "Related: 4 findings, 1 root cause — see 5.4") so one fix is not read as four independent problems. Scoring stays separate; only the presentation is grouped.

Section index​

#SectionReference file
01Clusters & Capacitycluster-capacity.md
02Networkingnetworking.md
03Task Definitionstask-definitions.md
04Services & Deployment Safetyservices-deployment.md
05Service Health & Autoscalingservice-health-scaling.md
06Observabilityobservability.md
07Security Posturesecurity-posture.md
08Operational Processesoperational-processes.md
—Report generationreport-generation.md