Adr 021 lambda microvms compute backend

ADR-021: AWS Lambda MicroVMs as a third ComputeStrategy backend

Number: candidate ADR-021 (ADR-020 is the highest accepted on main; ADR-018 is claimed by open PR #548, ADR-019 by open PR #663). Numbers are never reused. If a lower number frees before merge, renumber and coordinate with those PRs.

Status: proposed Date: 2026-07-29

Context

ABCA selects a per-repo compute backend through the Blueprint’s compute_type field (cdk/src/handlers/shared/repo-config.ts). Two backends exist today, resolved by resolveComputeStrategy (cdk/src/handlers/shared/compute-strategy.ts) behind a uniform ComputeStrategy interface (startSession / pollSession / stopSession):

• AgentCore Runtime (agentcore, default) — managed Firecracker MicroVM per session. Invoked via InvokeAgentRuntime; liveness is inferred from agent heartbeats in DynamoDB plus the FastAPI /ping endpoint (agent/src/server.py) — the strategy’s pollSession is a stub that always reports running. Constraints: 2 GB image limit, no substrate-level suspend API exposed to the orchestrator.
• ECS on Fargate (ecs) — always-on Fargate task (16 vCPU / 120 GB, ARM64) for repos that exceed AgentCore’s limits (#596). Invoked via RunTask in batch mode (bypassing the HTTP server); liveness via DescribeTasks. No suspend — an idle task blocked on an approval wait burns full compute the whole time.

AWS Lambda MicroVMs (launched 2026-06-22) is a serverless Firecracker sandbox primitive that AWS positions explicitly for AI coding agents: VM-level isolation, snapshot-based near-instant launch, suspend/resume with full memory + disk state preserved (compute charges stop while suspended), a dedicated JWE-authenticated HTTPS endpoint per instance, lifecycle hooks (/run, /suspend, /resume, /terminate), and up to 8 hours per session. It is not classic Lambda: the 15-minute function cap does not apply, and COMPUTE.md’s “Lambda: poor fit” verdict refers to functions, not MicroVMs.

#645 proposes adding Lambda MicroVMs as a third ComputeStrategy. The fit is strong but not free — pre-implementation review of the service’s lifecycle model surfaced real design tensions this ADR resolves:

Capability comparison (delta rows only — full matrix in COMPUTE.md)

	AgentCore Runtime	ECS on Fargate	Lambda MicroVMs
Isolation	MicroVM (managed)	Task-level (Firecracker)	MicroVM (Firecracker)
Max duration	8 h	No cap	8 h (running + suspended) — verified (L-B430C318 = 8 hours)
Suspend/resume	No orchestrator-visible API	No	Yes — explicit API + idle policy, state preserved, no compute charge while suspended. Verified: suspend reaches SUSPENDED in ~1 s with no idlePolicy; resume restores RUNNING in ~1 s with microvmId and endpoint byte-identical
Resources	AgentCore-managed	16 vCPU / 120 GB / 20–200 GB disk	Baseline 8 GiB RAM / 4 vCPU, auto-scaling to a 32 GiB / 16 vCPU peak; 32 GB disk. minimumMemoryInMiB configures the BASELINE (max 8,192 MiB); the service scales vertically on demand — capacity is baseline-priced with 4× burst headroom
Packaging	ECR image ≤ 2 GB	ECR image, no hard cap	Zip + Dockerfile in S3 → service-built snapshot image (versioned, storage billed)
Invocation	InvokeAgentRuntime (SigV4)	RunTask + container overrides	RunMicrovm (image ARN required — a bare name is rejected) → dedicated HTTPS endpoint + JWE token (CreateMicrovmAuthToken, ≤ 60 min TTL)
Liveness	Agent heartbeat + /ping	DescribeTasks	MicroVM state (RUNNING / SUSPENDED / TERMINATED) via control-plane API
Session storage	/mnt/workspace FUSE (no flock())	Ephemeral disk	Native disk in snapshot — survives suspend/resume, flock() works
Architecture	ARM64	ARM64	ARM64 (Graviton)
Regions (launch)	Broad	Broad	5 (us-east-1/2, us-west-2, eu-west-1, ap-northeast-1)

Rows marked verified were discharged empirically on 2026-07-31 (us-east-1); see docs/verification/645-p1-lambda-microvm-runbook.md. Two originally documented constants were refuted by that run and are corrected throughout this ADR: the runHookPayload cap (16 KB → 4 KB) and the claim that RunMicrovm accepts a bare image name (it does not).

Source hierarchy for service facts. Where sources disagree, the higher tier wins, and the tier is recorded next to the claim:

1. Live boundary probe — a request the service actually accepted or rejected in this account/Region. Strongest, and the only tier that can refute the others. (Both corrections above came from here.)
2. Modeled constraints — the CLI/SDK request schema and enumerated allowed values (--generate-cli-skeleton, ARM_64, ENABLED|DISABLED). Authoritative about request shape; silent about runtime behaviour.
3. Service developer guide — including its sizing/scaling tables. Authoritative about semantics a probe cannot see, which is exactly how the memory row was fixed: a probe can only show that 32,768 MiB is rejected as a minimumMemoryInMiB value; only the guide explains that the field is a BASELINE and that the service scales to a 32 GiB peak on its own. A boundary probe is the strongest evidence about a boundary and says nothing about what the boundary means.
4. SDK docstrings — generated, and demonstrably stale here: runHookPayload documents “Maximum: 16,384 bytes” against an enforced 4,096.
5. Launch blogs / skills / toolkit material — orientation only; never load-bearing on its own.

Omitted API fields mean “service default”, never “none”. Two live findings drove this rule: leaving ingressNetworkConnectors unset attaches a PUBLIC HTTP_INGRESS connector, and leaving /ready out of hooks makes the image un-creatable once any lifecycle hook is enabled. So for any security-relevant field, the desired posture must be requested explicitly and the test must assert the outcome (the ARN present in the request, the hook enabled) rather than the omission (expect(field).toBeUndefined()) — an omission assertion passes just as happily when the service is silently choosing something wider.

On the memory row specifically: CreateMicrovmImage enumerates the baseline sizes a base image supports — [512, 1024, 2048, 4096, 8192] MiB for al2023-1 — and rejects anything else, which is why the construct validates against that list at synth. The 32 GiB / 16 vCPU peak is reached by the service’s own vertical scaling, not by asking for it. The account memory quota (L-CD1C0CC4, 1024 GB, “burst up to 4×”) is an aggregate across MicroVMs, not a per-VM limit; note that concurrency arithmetic should be done against the PEAK, not the baseline, since that is what a busy fleet can actually consume.

Design tensions the strategy must resolve

1. Idle detection is inbound-traffic-based; the ABCA agent is outbound-only. MicroVM idle policies suspend when no traffic arrives at the endpoint. A busy agent running a 40-minute build receives no inbound traffic and would be suspended mid-work by a naive idle policy. Conversely, “no inbound traffic” is the agent’s normal state.
2. No self-suspend. The agent cannot suspend its own MicroVM from inside; only an external SuspendMicrovm call can. Suspend decisions must be owned by the orchestrator — which aligns with the unified liveness model proposed in #491.
3. Snapshots bake state. The image snapshot is captured once at build time; every MicroVM resumes from it. Secrets, tokens, and per-task identity must arrive at run time (runHookPayload, ≤ 4 KB, or fetched in the /run hook), never at image build. CSPRNGs must be reseeded on /run and /resume.
4. Auth tokens are short-lived. JWE tokens max out at 60 minutes; any orchestrator→agent HTTP interaction over the endpoint needs token refresh, unlike AgentCore’s SigV4 invoke or ECS’s no-endpoint model.
5. Identity delta — narrower than it looks. Most AgentCore services ABCA uses are standalone and substrate-portable: Memory is already consumed from ECS via an IAM grant plus MEMORY_ID (EcsAgentCluster), and Gateway (ADR-019) is portable by design (SigV4 inbound). The genuinely Runtime-coupled piece is the workload-access-token delivery mechanism (runtimeUserId → WorkloadAccessToken request header → BedrockAgentCoreContext, used by resolve_linear_api_token()), which has no MicroVM equivalent. The ECS backend already lives with this delta (env-var token delivery); MicroVMs inherit the same posture until the pluggable identity work (#249, ADR-016) redesigns the seam.
6. The service’s defaults are not our posture. Two of them, both discovered live: RunMicrovm attaches a public HTTP_INGRESS connector (and mints a public *.lambda-microvm.<region>.on.aws endpoint) when ingressNetworkConnectors is omitted, and CreateMicrovmImage requires the /ready hook whenever any lifecycle hook is enabled. Neither posture can be reached by leaving a field out — each needs an explicit control (see sub-decisions 3 and 4).

Decision

Adopt AWS Lambda MicroVMs as a third, opt-in ComputeStrategy backend named lambda-microvm, selected per repo via Blueprint compute_type. AgentCore remains the default. Five sub-decisions:

1. Strategy shape: extend the interface with mandatory suspend/resume

ComputeType widens to 'agentcore' | 'ecs' | 'lambda-microvm' (mirrored in cli/src/types.ts and the CLI’s inline unions). SessionHandle gains a { strategyType: 'lambda-microvm', microvmId, endpoint } variant — microvmId because every lifecycle API (suspend-microvm, resume-microvm, terminate-microvm, get-microvm, and create-microvm-auth-token — the latter not called in P1–P3, see sub-decision 3) takes only the MicroVM identifier, and endpoint because it is per-session (minted by RunMicrovm) and required for any orchestrator→agent HTTP interaction. Note the naming seam: the handle field is microvmId (matching RunMicrovmResponse.microvmId), while the request key on every lifecycle command is microvmIdentifier — the strategy is the only place that translates between the two. The image ARN is deliberately not in the handle: like the ECS task definition ARN, it is deployment-time configuration consumed by startSession (from construct-injected environment) and recorded in the session-start log entry for diagnostics, not per-session lifecycle state.

A second, sharper naming seam: imageIdentifier must be an ARN. The name suggests a bare image name is acceptable — create-microvm-image --name takes one, and this ADR originally assumed run-microvm --image-identifier would too. It does not: a bare name is rejected with ValidationException: Malformed ARN - doesn't start with 'arn:', and so is list-microvm-image-builds --image-identifier <name> (Invalid ARN format). The construct therefore resolves an operator-supplied name to its exact arn:${Partition}:lambda:${Region}:${Account}:microvm-image:${Name} ARN once — the same value it scopes the lifecycle IAM grant to — and injects THAT as MICROVM_IMAGE_IDENTIFIER. One derivation, two consumers, so a request field and an IAM resource can never disagree. The strategy validates the invariant and fails fast with the remedy, because the service’s own error names neither the env var nor the fix.

The ComputeStrategy interface gains mandatory suspendSession(handle) / resumeSession(handle) methods returning a typed result ({ supported: false } | { supported: true }-shaped, exact type at implementation time); the widening lands in P3 (see sub-decision 5), in one commit across all three strategies. Mandatory-with-explicit-stub is the codebase idiom, not optional methods: no behavioral interface in the codebase has an optional method, AgentCoreComputeStrategy.pollSession is already a mandatory explicit stub rather than an optional member, and the exhaustive-never switch culture means a fourth backend must make a compile-checked decision about its suspend semantics instead of silently falling through a strategy.suspendSession?.() feature-detection. The agentcore and ecs strategies return unsupported (not a silent success — a suspend that silently no-ops would let the orchestrator believe compute billing stopped when it did not); the orchestrator gates its suspend policy on the typed response, consistent with how pollTaskStatus already branches explicitly on computeType.

Poll semantics — the strategy reports, the orchestrator interprets. pollSession(handle) receives only the session handle and cannot see task state, so the health rules must live where the DynamoDB status lives. SessionStatus gains a 'suspended' variant; the strategy maps GetMicrovm state mechanically and the orchestrator cross-references against the task row — the same division of labor finalPollState already uses for ECS (substrate stopped + non-terminal DynamoDB status → failed) and pollTaskStatus uses for agentcore heartbeats: substrate suspended + task AWAITING_APPROVAL is healthy (orchestrator-intended suspend); suspended with any other task status is an anomaly to surface, not fail-fast; substrate terminal + non-terminal task status → classify failed.

The service’s MicrovmState enum has six members, not three, so the mapping is stated exhaustively (one line of rationale each, mirrored in the strategy’s doc comment):

MicrovmState	SessionStatus	Why
PENDING	running	Still booting; the same way ECS’s PENDING/PROVISIONING map to running.
RUNNING	running	—
SUSPENDING	suspended	Already on its way to frozen; reporting running would tell the orchestrator compute is still progressing when it is not. Both suspend states land on a report the orchestrator treats as benign-or-anomalous depending on task status, never as failure. Never observable in practice — suspend reached SUSPENDED in under 1 s live — so it is mapped for completeness and nothing may wait for it.
SUSPENDED	suspended	—
TERMINATING	completed	Terminal-bound and carries no exit code, so “the substrate is gone” is all the strategy can honestly say.
TERMINATED	completed	Success vs failure is the orchestrator’s call — it cross-references the DynamoDB status. This is the load-bearing terminal signal, not NotFound (see below).
unrecognized	running	A future service enum addition must never fail a healthy task; the strategy warns and keeps polling.

GetMicrovm ResourceNotFoundException → completed, but as a LATE fallback. This deliberately diverges from ecs-strategy, where DescribeTasks returning no task maps to failed. ECS keeps stopped tasks describable for roughly an hour, so a missing task there really is anomalous; a MicroVM is eventually reaped from the control plane by design, so failed would fail tasks that finished cleanly.

What the live run corrected is the timing, not the mapping. NotFound is not the near-term terminal signal: a terminated MicroVM reported TERMINATING at +1 s, TERMINATED at +3 s, and was still TERMINATED ~10 minutes later and at every subsequent checkpoint — ResourceNotFoundException was never observed in that window. So the branch that actually fires in practice is TERMINATED → completed in the table above; the NotFound rule covers only a VM reaped after a long gap (a poll resumed after a crash, a stranded-task reconciler sweep). Both are required and neither substitutes for the other: without the TERMINATED row the orchestrator would poll a finished VM until its safety net fired, and without the NotFound rule a late sweep would classify a cleanly-finished task as a poll error.

The divergence is safe because it does not weaken detection: the orchestrator still fails the task when a terminal report lands while the DynamoDB status is non-terminal, so a genuine mid-run disappearance is caught — it simply receives the substrate-failure classification instead of a misleading poll error. Because that cross-check acts on a status read earlier in the same poll cycle, the orchestrator re-reads the task row before failing (the normal shutdown order is “agent writes terminal status → agent exits → VM terminates”, which a stale read would otherwise turn into a spurious failure); ECS buys the same protection with a five-consecutive-poll patience counter instead.

Neither the mapping nor the NotFound rule is a health decision: both are mechanical restatements of substrate state, which is what keeps the “strategy reports, orchestrator interprets” split intact.

Normative requirements (EARS, per ADR-020):

• When a task’s Blueprint sets compute_type: 'lambda-microvm', the orchestrator shall resolve the LambdaMicrovmComputeStrategy via resolveComputeStrategy.
• When startSession is invoked, the strategy shall call RunMicrovm with maximumDurationInSeconds set to 28 800 (the service maximum, matching AgentCore’s 8-hour session cap and sitting inside the orchestrator’s ~8.5 h safety-net poll window).
• When startSession is invoked, the strategy shall pass a fully-qualified MicroVM image ARN as imageIdentifier.
• If the configured image identifier is not an ARN, then the strategy shall fail the session start with an error naming the environment variable and the redeploy remedy, before performing any AWS call.
• When startSession returns, the orchestrator shall persist the MicroVM handle (microvmId, endpoint) in the task row’s compute_metadata (the field cancel-task.ts already reads ECS handles from).
• The strategy shall omit idlePolicy on every RunMicrovm call, in every phase.
• The orchestrator shall be the sole initiator of suspension, via suspendSession.
• When pollSession observes MicroVM state SUSPENDED or SUSPENDING, the strategy shall report suspended without interpreting task state.
• When pollSession observes MicroVM state TERMINATED or TERMINATING, the strategy shall report completed (the observable terminal state persists for at least ~10 minutes, so completed shall not depend on the MicroVM being reaped).
• When pollSession observes a MicroVM state it does not recognize, the strategy shall report running.
• If GetMicrovm reports that the MicroVM does not exist, then the strategy shall report completed.
• If the strategy reports a terminal substrate state while the task’s DynamoDB status is non-terminal, then the orchestrator shall re-read the task row and, if it is still non-terminal, classify the task as failed with a substrate-failure remedy.
• If the strategy reports suspended while the task’s DynamoDB status is not AWAITING_APPROVAL, then the orchestrator shall surface an anomaly event and shall not fail-fast the task.
• If suspendSession or resumeSession is invoked on a strategy that does not support suspension, then the strategy shall return an explicit unsupported result.
• When the agent process reaches a terminal state, the agent shall exit.
• When the orchestrator finalizes a lambda-microvm task, the orchestrator shall call terminate-microvm (termination shall not rely on any substrate timeout, and shall not rely on the MicroVM self-terminating — it does not).

On the omitted idlePolicy: if the block is present all three fields are required, so omission is the unambiguous disabled state the invariant test asserts. This deliberately forgoes suspendedDurationSeconds — it lives inside idlePolicy and cannot be set without re-enabling the traffic-idle machinery — so the suspended-state bound is maximumDurationInSeconds plus orchestrator termination and the stranded-approval reconciler (see sub-decision 2). A tighter substrate-level suspended-TTL remains available later as an additive idlePolicy change if operators want it. On the fixed maximumDurationInSeconds: no wall-clock task budget exists in the platform (budgets are max_turns / max_budget_usd), so the value is parity with AgentCore’s 8 h cap rather than derived policy; a Blueprint override can be added later if a real need appears.

2. Lifecycle: suspend/resume reconciled with the agent-owned approval poll

The headline economic win is suspend during HITL approval waits (Cedar approval gates, CEDAR_HITL_GATES.md): while a task waits on a human decision, the MicroVM is suspended (compute charges stop; memory/disk state — cloned repo, warm build caches — is preserved) and resumed when the decision lands. Under Cedar decision #6 the approval window is bounded (default 300 s, ceiling 1 h, timeout → deny), so the saving per gate is bounded at ~1 h of compute — real at 16 vCPU, and it makes any future extension of gate ceilings (the off-hours posture §14.8 deliberately defers) cheap on this backend.

The handshake must respect the existing approval mechanics: the agent discovers decisions itself by polling DynamoDB (_poll_for_decision, monotonic timeout), the approve/deny Lambda writes only the decision rows, and AWAITING_APPROVAL holds the concurrency slot (Cedar decision #7). Nothing “delivers” an approval to the agent, and suspension freezes the agent’s monotonic clock — so the design is:

• Suspend — orchestrator-owned. The orchestrator’s durable poll observes AWAITING_APPROVAL on a lambda-microvm task and calls suspendSession after a grace period, and only when the gate’s remaining window exceeds grace + resume overhead (suspending a 30 s gate is pure loss). Suspend is a policy decision on a poll observation, not a user action.

• Resume — inline in the approve/deny Lambdas, orchestrator poll as backstop. After the transactional decision write commits, ApproveTaskFn/DenyTaskFn load the MicroVM handle from the task row’s compute_metadata (persisted at session start — the same field cancel-task.ts reads ECS handles from) via a post-commit strongly-consistent GetItem, then call resumeSession best-effort: on failure they log a warning and write a resume-orphan task event; the decision response never fails on a compute error (the decision row is already durable). The orchestrator poll reconciles: decision row present + MicroVM still SUSPENDED → retry resume (idempotent).

  Why inline rather than poll-only — codebase precedent: resume-on-approve is structurally identical to task cancellation — a user-initiated, latency-sensitive action whose purpose is an immediate compute-lifecycle side effect. cancel-task.ts already resolves this exact tension: the API-plane handler invokes ECS StopTask / AgentCore StopRuntimeSession inline, best-effort (a failed stop logs a warning and the state transition stands; a task_cancel_compute_orphan event is written when no stoppable compute handle exists, reason: missing_runtime_handle) — with the conditional IAM wired in task-api.ts. The resume path goes one step further than the precedent by also writing the orphan event on failed resume calls, because a failed resume strands a suspended VM awaiting a decision — a stronger liveness consequence than a failed stop of an already-cancelled task. The alternative (orchestrator-poll-only resume) preserves single-owner lifecycle purity but pays up to a full poll interval (~30 s) of latency on every approval, and the purity argument was already litigated and declined for cancel. approve-task.ts is deliberately minimal today (security-critical ownership comparison, Cedar finding #6); the resume call is therefore added after the transaction commits, cannot alter the decision outcome, and carries one conditional lambda:ResumeMicrovm grant — the same blast-radius trade the cancel handler accepted in review.

• Timeout under freeze — the agent re-bases on the wall clock it already owns. The agent’s monotonic gate timer freezes while suspended, so resuming near the deadline is not enough: the frozen timer would still hold its remaining budget and fire the deny minutes after the user-visible window — colliding with the approval row’s TTL (created_at + timeout_s + 120s) and triggering the “row reaped → stranded” fallback on a healthy gate. Instead, the gate expires at min(monotonic budget, created_at + timeout_s), evaluated on each poll iteration and on /resume. This is not a new principle: Cedar decision #6 is already “min wins” for timeouts, the wall-clock deadline is already durable in the approval row the agent itself writes (created_at is in the agent’s own clock domain — no skew), and §13.12’s late-approval race fix already establishes that the durable row is authoritative over the agent’s local timer. Deny authority stays agent-side (the conditional TIMED_OUT write + ConsistentRead re-read race protection is untouched); the orchestrator’s resume at deadline − margin is purely the wake-up mechanism, with no correctness role.

• Backstops, not mechanisms. maximumDurationInSeconds (mandatory on every RunMicrovm, pinned at 28 800 s — see sub-decision 1) is the substrate kill switch bounding running and suspended time; the orchestrator’s finalization terminate-microvm is the active cleanup path; the stranded-approval reconciler retains its role for orphaned waits. No idlePolicy-based bound is used in any phase — see sub-decision 1’s omit-idlePolicy invariant. The active terminate is mandatory, not belt-and-braces: a MicroVM whose hook never ran reached RUNNING in 12 s and stayed RUNNING with no stateReason through every checkpoint (live). Nothing self-terminates on this substrate, so nothing cleans up — a leaked VM bills until the 8 h cap.

• Concurrency slot stays held during suspend. Cedar decision #7’s rationale (“container alive, consuming memory”) weakens under suspend, and the harder replacement rationale — “AWS counts SUSPENDED MicroVMs toward the account memory quota, so releasing ABCA’s slot would not free real capacity” — is undischarged: the suspended VM stayed in list-microvms at every checkpoint, but that only proves listed. L-CD1C0CC4 (1024 GB, account-scoped) exposes no UsageMetric, AWS/Usage carries only CallCount per API, and no MicroVM memory metric exists in any namespace, so consumption is not observable safely — proving it would need a large concurrent fleet. The conclusion (hold the slot) stands as the conservative choice, not as a verified fact. Size the arithmetic against the 32 GiB peak rather than the 8 GiB baseline: a busy fleet scales up, so peak is what actually competes for the account quota.

The agent’s /suspend hook flushes progress events (durable writes before returning 200, within the 60 s hook budget); /resume reseeds CSPRNGs and refreshes cached credentials.

Normative requirements (EARS):

• While a lambda-microvm task is in AWAITING_APPROVAL and the gate’s remaining window exceeds the configured grace period plus resume overhead, the orchestrator shall call suspendSession after the grace period.
• When the approve or deny Lambda commits a decision for a lambda-microvm task, the Lambda shall load the MicroVM handle from compute_metadata and call resumeSession best-effort.
• If the inline resume fails, then the Lambda shall record a resume-orphan task event and shall still return the decision outcome.
• While a decision row exists and the MicroVM remains SUSPENDED, the orchestrator shall retry resumeSession.
• While a lambda-microvm task waits on an approval gate, the agent shall evaluate gate expiry as the earlier of its monotonic budget and the row’s wall-clock deadline (created_at + timeout_s), on each poll iteration and on /resume.
• If gate expiry is reached without a decision, then the agent shall deny.
• If no decision arrives by the gate’s wall-clock deadline minus the resume margin, then the orchestrator shall resume the MicroVM so the agent can evaluate expiry and fire the deny agent-side.

3. Packaging: same agent image source, new build path

The existing agent container (agent/ Dockerfile, already ARM64) is repackaged as a zip + Dockerfile artifact in S3 and built into a versioned MicrovmImage via CreateMicrovmImage. The agent runs its existing FastAPI server (agent/src/server.py) — the MicroVM path uses the HTTP entrypoint like AgentCore, not ECS’s batch bypass — plus the runtime lifecycle hooks (/run, /suspend, /resume, /terminate) and the /ready + /validate build hooks, all on the same port the server already listens on (8080, declared as the image’s hooks.port). Runtime hooks are fast-notification only (1–60 s): /run validates the payload and starts the pipeline asynchronously, mirroring how the agent loop already runs in a background thread behind /ping on AgentCore.

Hook phasing — corrected: /ready + /run are both P1. The original plan split declaring a hook from serving it, putting /run’s declaration in P1 and its implementation in P2. Live verification proved that split is not a reachable service state, on two independent counts:

• CreateMicrovmImage rejects an image that enables any lifecycle hook without /ready: “The ready (/ready) MicroVM image hook must be enabled when any MicroVM lifecycle hook (run, resume, suspend, or terminate) is enabled.” So a P1 image declaring only /run is not creatable.
• With /ready added but unserved, both chipset builds fail: “Ready hook check failed: the application returned a client error (HTTP 4xx) response.” So a declared hook must be served in the same phase.
• And an image with no hooks at all — the only other creatable shape — cannot receive a payload: “The run hook must be enabled in the MicroVM image to pass the run hook payload.” So deferring hooks entirely also defers the whole payload-delivery channel.

The phasing is therefore:

Hook	Declared by	Served by the agent	Notes
/ready	P1 (construct sets hooks.microvmImageHooks.ready)	P1	MANDATORY, not a quality nicety — see above. A 200 once uvicorn is bound is the whole P1 contract: it also proves server imported cleanly (pulling in pipeline → runner → the policy engine), so a missing policy file fails the BUILD instead of the first task.
/run	P1 (construct sets hooks.microvmHooks.run)	P1	The payload-delivery channel. Must be served in P1 because /ready forces hooks to exist at all, and a hook-less image cannot accept runHookPayload.
/validate	P2	P2	Build-time snapshot-quality hook. Still deliberately NOT declared: a /validate that 404s or reports failure fails every image build. Deeper warm-up assertions (Bedrock reachability, Memory access, tool availability) belong here.
/suspend, /resume, /terminate	P3 (suspend/resume), P2 (/terminate)	P3 / P2	Declaring a runtime hook the agent does not serve fails the corresponding lifecycle transition, so each is declared only in the phase that implements it. P1 termination is the orchestrator’s TerminateMicrovm, which needs no in-guest cooperation.

Consequence to state plainly, replacing the original “a P1-built MicroVM image is not runnable end to end”: a P1 image is creatable, launchable and payload-deliverable, but carries no smoke-parity guarantee. P1 delivers the strategy, the construct, the roles/buckets/connectors, the image resource, the packaging script, and the /ready + /run endpoints — so a lambda-microvm task can start a MicroVM and hand it a payload. What P1 has not established is anything P2 owns: AgentCore Memory grants and MEMORY_ID delivery, the agent’s non-secret env parity inside the snapshot, egress specifics from a running MicroVM, and heartbeat/progress behaviour end to end. No clone → change → PR run has happened on this substrate. P2 (“smoke parity”) is the phase that closes that gap. The construct and the packaging script both surface exactly this at synth/run time (abca:microvm-image-p1-smoke-unverified) so an operator cannot mistake a launchable substrate for a verified one.

Payload delivery reuses the ECS strategy’s S3-pointer pattern, adapted to runHookPayload (≤ 4 KB — measured, see below): payloads that fit ride inline; the rest are uploaded by the strategy to a platform payload bucket (the ECS payload bucket pattern in ecs-agent-cluster.ts: orchestrator write access, compute-role read-only scoped to the bucket, lifecycle expiry on objects) with only the S3 URI in runHookPayload — the MicroVM execution role holds the read grant, exactly as the ECS task role does today.

The cap is 4 096 bytes, not the 16 384 the SDK documents. Measured exactly: 4 096 passes, 4 097 is rejected with “Value at ‘runHookPayload’ failed to satisfy constraint: Member must have length less than or equal to 4096”. Two consequences follow. First, the original threshold would have inlined every envelope between 4 097 and 16 384 bytes and had the service reject all of them. Second, and more structurally: the S3-pointer path is now the dominant one, and inline is the exception. A hydrated task payload (prompt + issue thread + repo context) essentially always exceeds 4 KB, so “small payloads ride inline” describes tiny repo-less prompts rather than the common case. The payload bucket is therefore not a rarely-exercised overflow valve but a required part of every normal task, which raises its lifecycle rule (MICROVM_PAYLOAD_TTL_DAYS) and the execution role’s read grant from edge-case plumbing to load-bearing.

No orchestrator→agent HTTP path exists in P1–P3: payload arrives through the /run hook, all agent work is outbound, and therefore no JWE auth tokens are minted at all — token minting (and its ≤ 60 min TTL refresh problem) is deferred until a real consumer exists (e.g. operator shell access, #391). The endpoint stays in the SessionHandle because it is genuinely per-session state that becomes load-bearing the day such a consumer appears. But note the service does not agree by default: omitting ingressNetworkConnectors on RunMicrovm attaches a public HTTP_INGRESS connector, so the strategy passes the Lambda-managed NO_INGRESS connector explicitly on every launch (see sub-decision 4’s security table).

Constraint accepted: the configured baseline is 8 GiB RAM / 4 vCPU and the service scales vertically to a 32 GiB / 16 vCPU peak on its own, with 32 GB of disk. So capacity is baseline-priced with 4× burst headroom — good for the bursty compile-and-test shape of an agent task — but the SUSTAINED ceiling is still 32 GiB, so repos that motivated the 120 GB ECS sizing stay on ecs. What the construct configures (and validates) is the baseline; the peak is not something a deployment asks for.

Normative requirements (EARS). All of these are P1 now — the earlier P1/P2 split of this list existed only because the phasing table split declaring a hook from serving it, which the service does not permit:

• (P1) The image build shall not embed secrets, tokens, or per-task identity in the snapshot.
• (P1) If the task payload exceeds the 4 KB runHookPayload limit, then the strategy shall upload the payload to the platform payload bucket and pass only its S3 URI in runHookPayload.
• (P1) The MicroVM execution role shall hold read-only access to the payload bucket, scoped to that bucket.
• (P1) Where no ingress is configured for a deployment, the strategy shall pass the Lambda-managed NO_INGRESS network connector on every RunMicrovm call (the field shall not be omitted).
• (P1) Where the image enables any MicroVM lifecycle hook, the image shall also enable the /ready hook and the agent shall serve it.
• (P1) When the /run hook receives the task payload, the agent shall validate it, start the pipeline asynchronously, and return HTTP 200 within the hook budget.
• (P1) If the /run hook payload cannot be resolved to a task payload, then the agent shall reject the hook with a client error and shall not start a pipeline.
• (P1) The agent shall not execute the clone→verify→PR pipeline on the hook path.
• (P1) The agent shall resolve credentials at /run time.
• (P1) Where a deployment configures a MicroVM image before smoke parity is verified, the platform shall warn that the backend has no smoke-parity guarantee.
• (P2) Where the image declares the /validate build hook, the agent shall serve it.

4. Infra and IAM: conditional resources behind bootstrap ComputeTypes

Mirroring the ECS pattern: a compute-lambda-microvm bootstrap policy (cdk/src/bootstrap/policies/) gated on the ComputeTypes CFN parameter; a CDK construct provisioning the build role, execution role (admitted to the per-session role via AgentSessionRole.admitComputeRole, which was designed for exactly this), the S3 artifact bucket wiring, and image build automation. Egress uses the platform VPC via egress network connectors so the DNS Firewall / security-group / flow-log stack in COMPUTE.md applies unchanged; ingress is suppressed with the Lambda-managed NO_INGRESS connector (no SHELL_INGRESS — it is noted as a candidate for #391, operator session access, as a separate decision).

Two networking facts the construct has to encode, both established live:

• A VPC_EGRESS connector requires an operator role. CloudFormation’s generated L1 types operatorRole as optional and this ADR originally assumed Lambda would manage the ENIs with its own service-linked role. It does not: the connector fails to create with “NetworkConnectorOperatorRole is required for VPC_EGRESS connector type”. The construct creates one role — trusting lambda.amazonaws.com with aws:SourceAccount pinned, carrying AWSLambdaVPCAccessExecutionRole plus the ENI / tag / private-IP actions that policy omits — and shares it across both connectors, since it manages interfaces rather than traffic.

• Build-time egress needs port 80; runtime does not. agent/Dockerfile installs Debian packages and apt-get fetches over plain HTTP, so a 443-only egress path fails every snapshot build (Could not connect to deb.debian.org:80 … exit code: 100). Rather than widen the runtime posture, the construct provisions a second, build-only connector on the same private subnets with a 443 + 80 security group, referenced solely by the image resource and the packaging script. The agent at run time still has 443-only egress.

• Where the bootstrap ComputeTypes parameter includes lambda-microvm, the generated template shall attach the IaCRole-ABCA-Compute-LambdaMicrovms policy to the CloudFormation execution role.

• The orchestrator role shall receive only the MicroVM lifecycle actions it calls (lambda:RunMicrovm, lambda:SuspendMicrovm, lambda:ResumeMicrovm, lambda:TerminateMicrovm, lambda:GetMicrovm for pollSession, and lambda:PassNetworkConnector, which is required even for the default connectors), scoped to platform-created images.

• Where the lambda-microvm backend is enabled, the approve and deny Lambdas shall receive lambda:ResumeMicrovm and lambda:GetMicrovm — conditionally, mirroring the cancel handler’s conditional RUNTIME_ARN wiring in task-api.ts.

lambda:CreateMicrovmAuthToken is granted to no role in P1–P3 (no JWE consumer exists; see sub-decision 3).

Cost attribution. cdk/src/main.ts currently tags the whole stack with a single compute_type context value (default agentcore) — already imprecise with two backends, wrong with three. P1 must add backend-identifying cost-allocation tags on the MicroVM-specific resources (images, payload/artifact bucket wiring, log groups) and revisit the stack-level tag semantics (e.g. a compute_types list), keeping attribution consistent with #645’s cost/attribution acceptance criterion.

• Where a deployment enables the lambda-microvm backend, MicroVM-specific resources shall carry backend-identifying cost-allocation tags.

Security bar vs existing backends (#645 acceptance criterion)

Control	AgentCore	ECS	Lambda MicroVMs	Delta
Egress, runtime (DNS Firewall, TCP 443 SG, flow logs)	Platform VPC	Platform VPC	Platform VPC via egress network connector	None
Egress, image build	ECR build outside the platform VPC	ECR build outside the platform VPC	Platform VPC via a separate build-only connector, TCP 443 + 80 (apt-get is plain HTTP)	New surface: build-time egress is wider than runtime egress by one port, on a connector no running MicroVM can use
Tenant-data scoping	Per-session role (admitComputeRole)	Per-session role	Per-session role, execution role admitted identically	None
Secrets delivery	Runtime env + Identity injection	Task env vars	Fetched at /run; never in snapshot	New surface: snapshot must stay secret-free (EARS req., sub-decision 3)
Inbound exposure	None (SigV4 invoke only)	None (no endpoint)	None — but only because the strategy passes NO_INGRESS explicitly. The service default is a PUBLIC HTTP_INGRESS connector plus a public *.lambda-microvm.<region>.on.aws endpoint; no tokens are minted in P1–P3 either way	New surface and a new failure mode: “no inbound” is an active control, not an absence. Drop the NO_INGRESS argument and every agent MicroVM gets a public endpoint (EARS req., sub-decision 3)
Session isolation	MicroVM	Task-level	MicroVM (Firecracker)	None (≥ ECS)
State reuse	None	None	Snapshot shared across MicroVMs	New surface: CSPRNG reseed + credential refresh on /run//resume (EARS req.)
Workload-token injection	Yes (Runtime-coupled)	No (env-var posture)	No (env-var posture)	Shared with ECS; deferred to #249/ADR-016
Operator shell access	No	No	Not enabled (SHELL_INGRESS omitted; candidate for #391)	None by default
Auth-token minting	n/a	n/a	CreateMicrovmAuthToken granted to no role in any phase	Verified static-only: any principal holding the action can mint a working JWE, including against a SUSPENDED MicroVM, so the posture rests entirely on the grant being absent

Regional availability enforcement

Lambda MicroVMs launched in 5 regions (us-east-1/2, us-west-2, eu-west-1, ap-northeast-1) and will expand. ABCA is a single-region deployment, so the constraint is binary per stack: either the stack region supports the backend or the backend does not exist there. Enforcement is layered — one static check where offline determinism is required, live probes everywhere else so the platform self-heals as AWS adds regions (list-managed-microvm-images is the documented read-only availability probe):

Stage	Mechanism	Check
CDK synth/deploy	Static region constant (single exported list, documented update path)	Synth fails when ComputeTypes includes lambda-microvm in an unlisted region; context-flag escape hatch for newly launched regions ahead of the constant update
Repo onboarding	Live probe from the CLI	bgagent repo onboard --compute-type lambda-microvm calls list-managed-microvm-images in the stack region and rejects with a remedy (supported-region list + suggest agentcore/ecs)
bgagent platform doctor	Live probe (precedent: checkBedrockModel)	Reports backend availability for the stack region whenever any active blueprint selects lambda-microvm
Orchestration (defense in depth)	Error classification	startSession failures from a missing regional endpoint classify to a typed remedy in error-classifier.ts, never a cryptic SDK error on the task

• If an operator onboards a repo with compute_type: 'lambda-microvm' and the availability probe fails for the stack region, then the CLI shall reject the onboarding with the supported-region list and alternative backends as the remedy.
• If startSession fails because the MicroVM service is unavailable in the stack region, then the orchestrator shall classify the failure with a configuration remedy and shall not retry.
• When the platform doctor runs in a deployment where any active blueprint selects lambda-microvm, the doctor shall probe MicroVM availability in the stack region and report the result.

5. Rollout: phased, default unchanged

• P1 — strategy + infra + minimal hook serving: LambdaMicrovmComputeStrategy (start/poll/stop), CDK construct, bootstrap policy, types sync, unit + CDK assertion tests, and the agent’s /ready + /run endpoints. No suspend yet. The image IS creatable and launchable and the payload DOES reach the agent — but there is no smoke-parity guarantee (sub-decision 3’s phasing table).
• P2 — smoke parity: the agent serves the remaining hooks (/terminate, /validate); agent completes clone → change → PR on the backend with progress visible to bgagent watch; failure classification entries in error-classifier.ts; AgentCore Memory parity (IAM grant + MEMORY_ID delivery, following the EcsAgentCluster pattern — Memory is a standalone service already consumed cross-substrate, and omitting the grant silently no-ops cross-session learning); the agent’s remaining non-secret env parity inside the snapshot.
• P3 — suspend/resume: the interface widening from sub-decision 1 (mandatory methods, all three strategies in one commit), HITL-wait suspend policy, inline resume in the approve/deny Lambda with orchestrator-poll reconciliation (sub-decision 2), timeout-under-freeze wall-clock handling; coordinate with #491’s unified liveness model and update Cedar decision #7’s rationale note.
• Out of scope: replacing AgentCore as default; classic Lambda functions as a runtime; GPU; the Runtime-coupled workload-access-token injection path (delivery mechanism exists only on AgentCore Runtime; MicroVMs adopt the ECS env-var posture until #249/ADR-016 redesign the seam). Gateway integration is orthogonal: ADR-019/#641 is substrate-portable by design and applies to this backend when it lands.

Consequences

• (+) Suspend/resume economics. Tasks idling on approval waits stop billing compute while preserving full state — bounded at ~1 h per gate under the current Cedar ceiling (decision #6), and the enabler for cheap off-hours gate-ceiling extensions later (§14.8). Verified end to end at the substrate level: suspend and resume each complete in ~1 s, and microvmId and endpoint survive the cycle byte-identical, so a stored SessionHandle remains valid.
• (+) VM-level isolation without cluster ops. Firecracker isolation with no ECS cluster, task definition, or capacity management; one-session-per-MicroVM maps 1:1 onto ABCA’s task model.
• (+) Escapes AgentCore’s 2 GB image limit and FUSE flock() workaround — native disk in the snapshot supports uv/mise without the split-storage scheme. Note the size comparison must say WHICH measure it means: the same agent tree is 1.799 GB as an OCI image (629.7 MB compressed, i.e. under AgentCore’s limit) but reports codeInstallSizeInBytes of 2.17 GiB as a MicroVM snapshot (i.e. over it). The two straddle the limit and are not interchangeable; memory/disk snapshot sizes are a third thing again and must not be summed into the comparison.
• (+) Liveness becomes explicit. Unlike AgentCore’s stub pollSession, the strategy can report real substrate state, strengthening the #491 unification.
• (−) 32 GiB sustained ceiling (8 GiB baseline + automatic 4× vertical scaling), 32 GB disk. Not a successor to the ECS backend for heavy CI-parity builds; the platform now maintains three backends.
• (−) Capacity is baseline-priced with burst headroom, which is a narrower promise than “32 GiB”. The deployment configures an 8 GiB / 4 vCPU baseline and the service scales to 32 GiB / 16 vCPU on demand — well matched to an agent task, which is idle-ish while waiting on the model and spiky during builds, and cheaper than reserving the peak. But it is burst, not a reservation: a workload that needs 32 GiB sustained is relying on scaling behaviour this ADR has not measured, and against ECS’s 120 GB the gap for sustained-memory workloads is unchanged. So the value proposition remains the suspend economics, the observable control-plane state machine, and the absence of cluster ops — with capacity now a fair-to-good fit rather than a hard blocker. Repos with genuinely sustained heavy builds still belong on ecs.
• (−) New packaging pipeline. Zip + Dockerfile + service-side image builds with versioned snapshots (storage billed per version) alongside the existing ECR flow; image versions need lifecycle cleanup — including versions left behind by FAILED builds, and noting the last version of an image cannot be deleted individually (delete the image, which reaps it).
• (−) The payload bucket is on the hot path, not the overflow path. With a 4 KB runHookPayload cap, virtually every real task delivers its payload via S3, so the bucket, its TTL rule and the execution role’s read grant are load-bearing for normal operation rather than an edge case (sub-decision 3).
• (−) 8-hour hard cap includes suspended time, and with idlePolicy omitted there is no tighter substrate-level suspended-TTL — the suspended-state bound is maximumDurationInSeconds plus orchestrator termination and the stranded reconciler. A manually suspended VM was observed alive at 1 h with no TTL in sight (observation truncated there), so nothing contradicts this bound, but nothing narrows it either. Under today’s 1 h gate ceiling it is comfortably sufficient; any future extension of gate ceilings must revisit the bound (an additive idlePolicy change) and give the orchestrator a checkpoint-and-restart path (push branch, new session) beyond the cap.
• (!) Idle-policy foot-gun. Traffic-based auto-suspend would freeze a busy outbound-only agent; the decision to disable auto-suspend must be enforced in code and covered by tests, not left to configuration discipline.
• (!) Service defaults are not the desired posture. Two live-caught cases (public HTTP_INGRESS by default; /ready mandatory) mean an omitted field on this backend does not mean “off” — it can mean “the service picks, and it picks wider than we want”. Every new RunMicrovm / CreateMicrovmImage field should be assumed to have an opinionated default until checked.
• (!) Nothing self-terminates. A MicroVM whose hook never ran still reaches RUNNING and stays there, billing, until the 8 h cap. The orchestrator’s TerminateMicrovm on finalize is the only cleanup, so a leaked handle is a cost incident, not just an untidy state.
• (!) Snapshot uniqueness. Shared memory snapshots require CSPRNG reseeding and credential refresh in /run / /resume hooks; missing this is a silent security defect.
• (!) Regional availability (5 regions at launch, expanding) — enforced in layers (synth-time static check, onboarding + doctor live probes, orchestration-time classification; see sub-decision 4). The static CDK constant is the one piece that rots as AWS expands; its update path and context-flag escape hatch are deliberate.
• (!) Workload-token injection delta persists (shared with the ECS backend) until #249/ADR-016 land; document it in the security bar comparison rather than blocking on it. Memory and Gateway are explicitly not deltas — both are standalone services consumed via IAM from any substrate.

Testing

P1 (start/poll/stop — no suspend):

• Unit tests for the strategy: start/poll/stop mapping (including SessionStatus 'suspended' reported mechanically, without task-state interpretation), payload-size branching (inline vs S3 pointer, at the exact 4 096/4 097-byte boundary), the image-identifier-must-be-an-ARN guard, the explicit NO_INGRESS argument (including the blank-env-var fallback, which must never omit the field), error classification (ServiceQuotaExceededException, ThrottlingException, ResourceNotFoundException, regional-unavailability), the omit-idlePolicy invariant, and maximumDurationInSeconds fixed at 28 800.
• Agent tests: /ready returns 200 once the server is up and starts nothing; /run accepts both envelope shapes (inline and S3 pointer), starts the pipeline asynchronously through the same mapper /invocations uses, returns before the pipeline finishes, and rejects every unusable envelope with a named code before spawning; /validate, /suspend, /resume and /terminate are NOT served.
• Orchestrator tests: substrate-terminal + non-terminal task status → failed classification; suspended + non-AWAITING_APPROVAL status → anomaly event, no fail-fast; compute_metadata persisted with microvmId/endpoint after startSession.
• CDK assertions: MicroVM resources present only when ComputeTypes includes the backend; synth failure for unsupported regions (plus the context-flag escape hatch); memory size validated against the accepted list at synth; the connector operator role and its trust; two connectors with the build-only one carrying port 80 and the runtime one not; the /ready + /run hook declaration and the absence of the others; IAM actions scoped as specified (orchestrator lifecycle set; no CreateMicrovmAuthToken anywhere); payload-bucket grants (execution role read-only); backend cost-allocation tags; types-sync check covers the widened ComputeType.
• CLI tests: onboarding rejection with remedy when the availability probe fails; doctor check present when a blueprint selects the backend.
• P1 verification items (external service facts) — executed 2026-07-31, us-east-1; see docs/verification/645-p1-lambda-microvm-runbook.md for the full evidence. Discharged: runHookPayload limit (4 096, not 16 KB), the accepted baseline memory sizes ([512…8192] MiB — note the developer guide, not the probe, is what establishes that this is a BASELINE with a 32 GiB peak), image-identifier ARN requirement, IAM action names and the observed image-ARN shape, region probe behaviour, manual suspend/resume without idlePolicy, terminate timing and the TERMINATED-persists-≥10-min finding, the default public HTTP_INGRESS, and the /ready requirement. Not discharged: account-quota treatment of SUSPENDED MicroVMs (not observable safely), suspended TTL beyond 1 h (truncated), the vertical-scaling behaviour itself (no workload here approached the baseline, so the 4× peak is documented rather than observed), and the AWS::Lambda::MicrovmImage CloudFormation value shapes (never exercised — the run used the out-of-band script path). Record the closed answers in COMPUTE.md.

P2 (smoke parity):

• Smoke (gated like the ECS backend): clone → change → PR with bgagent watch progress; Memory write parity (no AccessDenied no-op).

P3 (suspend/resume):

• Unit tests: suspend/resume mapping; agentcore/ecs unsupported stubs; approve/deny inline resume with handle loaded from compute_metadata.
• HITL lifecycle tests: inline resume failure leaves the decision outcome intact and records the orphan event; orchestrator backstop retries resume; gate expiry fires at min(monotonic budget, created_at + timeout_s) — including the suspend/resume case where the monotonic budget exceeds the wall-clock remainder — without disturbing the §13.12 late-approval race protection.
• Smoke: suspend/resume across a simulated approval wait preserving workspace state.

All phases: docs sync for COMPUTE.md (new column distinguishing MicroVMs from classic Lambda) and ORCHESTRATOR.md (liveness + suspend lifecycle).

References

• Issue #645 — originating RFC proposal
• Issue #491 — unified liveness decision model (soft dependency, P3)
• Issue #641 / ADR-019 (PR #663) — substrate-portable tool plane
• PR #596 — ECS Fargate backend (pattern source for conditional wiring)
• AWS Lambda MicroVMs — developer guide; Running and using MicroVMs — lifecycle APIs and hooks
• Agent Toolkit for AWS — aws-lambda-microvms skill — operational constraints (no self-suspend, idle-policy semantics, snapshot uniqueness, size limits)
• ADR-020 — EARS syntax used for the normative requirements above
• CEDAR_HITL_GATES.md — approval-gate mechanics (decisions #6, #7) the suspend/resume handshake preserves; cancel-task.ts / task-api.ts — the inline best-effort + reconciler-backstop pattern the resume path mirrors
• COMPUTE.md, ORCHESTRATOR.md — design docs to be updated by the implementing PRs