Skip to main content
Architecture Patterns

These are target-state architecture patterns for agent operations at enterprise scale, not required prerequisites for a first prototype. Native AgentCore capabilities are clearly distinguished from customer-built patterns throughout this page.

Agent Ops

Who this is for

Platform engineers designing deployment and operations for agents on AgentCore — specifically the CI/CD pipeline, traffic management, A/B testing, lifecycle governance, and operational feedback loops.

Agent Ops extends DevOps practices to the unique challenges of AI agents. Because agents are non-deterministic and continuously evolving, they require purpose-built CI/CD pipelines, continuous evaluation, traffic management, and lifecycle governance. This page covers the full deployment pipeline, ongoing operations, and the clear distinction between what AgentCore Runtime provides natively and what you build on top.

Agent CI/CD Pipeline​

An agent deployment pipeline validates quality, safety, and performance at every stage. It goes beyond "does it build?"

Commit→Build & Test→Validate (Evals)→Promote to Stage→Registry→Promote to Prod

Pipeline Stages​

StageWhat HappensGate Criteria
CommitCode pushed, PR createdLinting, type checks, unit tests pass
Build & TestContainer image built, integration tests runAll tests pass, image scans clean
Validate (Evals)Deterministic tests plus batch evaluations against test datasetDeterministic checks pass, quality score ≥ threshold, latency ≤ SLA
Promote to StageDeploy to staging environmentSmoke tests pass, no regression vs. baseline
RegistryAgent version registered with metadataApproval workflow complete (if required)
Promote to ProdDeploy to production with traffic rampCanary metrics healthy, no alerts triggered

Quality Feedback Loop​

When evaluations fail at the Validate stage, the pipeline provides actionable feedback:

Eval Fails→Identify Failure Cases→Optimize (Prompt / Model / Data)→Re-commit→Re-validate
tip

Treat evaluation failures like test failures - they block the pipeline. This ensures only validated agents reach production.

Ongoing Operations​

Once an agent is in production, operations shift to continuous monitoring, experimentation, and lifecycle management.

Monitoring & Evaluations​

Continuous operational health checks powered by AgentCore Observability, Evaluations, CloudWatch Alarms, and Cost Explorer.

SignalMonitorsAction on Breach
QualityBuiltin.Faithfulness, Builtin.ResponseRelevance, Builtin.Correctness scores (evaluators)Alert owner, escalate if sustained
CostToken spend, tool invocation costs per agentThrottle, switch model, notify team
BehaviorResponse distribution, tool usage patternsInvestigate drift, trigger on-demand eval
AvailabilitySuccess rate, timeout rate, error rateAuto-scale, circuit break, rollback

Versioning, Endpoints, and Canary Rollout​

AgentCore Runtime versions automatically. Any change that affects behavior (protocol settings, container image, network configuration) creates a new immutable version; unrelated changes do not. Endpoints are the mechanism for controlling which version actually serves traffic. See Runtime lifecycle settings for full details on endpoint management.

Change TypeVersion CreatedDEFAULT EndpointNamed Endpoint (e.g. PROD)
Initial creationV1Points to V1Not yet created
Container image updateV2Moves to V2Stays on V1 until updated manually
Point PROD at V2 explicitlyNo new versionUnchangedMoves to V2
Container image update againV3Moves to V3Stays on V2

See Agent Runtime versioning for the full versioning model.

This gives you blue-green deployment out of the box: cut a new version, validate it behind a staging endpoint, then repoint PROD to it once you're confident.

A canary pattern built on top of endpoint versioning looks like:

  1. Deploy the new agent version, create a canary endpoint pointed at it, leave PROD on the current version.
  2. Route a small percentage of live traffic to canary through your routing layer.
  3. Compare quality, latency, and cost between canary and PROD using AgentCore Observability's per-version metrics.
  4. If the canary holds or improves on quality thresholds, repoint PROD to the new version and delete the canary endpoint. If it regresses, delete canary and repoint traffic back to PROD. No code change is needed since PROD never moved.

Native A/B Testing​

AgentCore provides native A/B testing with Gateway-driven weighted traffic splitting. This lets you compare agent variants scientifically — with statistical significance — without building your own routing layer.

Two modes are available (A/B testing overview):

  • Config-bundle mode — compares prompt, tool, or model configuration variants running on the same runtime. Use this when you want to test whether a prompt rewrite or model upgrade improves quality without changing infrastructure. See config-bundle A/B testing.
  • Target-based mode — routes traffic to two different runtime endpoints. Use this for comparing fundamentally different agent architectures or framework versions. See target-based A/B testing.

Both modes support percentage-weighted traffic splits via --control-weight / --treatment-weight (0–100, default 50/50). Routing is sticky by runtimeSessionId, so a given session always reaches the same variant. Online evaluations score both variants throughout the test, and statistical significance is reported automatically.

# Start an A/B test with a 90/10 traffic split
agentcore run ab-test \
  --control-endpoint arn:aws:bedrock-agentcore:...:endpoint/prod \
  --treatment-endpoint arn:aws:bedrock-agentcore:...:endpoint/candidate \
  --control-weight 90 \
  --treatment-weight 10

You can also create and query A/B tests programmatically via the CreateABTest / GetABTest APIs.

Agent Lifecycle Management​

An agent's operational lifecycle (is it being built, tested, or serving traffic) and its registry status (is it discoverable and approved for reuse) are two different, loosely-coupled tracks. Runtime versioning and endpoints drive the first; AWS Agent Registry drives the second.

Operational PhaseWhat's HappeningTypical Registry Status
DevelopmentBuild, test locally, iterate on promptsDRAFT, not yet submitted
ReviewSubmitted for security/compliance reviewPENDING_APPROVAL
ProductionServing live traffic behind a PROD endpointAPPROVED, discoverable for reuse
DeprecatedMarked for removal, consumers notifiedDEPRECATED, no longer surfaced in search

A record can also land in REJECTED if a reviewer declines it. That's not a phase an agent passes through on the way to production. It's a dead end that sends the publisher back to DRAFT.

Key lifecycle operations:

  • Version History: AgentCore Runtime creates an immutable version on every behavior-affecting change (see the endpoint versioning table above). This is independent of Registry status.
  • Rollback: repoint the PROD endpoint to a previous Runtime version. Registry status does not need to change for a rollback. The record still describes the same logical agent.
  • Deprecation Workflow: move the Registry record to DEPRECATED so it stops appearing in search and the MCP discovery endpoint, notify known consumers, and set a sunset date for the underlying Runtime endpoint. Traffic redirection after the sunset date is handled by your routing layer, not by Registry itself.

Services Summary​

CapabilityPrimary ServiceLabel
CI/CD PipelineCodePipeline + CodeBuild (or GitHub Actions)Adjacent AWS
Evaluation GatesAgentCore EvaluationsAgentCore-native (docs)
Monitoring & AlertsAgentCore Observability + CloudWatch AlarmsAgentCore-native + Adjacent AWS
Cost TrackingAWS Cost Explorer + Cost Allocation TagsAdjacent AWS
Version ManagementAgentCore Runtime endpoint versioning (docs)AgentCore-native
A/B TestingAgentCore native — Gateway-driven weighted splitting (docs)AgentCore-native
Canary Rollout (custom %)Customer-built routing layer + AgentCore ObservabilityCustomer-built pattern
Agent CatalogAWS Agent RegistryPreview (docs)

Further Reading​