Architecture Patterns

These are target-state architecture patterns for agent observability and evaluation at enterprise scale, not required prerequisites for a first prototype. AgentCore Observability and Evaluations are native services; CloudWatch alarms and CI/CD integrations are adjacent AWS services you wire together.

Observability & Evaluations

Who this is for

Teams instrumenting observability and evaluation pipelines for production agents — the observe-evaluate-optimize loop, built-in evaluator IDs, evaluation modes, and how to close the feedback loop with CI/CD and alerting.

Agents are non-deterministic. The same input can produce different outputs, take different reasoning paths, and invoke different tools. Traditional monitoring (CPU, memory, error rates) is necessary but insufficient. Agent platforms need an observe → evaluate → optimize loop that continuously measures quality, safety, and performance. This page covers what AgentCore Observability and Evaluations provide natively, how to close the loop with adjacent AWS services, and which actions (such as rollback) remain customer-built patterns.

The Observe → Evaluate → Optimize Loop

Observe→ Evaluate→ Alert & Optimize→ Repeat

Observe

Collect comprehensive telemetry from every agent interaction. The observation layer captures what happened, when, and how long it took.

Services: AgentCore Observability, CloudWatch, X-Ray

Signal	What's Captured	Purpose
Traces	End-to-end request flow, tool calls, model invocations, latency per step	Debug failures, understand reasoning paths
Metrics	Token usage, latency percentiles, success/error rates, cost per invocation	SLA monitoring, capacity planning
Logs	Agent reasoning, tool inputs/outputs, memory operations	Root cause analysis, audit
User Feedback	Thumbs up/down, corrections, escalations	Ground truth signal for evaluation
Guardrail Events	Policy violations, blocked content, PII detections	Safety monitoring, policy tuning

Evaluate

Automatically assess agent behavior against defined quality, safety, and performance criteria. Evaluations turn raw telemetry into actionable quality signals.

Service: AgentCore Evaluations

Dimension	What's Measured	Example Metrics
Quality	Response accuracy, relevance, completeness, groundedness	`Builtin.Faithfulness`, `Builtin.ResponseRelevance`, `Builtin.Correctness`, `Builtin.GoalSuccessRate`
Safety	Guardrail compliance, content policy adherence	`Builtin.Harmfulness`, PII leak rate, topic violation count
Performance	Latency, throughput, token efficiency	P95 latency, tokens per task, `Builtin.ToolSelectionAccuracy`, tool call efficiency
Cost	Token consumption, compute cost, tool invocation cost	Cost per conversation, cost per task completion
Drift	Behavior change over time, model degradation	Quality score trend, response distribution shift

Alert & Optimize

Close the loop by triggering automated actions when evaluations detect issues.

Services: CI/CD pipelines, CloudWatch Alarms

Action	Trigger	Result
Alerts	Quality score drops below threshold	PagerDuty/Slack notification to agent owner
Rollback	Safety evaluation fails in production	Repoint the `PROD` endpoint to a previous Runtime version (customer-built pattern using Runtime endpoint versioning + Observability metrics)
Throttle	Cost exceeds budget threshold	Reduce rate limits or switch to cheaper model
Update Evals	New failure patterns detected	Add new evaluation criteria to catch edge cases
Optimize	Ongoing quality metrics	Model tuning, prompt refinement, data augmentation

Evaluation Framework

Evaluations run in three modes, each serving a different point in the agent lifecycle.

Evaluation Modes

Mode	When	Trigger	Purpose
Batch	Every deployment	CI/CD pipeline gate	Prevent regressions from reaching production
On-Demand	Event-driven or human-triggered	Manual request, incident response, experiment	Deep-dive analysis, A/B comparison
Online	Continuous in production	Sampling (e.g., 10% of traffic)	Detect drift, measure real-world quality

Batch - CI/CD Gate

Runs on every deployment. Evaluates against a curated test dataset. Blocks promotion if quality, safety, or performance thresholds aren't met.

Every Deploy

On-Demand

Triggered by humans or events. Used for incident investigation, experiment analysis, or deep-dive into specific agent behaviors.

Event Triggered

Online - Production Sampling

Continuously samples live traffic. Detects quality drift, new failure modes, and model degradation before users report issues.

Continuous

Evaluation Dimensions

A full evaluation framework measures agents across five dimensions:

Dimension	Batch	On-Demand	Online	Example Evaluators
Quality	✅ Regression test	✅ A/B comparison	✅ Sampling	`Builtin.Faithfulness`, `Builtin.ResponseRelevance`, `Builtin.Correctness`
Safety	✅ Adversarial inputs	✅ Red team results	✅ Guardrail events	`Builtin.Harmfulness`, PII detection, topic violations
Performance	✅ Latency benchmarks	✅ Load testing	✅ P95/P99 tracking	End-to-end latency, token efficiency
Cost	✅ Cost projection	✅ Budget analysis	✅ Spend tracking	Cost per task, cost vs. quality tradeoff
Operational	✅ Error rate baselines	✅ Incident postmortem	✅ Availability	Success rate, retry rate, timeout rate

info

Start with Quality and Safety evaluations. Add Performance, Cost, and Operational dimensions as your agent fleet matures and you need optimization signals.

The Observe → Evaluate → Optimize Loop​

Observe​

Evaluate​

Alert & Optimize​

Evaluation Framework​

Evaluation Modes​