Skip to main content

Evaluations for Workflow Agents

Who this is for

Builders who have a deployed workflow agent and want to measure quality continuously in production. If you're still in the build phase, finish the quickstart first.

Workflow agents make real decisions with real consequences. You need to know, continuously and not just at launch, whether those decisions are correct. Conversational agents have users who immediately notice bad answers. Workflow agents operate in the background, so bad decisions can compound silently for days before anyone notices.

Built-in Evaluators​

AgentCore provides built-in evaluators that score agent behavior against your session telemetry. Trajectory evaluators use programmatic scoring (no LLM calls, zero token cost), while Builtin.GoalSuccessRate and Builtin.Correctness use LLM-as-a-Judge. For the offline vs. online evaluation model, see the AgentCore developer guide.

EvaluatorLevelGround Truth FieldWhat It Measures
Builtin.GoalSuccessRateSessionassertionsBehavior satisfies natural-language assertions (LLM-as-Judge)
Builtin.CorrectnessTraceexpectedResponseResponse matches the expected answer (LLM-as-Judge)
Builtin.TrajectoryExactOrderMatchSessionexpectedTrajectoryExact tool sequence: same tools, same order, no extras (programmatic)
Builtin.TrajectoryInOrderMatchSessionexpectedTrajectoryExpected tools appear in order; extras allowed between (programmatic)
Builtin.TrajectoryAnyOrderMatchSessionexpectedTrajectoryAll expected tools present, any order, extras allowed (programmatic)
Custom (LLM-as-judge)—placeholdersYour domain-specific criteria (e.g., "followed escalation policy")

Ground Truth Evaluations​

Ground truth is the known-correct answer or expected behavior for a given input. It is the "gold standard" you compare actual results against. You provide reference inputs alongside session spans when calling the Evaluate API, and the service scores actual behavior against expected behavior.

FieldScopeDescription
expectedResponseTraceExpected agent response for a specific turn (scoped by traceId)
assertionsSessionNatural-language statements that should be true about the behavior
expectedTrajectorySessionExpected sequence of tool-call names for the session

Ground truth fields are optional. If you omit them, evaluators fall back to their ground-truth-free mode. For workflow agents, build a benchmark dataset of past events with known-correct outcomes to catch regressions before they reach production.

Prerequisites

Evaluations require an agent deployed on AgentCore Runtime with observability enabled (or a supported framework like Strands Agents or LangGraph configured for AgentCore Observability), Transaction Search enabled in CloudWatch, and Python 3.10+.

Running an Evaluation​

Use the AgentCore CLI to evaluate a session against ground truth. Run this from inside an AgentCore project directory (created with agentcore create); use the --agent-arn flag instead of --runtime to run it from outside a project directory. See the on-demand evaluation getting started guide for full setup instructions.

agentcore run eval \
  --runtime YOUR_RUNTIME_NAME \
  --session-id YOUR_SESSION_ID \
  --evaluator "Builtin.GoalSuccessRate" \
  --evaluator "Builtin.TrajectoryExactOrderMatch"

Results are saved locally and can be reviewed later with agentcore evals history.

For programmatic use, the AgentCore SDK offers the same capability:

from bedrock_agentcore_starter_toolkit import Evaluation

eval_client = Evaluation()
results = eval_client.run(
    agent_id="YOUR_AGENT_ID",
    session_id="YOUR_SESSION_ID",
    evaluators=["Builtin.GoalSuccessRate", "Builtin.TrajectoryExactOrderMatch"]
)

for r in results:
    print(f"{r['evaluatorId']}: {r['value']} ({r['label']})")
Verified CLI version

agentcore run eval was checked with AgentCore CLI 0.26.0. Run agentcore run --help when using a different CLI version.

Cost-efficient quality gates

Trajectory evaluators are programmatic (zero token cost), so you can run them on 100% of invocations cheaply. Reserve the LLM-judged evaluators (Builtin.GoalSuccessRate, Builtin.Correctness) for sampled traffic or high-priority events.

Observability & Online Evaluation​

Observability & Online Evaluation

Full OpenTelemetry tracing for every agent invocation, from trigger event to final action. Track tool selection accuracy, goal success rate, and correctness continuously in production with online evaluations. Know why the agent made each decision via per-invocation traces in CloudWatch.

Observability

Further Reading​