Skip to main content
Architecture Patterns

These are target-state architecture patterns for multi-tenant agent platforms at enterprise scale, not required prerequisites for a first prototype. Start with a single-account setup and evolve toward multi-account isolation as your platform matures.

Multi-Tenancy & Reference Architecture

Who this is for

Architects designing multi-tenant agent deployments — isolation models, account topology, operating models, and the maturity path from experimentation to governed platform.

Multi-tenancy is a cross-cutting concern that affects every layer of the agent platform. As multiple teams, business units, and use cases share the platform, proper isolation ensures security, performance predictability, and cost accountability.

Multi-Tenancy Dimensions​

Each platform layer requires its own isolation strategy. A "tenant" may be a team, business unit, application, or customer - depending on your organizational model.

DimensionIsolation RequirementImplementation
Runtime & ComputeAgents from different tenants cannot share execution contextsSeparate containers/micro-VMs, namespace isolation
Tool Scope BoundariesTenants access only their authorized tools and MCP serversPer-tenant Gateway policies, Cedar authorization. Combined Cedar schema per policy engine is capped at 400 KB, driven by both tool count and parameter complexity; large multi-tenant tool catalogs may need multiple policy engines.
Data & Prompt IsolationOne tenant's data never leaks into another's contextTenant-scoped storage, prompt injection guardrails
Memory IsolationAgent memory is partitioned per tenantTenant-prefixed memory stores, access policies
Eval & Telemetry SeparationMetrics, traces, and evals are attributed to the correct tenantTenant tags on all telemetry, separate dashboards
Registry & AccessEach tenant sees only their agents and approved shared agentsRBAC on registry, tenant-scoped discovery

Multi-Account Governance Architecture​

For enterprise-scale deployments, a multi-account architecture provides the strongest isolation boundaries using AWS Organizations.

Platform Account (Shared Services)

Central platform services: Agent Registry, Gateway configuration, shared guardrails, platform CI/CD, cost aggregation, and tenant onboarding automation.

Central

Domain Accounts (A, B, ...)

Per-team or per-business-unit accounts. Agents run here with tenant-specific resources: runtime, memory, tools, and data stores.

Per-Team

DevOps Account

Shared CI/CD pipelines, artifact repositories, and deployment orchestration. Deploys to domain accounts via cross-account roles.

Shared

Security & Networking Account

Centralized security controls: SCPs, VPC sharing, private link endpoints, WAF rules, and network firewall policies.

Security

Observability Account

Aggregated telemetry from all domain accounts. Cross-account CloudWatch dashboards, centralized alerting, and compliance reporting.

Observability

AgentCore Memory supports cross-account access, which is what makes a shared Memory resource in the Platform Account usable by agents running in Domain Accounts. A resource-based policy on the Memory resource can grant principals in another account permission to create events, write and retrieve records, and perform semantic search. Memory can also deliver payloads and stream events to S3 buckets, SNS topics, and Kinesis Data Streams in other accounts. See cross-account Memory access for setup.

Operating Models​

How the platform team and domain teams interact determines the platform's operating model.

ModelDescriptionBest For
CentralizedPlatform team builds and operates all agents. Domain teams submit requests.Small organizations, early stages, regulated industries
DecentralizedEach team builds and operates independently. Minimal shared services.Highly autonomous engineering cultures (high Shadow AI risk)
Federated ⭐Platform team provides shared services and guardrails. Domain teams build autonomously within boundaries.Most enterprises - balances speed with governance
Recommended

The Federated model is recommended for most enterprises. It provides the guardrails of centralized governance with the velocity of decentralized execution.

Organization Model​

The federated model requires a dedicated Platform / Center of Excellence (CoE) team that enables domain teams.

Platform / CoE Team​

RoleResponsibility
Platform PMRoadmap, prioritization, stakeholder alignment
Platform ArchitectReference architecture, standards, technology decisions
Platform EngineersBuild and maintain platform services, IaC constructs, self-service tooling
Platform OpsReliability, incident response, capacity management for shared services
Platform SecuritySecurity standards, policy authoring, threat modeling, compliance

Domain Teams​

Domain teams consume platform services and build agents for their business domain. They own:

  • Agent logic and prompts
  • Domain-specific tools and data
  • Quality and evaluation criteria
  • On-call for their agents

The platform team enables them with:

  • Self-service onboarding
  • Reusable templates and constructs
  • Shared observability and governance
  • Consultation and architecture reviews

Platform Maturity Model​

Organizations evolve through maturity levels. Each level builds on the previous one.

LevelNameCharacteristics
L1ExperimentationIndividual teams building PoCs. No shared infrastructure. Ad-hoc security.
L2FoundationShared runtime and gateway deployed. Basic registry. Central observability.
L3ScalingSelf-service onboarding. CI/CD with eval gates. Multi-team adoption. A/B testing via agentcore run ab-test (native Gateway-driven weighted splitting).
L4Governed & OptimizedFull governance automation. Cost optimization. Continuous evals. Federated ops.
Native A/B testing vs infrastructure-level routing

AgentCore provides native A/B testing with Gateway-driven weighted traffic splitting. Use agentcore run ab-test (CLI) or the CreateABTest API to configure tests. Two modes are available:

  • Config-bundle mode — tests prompt, tool, or model configuration variants on the same runtime (docs)
  • Target-based mode — routes traffic across different runtime endpoints (docs)

For infrastructure-level traffic splitting across entirely separate deployment targets outside AgentCore (for example, a green-field re-architecture comparison), external mechanisms such as CloudFront, ALB weighted target groups, or Route 53 weighted routing remain an option. See the A/B testing overview for guidance on choosing the right mode.

Path to Maturity​

L1 → L2: Lay the Foundation

Steps:

• Deploy AgentCore Runtime and Gateway
• Establish Agent Registry with basic metadata
• Enable AgentCore Observability for all agents
• Define initial security baseline (identity, guardrails)
• Create first IaC templates for common patterns

Foundation

L2 → L3: Scale Adoption

Steps:

• Build self-service onboarding workflows
• Implement CI/CD pipelines with evaluation gates
• Add cost attribution and per-team budgets
• Enable native A/B testing via agentcore run ab-test — compare configuration variants (config-bundle mode) or different runtime endpoints (target-based mode) with Gateway-driven weighted traffic splitting and online evaluation scoring
• Use infrastructure-level traffic splitting, such as CloudFront, ALB weighted target groups, or Route 53 weighted routing, only when you need network-layer distribution across deployment targets entirely outside AgentCore
• Onboard 3+ domain teams to the platform

Scaling

L3 → L4: Govern & Optimize

Steps:

• Automate governance with policy-as-code
• Implement continuous online evaluations
• Deploy multi-account architecture with SCPs
• Establish federated operating model with CoE
• Optimize cost with model routing and caching
• Shadow AI detection and prevention workflows

Governed
info

You don't need to reach L4 on day one. Start with L1 (experimentation), validate the value, then invest in platform capabilities incrementally. Most organizations should target L2 within 3 months and L3 within 6-9 months.

Further Reading​