Internal testing

This guide covers contributor-facing integration tests that are intentionally separate from the managed installation documentation.

Enterprise SSO integration test

The repository includes a disposable Cognito User Pool that behaves as an upstream OIDC provider. Contributors can use it to validate the browser redirect, claim mapping, access gate, authoritative role behavior, and sso-only UI without an Entra or Okta tenant.

This workflow intentionally uses deploy-terraform.sh and deploy-frontend.sh against a contributor test deployment. Product installations should use install.sh as documented in the Enterprise SSO guide.

The main stack and disposable identity provider create billable AWS resources. Use a non-production environment, keep both Terraform states, and complete the cleanup step.

1. Deploy the test stack in local mode

Start from a repository checkout with a direct deployment already configured:

• AWS credentials select the test account.
• terraform/environments/<environment>.tfvars and the matching .s3.tfbackend file exist.
• The tfvars region matches the account and environment being tested.

The commands below use dev; substitute the configured test environment consistently:

./scripts/deploy-terraform.sh dev --auth-mode local
./scripts/deploy-frontend.sh dev

export AIDLC_OIDC_CALLBACK="$(
  terraform -chdir=terraform output -raw oidc_idp_callback_url
)"
export AIDLC_TEST_REGION="$(
  terraform -chdir=terraform output -raw aws_region
)"

2. Deploy the disposable identity provider

Use the same AWS profile and region as the main deployment:

terraform -chdir=test/fixtures/oidc-idp init
terraform -chdir=test/fixtures/oidc-idp apply \
  -var "aws_region=$AIDLC_TEST_REGION" \
  -var "downstream_callback_url=$AIDLC_OIDC_CALLBACK"

terraform -chdir=test/fixtures/oidc-idp output -json provider_config \
  > /tmp/aidlc-test-idp.json

export TEST_IDP_POOL="$(
  terraform -chdir=test/fixtures/oidc-idp output -raw user_pool_id
)"

The fixture state contains a generated OIDC client secret. Treat that state as sensitive and destroy the fixture after testing.

3. Create test identities

Create three users with separate email addresses and strong disposable passwords. The following commands create the member:

aws cognito-idp admin-create-user \
  --region "$AIDLC_TEST_REGION" \
  --user-pool-id "$TEST_IDP_POOL" \
  --username member@example.com \
  --message-action SUPPRESS \
  --user-attributes \
    Name=email,Value=member@example.com \
    Name=email_verified,Value=true \
    Name=name,Value="Test Member"

aws cognito-idp admin-set-user-password \
  --region "$AIDLC_TEST_REGION" \
  --user-pool-id "$TEST_IDP_POOL" \
  --username member@example.com \
  --password '<disposable-strong-password>' \
  --permanent

Repeat both commands for admin@example.com and denied@example.com, changing the email, username, display name, and password. Assign the member and administrator groups:

aws cognito-idp admin-add-user-to-group \
  --region "$AIDLC_TEST_REGION" \
  --user-pool-id "$TEST_IDP_POOL" \
  --username member@example.com \
  --group-name aidlc-user

aws cognito-idp admin-add-user-to-group \
  --region "$AIDLC_TEST_REGION" \
  --user-pool-id "$TEST_IDP_POOL" \
  --username admin@example.com \
  --group-name aidlc-user
aws cognito-idp admin-add-user-to-group \
  --region "$AIDLC_TEST_REGION" \
  --user-pool-id "$TEST_IDP_POOL" \
  --username admin@example.com \
  --group-name aidlc-admin

Leave denied@example.com out of aidlc-user.

4. Verify hybrid login and role mapping

./scripts/deploy-terraform.sh dev \
  --auth-mode hybrid \
  --sso-config /tmp/aidlc-test-idp.json
./scripts/deploy-frontend.sh dev

Create a separate browser profile or isolated browser context for the local administrator, test member, test administrator, and denied user before signing in. Different tabs and additional private windows in the same browser can share the upstream IdP cookie and are not isolated. Separate browser profiles, different browsers, Firefox containers, or separate Playwright browser contexts are suitable.

This isolation is intentional. AI-DLC logout clears the application and federation-broker session, but an application cannot portably terminate both OIDC and SAML IdP sessions. The disposable IdP therefore reuses the identity already authenticated in a browser context. It receives no fixture-specific account-switch behavior; this is the same upstream-session boundary used for every configured identity provider.

Verify each path in its assigned context:

Cognito can reduce a pre-token access-gate rejection to a bare invalid_grant, the same error used for expired or replayed authorization codes. The callback only shows Access denied when Cognito or the upstream provider preserves an explicit SSO_ACCESS_DENIED or access_denied marker; otherwise it offers a generic retry.

In Admin -> Users, both admitted federated users should say Managed by DisposableCognito and have no role-edit button. The denied identity also remains in this administrative directory because Cognito creates its user record before the access gate runs. It must be marked access denied, must not have an effective Admin badge, and must not appear in a space's Add member user list.

As a space owner, open Project Settings -> Members and verify:

1. Both admitted federated users appear as DisposableCognito identities.
2. The local administrator appears separately as a Cognito account.
3. The denied user does not appear.
4. Adding the test member allows that identity to open the space after sign-in.
5. The assigned project role is reflected in the available settings and discussion controls.
6. A discussion message posted by the test member is rendered as that user's own message, not as another member's message.

In the test administrator's browser context, remove aidlc-admin, log out of AI-DLC, and sign in again. Reusing that context for the same identity is expected here. The Admin entry must disappear:

aws cognito-idp admin-remove-user-from-group \
  --region "$AIDLC_TEST_REGION" \
  --user-pool-id "$TEST_IDP_POOL" \
  --username admin@example.com \
  --group-name aidlc-admin

Add the group back before testing sso-only. To verify that external roles are authoritative, manually add the downstream federated username to its Cognito platform-admin group. After a fresh SSO login, the token must still contain only roles mapped from the upstream claim.

5. Verify SSO-only and clean up

After the SSO administrator works:

./scripts/deploy-terraform.sh dev \
  --auth-mode sso-only \
  --sso-config /tmp/aidlc-test-idp.json
./scripts/deploy-frontend.sh dev

The Cognito account form must be absent. Restore local mode before deleting the fixture because the main Terraform configuration still reads its client secret:

./scripts/deploy-terraform.sh dev --auth-mode local
./scripts/deploy-frontend.sh dev

terraform -chdir=test/fixtures/oidc-idp destroy \
  -var "aws_region=$AIDLC_TEST_REGION" \
  -var "downstream_callback_url=$AIDLC_OIDC_CALLBACK"

rm -f /tmp/aidlc-test-idp.json
unset AIDLC_OIDC_CALLBACK AIDLC_TEST_REGION TEST_IDP_POOL

AgentCore tests

AgentCore has two separate test layers:

1. The deterministic AgentCore test project, which uses local test containers and does not call a model.
2. The credentialed local E2E, which runs the real Claude, Kiro, OpenCode, and Codex CLIs against real models.

Neither test requires a deployed AI-DLC stack.

Deterministic AgentCore tests

Run the AgentCore project from the repository root:

npx vitest run --project=agentcore

These tests are suitable for normal pull request validation. They require a running Docker daemon because the Vitest global setup starts DynamoDB Local and Gremlin Server with Testcontainers. They do not require model credentials and do not incur model usage.

Run the repository checks before opening a pull request:

npm run lint
npm run format:check
npm run secretlint
npx vitest run --project=agentcore

Local multi-CLI E2E

scripts/agent-e2e-testing.sh is the only user-facing runtime E2E command. It builds and runs the production AgentCore container locally, using DynamoDB Local and Gremlin Server instead of deployed AWS resources.

The E2E makes real model calls and can incur usage charges. Claude, Kiro, OpenCode, and Codex run sequentially to limit spend and avoid shared conversation-store races.

Prerequisites

• Docker with Buildx
• Native ARM64 execution or working linux/arm64 emulation
• Outbound HTTPS access from Docker containers
• A Bedrock API key for Claude, OpenCode, and Codex
• A Kiro API key for Kiro
• For Codex, the OpenAI models (openai.gpt-5.*) enabled in Bedrock for the chosen AWS_REGION

Before making model calls, the script checks Docker, Buildx, ARM64 execution, key presence, model syntax, and outbound connectivity. It supplies inert AWS credentials to DynamoDB Local and does not require an AWS profile, Terraform state, Cognito login, or deployed stack.

When running behind an HTTP proxy, export the standard proxy variables before starting the E2E:

export HTTP_PROXY=http://proxy.example.com:8080
export HTTPS_PROXY=http://proxy.example.com:8080
export NO_PROXY=localhost,127.0.0.1
./scripts/agent-e2e-testing.sh

The harness forwards non-empty standard proxy variables, including lowercase variants, to the Buildx build, outbound-connectivity check, and AgentCore test containers. No proxy flags or container variables are added when none are set. The Docker daemon still needs its own proxy configuration to pull base and test images.

Run all CLIs

To avoid placing credentials directly in shell history, enter them in a Bash session without echo:

read -rsp 'Bedrock API key: ' BEDROCK_API_KEY; printf '\n'
read -rsp 'Kiro API key: ' KIRO_API_KEY; printf '\n'
export BEDROCK_API_KEY KIRO_API_KEY

./scripts/agent-e2e-testing.sh

unset BEDROCK_API_KEY KIRO_API_KEY

For non-interactive use, the equivalent command is:

BEDROCK_API_KEY=... KIRO_API_KEY=... ./scripts/agent-e2e-testing.sh

AWS_BEARER_TOKEN_BEDROCK is accepted when BEDROCK_API_KEY is absent.

Run selected CLIs

E2E_CLIS is a comma-separated list. Only credentials needed by the selected CLIs are required:

# Claude, OpenCode, and Codex use the same Bedrock key.
E2E_CLIS=claude,opencode,codex ./scripts/agent-e2e-testing.sh

# Kiro only.
E2E_CLIS=kiro ./scripts/agent-e2e-testing.sh

# Codex only.
E2E_CLIS=codex ./scripts/agent-e2e-testing.sh

The examples assume the corresponding key was already exported.

Configuration

Do not prefix BEDROCK_MODEL with amazon-bedrock/. The harness adds that provider prefix for OpenCode and passes the bare value to Claude.

To reuse an image:

docker buildx build \
  --platform linux/arm64 \
  --load \
  --tag aidlc-agentcore:e2e \
  --file lambda/agentcore/Dockerfile \
  lambda

AGENTCORE_IMAGE=aidlc-agentcore:e2e ./scripts/agent-e2e-testing.sh

What the E2E verifies

For each selected CLI, the harness:

1. Seeds an isolated DynamoDB execution, Gremlin Intent, stage, and workspace.
2. Starts a fresh AgentCore container and runs the real CLI through runStage.
3. Requires the agent to call ask_question and park.
4. Verifies the pending gate and persisted CLI session ID.
5. Removes the container, answers the gate directly in the process store, and starts a new container with the same workspace volume.
6. Resumes the same CLI session.
7. Requires create_artifact, send_output, and collect_metric.
8. Verifies the successful stage, graph edge, output, metric, session identity, native edit parsing, output timestamps, and Git runtime exclusions.

Starting fresh and resume legs in separate containers exercises Claude's and Codex's durable JSONL state and Kiro/OpenCode SQLite restore and persistence.

The script continues after an individual CLI failure and prints a flat summary:

Claude:   PASS
Kiro:     PASS
OpenCode: PASS
Codex:    PASS

It exits nonzero if any selected CLI fails.

Each run also writes one normalized transcript report per selected CLI and refreshes the standalone frontend fixture at /agent-output-preview.html. Start it without model calls with:

npm run dev:agent-output

Credentials and cleanup

Keys are written to a mode-0600 temporary file, mounted read-only into local test containers, and deleted by the exit trap. Key values are not passed through Docker command arguments or --env, stored in container metadata, written to SSM, or sent to a deployed stack.

Successful runs remove their containers, named volumes, private network, local fixtures, and logs. To retain a failed run:

KEEP_E2E=1 ./scripts/agent-e2e-testing.sh

The failure output prints the resource label and log directory. Inspect retained resources with:

docker ps -a --filter 'label=aidlc.e2e=<run-id>'
docker volume ls --filter 'label=aidlc.e2e=<run-id>'
ls -la '<log-directory>'

After inspection, remove retained resources:

docker ps -aq --filter 'label=aidlc.e2e=<run-id>' |
  while read -r id; do docker rm -f "$id"; done
docker volume ls -q --filter 'label=aidlc.e2e=<run-id>' |
  while read -r id; do docker volume rm -f "$id"; done
docker network rm 'aidlc-e2e-<run-id>'
rm -rf '<log-directory>'

The temporary credential file is deleted even when KEEP_E2E=1.

Deployed-stack diagnostics

scripts/phaseb.sh remains a diagnostic tool for an already deployed AgentCore stack. It reads Terraform outputs and exercises deployed routing and persistence. The local E2E does not invoke it.

Use agent-e2e-testing.sh for local container lifecycle coverage. Use phaseb.sh only when diagnosing deployed infrastructure or routing behavior.

The credentialed E2E is intentionally local and manual; it is not a GitHub Actions workflow.