sample-aiml-security-assessment

AWS AI/ML Security Assessment for Amazon Bedrock, Amazon SageMaker AI, Amazon Bedrock AgentCore, and AWS Agent Registry

A serverless framework based assessment that scans your AWS accounts for AI/ML security misconfigurations and produces an interactive, shareable report.

License: MIT-0 Python 3.12 AWS SAM

Open-source automated security scanner for generative AI and machine learning workloads on AWS. It brings together separate assessments for Amazon Bedrock, Amazon SageMaker AI, Amazon Bedrock AgentCore, and AWS Agent Registry. Core checks are guided by the AWS Well-Architected Generative AI Lens. The optional Responsible AI GRC module adds technical checks for AI governance, risk, and compliance. Optional OWASP Top 10 for LLM checks extend coverage across common LLM security risks. Responsible AI GRC checks draw on the AWS User Guide to Governance, Risk, and Compliance for Responsible AI Adoption.

Run 208 checks across AWS accounts and regions:

Deploy in a single account or across AWS Organizations. Assessments support multi-region execution within the standard AWS commercial partition and produce interactive, shareable reports with severity ratings, filtering, search, remediation references, and per-account/per-region views. Assessment artifacts are stored in your AWS account; the deployment build pulls source from the configured repository.

Scope note: Responsible AI GRC provides selected AWS configuration checks for AI governance, risk, and compliance. It complements architectural reviews such as the AWS Well-Architected Responsible AI Lens and broader compliance programs. See Responsible AI GRC scope, sources, and compatibility.

See It In Action

The framework generates professional, interactive security assessment reports with filtering, search, and dark mode support.

Download Sample Reports Single Account Multi-Account
AWS AI/ML security assessment dashboard showing Amazon Bedrock, Amazon SageMaker AI, Amazon Bedrock AgentCore, and AWS Agent Registry findings by severity

Executive Dashboard (Light Mode)

 AWS AI/ML security assessment dashboard showing Amazon Bedrock, Amazon SageMaker AI, Amazon Bedrock AgentCore, and AWS Agent Registry findings by severity

Executive Dashboard (Dark Mode)
Detailed Findings Table

Interactive Findings Table with Filtering

Key Features

Table of Contents

What It Does

This serverless assessment framework automatically evaluates your AI/ML workloads against AWS security best practices. It uses AWS serverless services to gather data from the control plane and generate reports containing the status of various security checks, severity levels, and recommended actions.

Designed for workloads using Amazon Bedrock, Amazon Bedrock AgentCore, AWS Agent Registry, Amazon SageMaker AI, or the optional Responsible AI GRC assessment.

Why Use This Framework?

Challenge How This Framework Helps
Manual security audits are time-consuming Fully automated scanning with one-click CloudFormation deployment
Inconsistent security checks across teams Standardized 208-check assessment based on AWS Well-Architected Generative AI Lens and Agentic AI Lens best practices, AWS Responsible AI governance/risk/compliance guidance, and OWASP Top 10 for LLM
Difficulty tracking AI/ML security posture Interactive HTML dashboards with severity breakdown and per-account visibility
Multi-account complexity Consolidated reporting across AWS Organizations with cross-account role assumption
Compliance and audit support Exportable reports to supplement your compliance program, with remediation guidance linked to AWS documentation
Generative AI security gaps Purpose-built checks for LLM guardrails, model access controls, and prompt injection prevention

Services Covered:

Deployment Options:

How It Works:

  1. Deploy through AWS CloudFormation (one-click deployment)
  2. Framework automatically scans your AI/ML resources
  3. Generates interactive HTML reports stored in your Amazon S3 bucket
  4. All data stays in your AWS account - no external dependencies

Scope and Limitations

This tool operates within the AWS Shared Responsibility Model. It assesses your configuration responsibilities (IAM policies, encryption settings, network isolation, logging) for AI/ML services. It does not assess AWS-managed infrastructure, physical security, or the underlying service platform.

Point-in-time assessment. Each run captures your security posture at the moment of execution. Resource configurations can change immediately after an assessment completes. Run assessments regularly and after significant changes to maintain visibility.

No guarantee of security or compliance. This framework identifies common misconfigurations based on AWS best practices and the AWS Well-Architected Framework. It does not cover all possible security risks, does not replace formal compliance audits (SOC 2, HIPAA, and similar), and does not guarantee that your workloads are secure. Use the results as one input into your broader security program.

208 checks across seven areas. The assessment covers Amazon Bedrock, Amazon SageMaker AI, Amazon Bedrock AgentCore, AWS Agent Registry, always-on Agentic AI Security, optional Responsible AI GRC checks, and optional OWASP Top 10 for LLM checks. Other AI/ML services (Amazon Comprehend, Amazon Rekognition, Amazon Textract, and others) are not currently assessed.

AWS partition support. The deployment and assessment are validated only in the standard AWS commercial partition (aws). AWS GovCloud (US) (aws-us-gov) and AWS China (aws-cn) are not currently validated or supported. Partition-aware ARN handling and region discovery in parts of the codebase do not constitute end-to-end support for those partitions.

Quick Start

Architecture

Architecture

Prerequisites

Single-Account Deployment

  1. Download the aiml-security-single-account.yaml CloudFormation template.
  2. Deploy to AWS CloudFormation
  3. Upload the template and provide a stack name.
  4. Optionally specify your email address to receive notifications.

  5. (Optional) Multi-Region: Set TargetRegions to scan multiple regions:

    • Leave empty to scan only the deployment region (default)
    • Comma- or space-separated list (for example, us-east-1,us-west-2,eu-west-1 or us-east-1 us-west-2 eu-west-1)
  6. Review the optional security policy baselines, especially the Marketplace endpoint CMK requirement, which defaults to enabled.
  7. Acknowledge IAM capabilities and click Submit.
  8. Once complete, CodeBuild automatically runs the assessment.
  9. View results: go to the stack Outputs tab → copy AssessmentBucket → open the report under the /{account_id}/ prefix in that S3 bucket.

Tip: The deployment creates two stacks. Your results are in the stack you named, not the auto-generated aiml-sec-* stack. See Troubleshooting for details.

Multi-Account Deployment

Step 1: Deploy Member Roles

Deploy 1-aiml-security-member-roles.yaml to all target accounts using CloudFormation StackSets with service-managed permissions.

  1. Navigate to CloudFormation > StackSets in the AWS Organizations management account or delegated administrator account
  2. Upload the template and set ManagementAccountID to the account ID where the central multi-account CodeBuild project runs
  3. Select Service-managed permissions and target your OUs
  4. Select your target region and submit

Step 2: Deploy Central Infrastructure

Deploy 2-aiml-security-codebuild.yaml in your central assessment account. This can be your AWS Organizations management account or a delegated administrator/central tooling account.

  1. Upload the template and set MultiAccountScan to true
  2. Optionally set TargetRegions for multi-region scanning
  3. Optionally provide an email address for notifications
  4. Configure the optional security policy baselines; the central values are propagated to every per-account SAM deployment
  5. Acknowledge IAM capabilities and submit
  6. Stack creation automatically triggers the assessment across all accounts

Upgrading an Existing Deployment

The required upgrade steps depend on which files changed. For a code-only fix, running the existing CodeBuild project is normally sufficient. The top-level CloudFormation templates and multi-account member-role StackSet only need to be updated when the corresponding templates changed.

Files changed in the target release Required action
aiml-security-assessment/functions/**, aiml-security-assessment/statemachine/**, either AWS SAM template*.yaml, Lambda requirements.txt, buildspec.yml, or consolidate_html_reports.py Run CodeBuild so it builds and updates the AWS SAM assessment stack
deployment/aiml-security-single-account.yaml Update the single-account infrastructure stack
deployment/1-aiml-security-member-roles.yaml Update every targeted multi-account member-role StackSet instance before running CodeBuild
deployment/2-aiml-security-codebuild.yaml Update the multi-account central infrastructure stack
Only documentation, tests, examples, or GitHub workflow files No deployed-resource update is required

Use the templates and source from the same release. GitHubBranch accepts a branch, tag, or commit; an immutable release tag or commit is recommended for reproducible deployments.

Determine what changed

Check CHANGELOG.md first. Its Unreleased or target-version Deployment impact section identifies the required actions. If the changelog does not cover the exact revisions being compared, infer the actions directly from the repository by comparing the commit used by the last successful build with the target revision:

git diff --name-only <deployed-commit>..<target-tag-or-commit> -- \
  deployment/ \
  aiml-security-assessment/ \
  buildspec.yml \
  consolidate_html_reports.py

For a branch such as main, use the resolved source commit shown by the last successful CodeBuild execution—not merely the branch name—as <deployed-commit>.

If the deployment is pinned to a tag or commit, update the infrastructure stack’s GitHubBranch parameter before starting CodeBuild. When its CloudFormation template did not change, this can be a parameter-only update using the current template. A deployment tracking a moving branch such as main can pull the latest code by starting CodeBuild without a stack update.

Single-account upgrade

  1. If aiml-security-single-account.yaml changed, update the existing infrastructure stack by replacing its template with the target release’s version. Do not create a second stack.
  2. If the source is pinned, update GitHubRepoUrl or GitHubBranch as needed. If the infrastructure template did not change, use the current template and change only these parameters.
  3. Preserve all other parameter values unless intentionally changing the assessment configuration.
  4. For a template update, review the change set, including IAM changes, acknowledge CAPABILITY_NAMED_IAM, and wait for UPDATE_COMPLETE.
  5. If deployable assessment code changed, manually start CodeBuild and confirm that it updates the existing aiml-sec-{account_id} AWS SAM stack.

Multi-account upgrade

Perform only the applicable steps, in this order:

  1. If 1-aiml-security-member-roles.yaml changed, update the existing CloudFormation StackSet with the target release’s template. Preserve ManagementAccountID, deployment targets, regions, and other settings. Wait until every targeted StackSet instance reports success.
  2. If 2-aiml-security-codebuild.yaml changed, update the existing central infrastructure stack by replacing its template with the target release’s version.
  3. If the source is pinned, update GitHubRepoUrl or GitHubBranch as needed. If the central template did not change, use the current template and change only these parameters.
  4. Preserve all other parameter values unless deliberately changing the deployment. For a template update, review IAM changes, acknowledge CAPABILITY_NAMED_IAM, and wait for UPDATE_COMPLETE.
  5. If deployable assessment code changed, manually start CodeBuild and confirm that it updates the existing per-account AWS SAM stacks.

When the member-role template changed, update it before CodeBuild runs so the central build uses the release’s intended cross-account deployment, execution polling, and report-retrieval permissions. Assessment API permissions are deployed on the SAM-created Lambda execution roles.

The custom resource in the infrastructure templates starts CodeBuild only when the infrastructure stack is initially created. It does not start a new build for stack updates, so manually start CodeBuild whenever deployable assessment code needs to be applied.

Users who deployed the AWS SAM templates directly, without the top-level CloudFormation templates, must build and deploy the new release’s aiml-security-assessment/template.yaml or aiml-security-assessment/template-multi-account.yaml to the existing stack name while preserving its parameter values.

For verification and additional detail, see Upgrading to a New Release.

Optional Security Policy Baselines

The deployment templates expose organization-specific baselines for checks that cannot infer your intended trust or hardening policy. Defaults preserve advisory behavior except for Marketplace endpoint customer-managed encryption, which is enforced by default.

CloudFormation parameter Default Affected check Behavior
RequireBedrockZeroDataRetention false BR-37 When true, the Bedrock account retention modes default and inherit fail the explicit zero-data-retention baseline. provider_data_share fails regardless of this setting.
RequireMarketplaceEndpointCMK true BR-40 When true, a Bedrock Marketplace model endpoint without a customer-managed KMS key fails. BR-40 uses kms:DescribeKey and requires KeyManager=CUSTOMER; AWS-managed keys do not pass. When false, a missing or AWS-managed key is reported as an informational N/A hardening advisory.
RequireAgentCoreOnlineEvaluation false AC-17 When true, missing or incomplete active AgentCore online evaluation coverage fails. When false, absent coverage is informational.
RequireAgentRegistryManualApproval false AR-03 When true, Agent Registry instances configured to approve all submitted records fail. When false, automatic approval is reported as an informational governance advisory.
RequireAgentRegistryCMK false AR-05 When true, registries using the default AWS owned encryption key fail. When false, AWS owned key encryption is reported as an informational hardening advisory; registries with a customer-managed KMS key pass.
AgentCoreTokenVaultId default AC-14 Selects the regional AgentCore Identity token vault whose customer-managed KMS encryption is assessed.
ApprovedExternalAccountIds Empty SM-30 Comma-separated 12-digit AWS account IDs approved to receive SageMaker Model Registry access. Accounts outside the configured boundary fail.
ApprovedOrganizationIds Empty SM-30 Comma-separated AWS Organizations IDs approved to receive SageMaker Model Registry access. Organizations outside the configured boundary fail.

For the approved-account and approved-organization lists, do not include spaces. Leaving both lists empty means SM-30 still detects public access, but external sharing that cannot be compared with an explicit organizational boundary is reported informationally.

When updating a stack with the AWS CLI, use a JSON parameter file for these comma-separated values. The CLI shorthand syntax also uses commas as field separators, so an unescaped ParameterValue=111122223333,444455556666 is not treated as one value.

Create params.json:

[
  {
    "ParameterKey": "ApprovedExternalAccountIds",
    "ParameterValue": "111122223333,444455556666"
  },
  {
    "ParameterKey": "ApprovedOrganizationIds",
    "ParameterValue": "o-a1b2c3d4e5,o-f6g7h8i9j0"
  }
]

Then pass the file to the stack update:

aws cloudformation update-stack \
  --stack-name <stack-name> \
  --use-previous-template \
  --capabilities CAPABILITY_NAMED_IAM \
  --parameters file://params.json

These parameters are available in both top-level deployment templates and both direct SAM templates. CodeBuild passes the selected values to every deployed assessment stack. Updating a value requires a CloudFormation stack update and a new assessment run.

Multi-Region Scanning

Both deployment modes support scanning multiple AWS regions in parallel via the TargetRegions parameter:

Value Behavior
Empty (default) Scans deployment region only — fully backward compatible
Comma- or space-separated (for example, us-east-1,us-west-2 or us-east-1 us-west-2) Scans those regions in parallel

Scanning uses a Step Functions Map state and runs up to MaxRegionConcurrency regions concurrently. This reduces elapsed time compared with sequential scanning, although total duration and AWS API usage still depend on the number of regions and resources assessed. Services unavailable in a region produce an informational N/A finding.

TargetRegions selects regions within the supported commercial partition; it does not enable cross-partition deployment or establish support for GovCloud or China regions.

The HTML report includes a Region column, filter dropdown, and “Risk by Region / Scope” summary.

Changing an existing deployment to multi-region? See Troubleshooting. This is a parameter-only change. For a new software release, follow the full upgrade procedure.

How It Works

  1. Deploy — CloudFormation creates CodeBuild, S3, IAM roles, and a Lambda trigger
  2. CodeBuild runs — builds and deploys the SAM assessment stack (per account in multi-account mode)
  3. Step Functions execute — orchestrates: S3 cleanup → IAM permission caching → resolve regions → Map state fans out across regions. Within each region, Bedrock, SageMaker, AgentCore, and AWS Agent Registry run in parallel; Responsible AI GRC runs once from the first region when either Responsible AI GRC or OWASP requires its source rows; OWASP then runs per region when enabled → generate consolidated report
  4. Results — HTML and CSV reports are stored in your S3 bucket

Selecting Service Assessments

Choose which direct service assessments run with these deployment parameters:

Parameter Assessment Default
EnableBedrockAssessment Amazon Bedrock true
EnableSageMakerAssessment Amazon SageMaker AI true
EnableAgentCoreAssessment Amazon Bedrock AgentCore true
EnableAgentRegistryAssessment AWS Agent Registry true

For a Bedrock-only run, leave EnableBedrockAssessment=true and set the other three parameters to false. The switches are available in both top-level CloudFormation deployment paths and both SAM templates. Existing deployments keep all four services enabled unless their parameters are changed.

Disabled service Lambdas are not invoked and do not produce CSV files. The HTML report labels their areas Not selected, rather than showing an assessed service with zero findings or N/A results. Selecting no direct services is supported and produces an explicit scope report even when there are no findings.

The Agentic AI lens contains only rows produced by the selected Bedrock, AgentCore, and Agent Registry assessments; disabling a source reduces its coverage. Responsible AI GRC and OWASP retain their separate opt-in switches. Responsible AI GRC still assesses deselected services and calls their APIs when enabled, including when it runs as an OWASP dependency. OWASP also runs its native checks. To limit execution to selected direct-service assessments, disable both EnableResponsibleAIGRCAssessment and EnableOWASPAssessment.

OWASP maps only selected Bedrock, SageMaker, and AgentCore CSVs plus its GRC source; Agent Registry does not feed OWASP. Each affected OWASP control includes an N/A/Informational coverage notice when direct-service evidence is omitted. A control that loses its only source (such as OW-07 when Bedrock is off) remains visible as unassessed. Findings from remaining sources retain their own status.

Compare pass rates only across equivalent assessment scopes: deselecting a well-configured service can lower the rate by removing passing controls from the denominator. Responsible AI GRC and derived lens/compliance rows are excluded from direct-service scores and remain visible in their own assessment areas.

The central reporting bucket preserves historical artifacts. An older CSV from a now-deselected service may remain there; consumers must select artifacts by execution ID, rather than treating every CSV in the bucket as current evidence.

Selection controls execution, not provisioning: the Lambda functions and their existing IAM roles remain deployed. Switches are resolved from the SAM stack’s parameters at the start of each execution; passing a different ServiceSelection in StartExecution does not override the deployment configuration. Update the stack and start a new assessment to change selection. In the top-level deployment paths, update the infrastructure parameters and then run CodeBuild.

Optional: Responsible AI GRC Checks (EnableResponsibleAIGRCAssessment)

The 64 Responsible AI GRC (FS-XX) checks are opt-in and default to false. Set the EnableResponsibleAIGRCAssessment deployment parameter to true when you want the additional Responsible AI GRC assessment. When enabled, the Responsible AI GRC assessment Lambda runs and its findings appear in a dedicated Responsible AI GRC section of the HTML report. When left false and OWASP is also disabled, no Responsible AI GRC findings are produced and the report omits that section entirely. The toggle is threaded into the Step Functions execution input (enableResponsibleAIGRC); the Responsible AI GRC Lambda is always deployed and is invoked when either enableResponsibleAIGRC or enableOWASP is true. In the OWASP-only case, its FS-* findings are hidden source rows rather than a customer-visible Responsible AI GRC section. A legacy EnableFinServAssessment parameter is also available as an alias — see Responsible AI GRC alias migration guide.

Deployment path note. The EnableResponsibleAIGRCAssessment parameter is wired through the CodeBuild-based deployment templates (deployment/aiml-security-single-account.yaml and deployment/2-aiml-security-codebuild.yaml), which thread it into every Step Functions start-execution call as enableResponsibleAIGRC. This is the supported install path. If you instead deploy aiml-security-assessment/template.yaml directly with sam deploy and start executions yourself, the state machine has no built-in trigger, so Responsible AI GRC checks stay off unless you include "enableResponsibleAIGRC": "true" in the execution input you pass to StartExecution.

Optional: OWASP Top 10 for LLM Checks (EnableOWASPAssessment)

The 12 OWASP Top 10 for LLM (OW-XX) checks are opt-in and default to false. Set the EnableOWASPAssessment deployment parameter to true when you want the additional compliance-standard assessment. When enabled, the OWASP Lambda runs per region after the Bedrock/SageMaker/AgentCore/AWS Agent Registry/Responsible AI GRC Lambdas complete: it reads Bedrock, SageMaker, and AgentCore per-region CSVs plus the Responsible AI GRC execution-scoped CSV when needed, applies mapping rules to emit OW-01..OW-10 rows derived from existing findings, and runs two net-new checks for LLM07 (System Prompt Leakage). Findings appear in a new “By Compliance Standard” sidebar section of the HTML report. When left false, no OWASP findings are produced and the section is omitted entirely. The toggle is threaded into the Step Functions execution input (enableOWASP); the OWASP Lambda is always deployed but is invoked only when the flag is true.

OWASP → Responsible AI GRC dependency (transparent to users). Roughly two-thirds of the OWASP mapping rows — including all of LLM05 (Improper Output Handling) — derive from the Responsible AI GRC (FS-XX) checks. To provide the Responsible AI GRC evidence used by OWASP, the state machine automatically runs the Responsible AI GRC Lambda whenever EnableOWASPAssessment=true, even when EnableResponsibleAIGRCAssessment=false. When the customer did not enable Responsible AI GRC explicitly, its findings are used only to power the OW-XX mappings, are hidden from the report UI — no Responsible AI GRC nav item, service card, or section appears — and the raw responsible_ai_grc_security_report_*.csv is not copied to the customer-facing report bucket. Setting both flags to true surfaces the Responsible AI GRC section and CSV normally.

The “By Compliance Standard” section is extensible: adding NIST AI RMF (EnableNISTAssessment) or EU AI Act (EnableEUAIActAssessment) later follows the same pattern.

Scope and limitations

Changes Since Last Assessment (EnableAssessmentHistory)

After each run, the framework compares each account’s findings with that account’s previous usable run and writes security_assessment_changes_<timestamp>.html and .csv next to the run’s main report. Each finding is labeled Resolved, Still open, Regressed, New, No longer reported, or No longer assessed. This is on by default; set the EnableAssessmentHistory deployment parameter to false to turn it off. The first run of an account has nothing to compare with and is skipped. The comparison reads only the findings CSVs already in the bucket and can’t fail an assessment run. See Changes Since Last Assessment for how runs are chosen and matched, and how to read the report.

For detailed architecture, execution flow, and extension guidance, see the Developer Guide.

Viewing Results

  1. Open your infrastructure stack in CloudFormation → Outputs tab → copy AssessmentBucket
  2. Navigate to that S3 bucket
  3. For single-account, open {account_id}/security_assessment_single_account_*.html
  4. For multi-account, open consolidated-reports/security_assessment_multi_account_*.html
  5. To see what changed since an account’s previous run, open {account_id}/security_assessment_changes_*.html (see Changes Since Last Assessment)

Assessment Execution Process

Automatic Trigger

Multi-Account Orchestration

  1. Account Discovery: AWS CodeBuild queries AWS Organizations for active accounts
  2. Role Assumption: Assumes AIMLSecurityMemberRole in each target account

  3. Module Deployment: Deploys the AI/ML assessment module:

    • Amazon Bedrock Assessment AWS Lambda
    • Amazon SageMaker AI Assessment AWS Lambda
    • Amazon Bedrock AgentCore Assessment AWS Lambda
    • Responsible AI GRC Assessment AWS Lambda
    • OWASP Top 10 for LLM Assessment AWS Lambda
    • Amazon S3 Cleanup AWS Lambda
    • AWS IAM Permission Caching AWS Lambda
    • Region Resolution AWS Lambda
    • Consolidated Report Generation AWS Lambda
  4. Assessment Execution: AWS Step Functions orchestrate parallel AWS Lambda execution
  5. Results Collection: Individual AWS Lambda functions store results in local Amazon S3 buckets
  6. Consolidation: AWS CodeBuild collects and consolidates results from all accounts
  7. Reporting: Generates multi-account HTML and CSV reports
  8. Notification: Sends completion notification through Amazon SNS (if configured)

Monitoring and Results

You can check the AWS CodeBuild console to confirm the assessment completed successfully before accessing the results.

Accessing Results

  1. Find the Amazon S3 Bucket Name:

    • Navigate to AWS CloudFormation > Stacks in the AWS Console
    • For single-account deployments using the standalone template (aiml-security-single-account.yaml), select the stack you deployed (for example, aiml-security-single-account) and find the AssessmentBucket output. Results are synced to this bucket under the {account_id}/ prefix.
    • For multi-account deployments, select the aiml-security-multi-account stack created in Step 2: Deploy Central Infrastructure and find the AssessmentBucket output
    • Go to the Outputs tab
    • Copy the Amazon S3 bucket name

    Note: The deployment creates multiple Amazon S3 buckets. Only use the bucket from the AssessmentBucket output above. Other buckets (such as aiml-sec-*-aimlassessmentbucket-* from nested stacks or aws-sam-cli-managed-* for deployment artifacts) are for internal use and can be ignored.

  2. Navigate to the Amazon S3 Bucket:

    • Go to Amazon S3 in the AWS Console
    • Search for and open your assessment bucket
    • For single-account deployments, open the {account_id}/ folder and then open the security_assessment_single_account_YYYYMMDD_HHMMSS.html report
    • For multi-account deployments, follow the Report Structure guidance below

Report Structure

Consolidated Reports

Individual Account Reports

Understanding Results

Severity Meaning
High Critical — immediate action required
Medium Important — should be addressed
Low Minor — best practice optimization
Informational Advisory — no action required
Status Meaning
Failed Security issue identified
Passed Resource meets best practice
N/A Not applicable, advisory/manual review, unavailable API or region, or assessment could not determine a result

How finding severities are determined

Responsible AI GRC (FS-) check severities are assigned by a documented, reproducible methodology rather than per-check intuition. Each control is scored on two axes — Impact (harm if the control is absent) and Likelihood (probability the adverse outcome occurs given the control is absent) — and the pair is mapped to a severity via a 3×3 matrix. The labels align with the AWS Security Hub ASFF severity scale, so findings can be forwarded to Security Hub with consistent severities:

Label ASFF normalized Meaning
Informational 0 No actionable issue (control not applicable, advisory/manual-review, or could-not-assess context)
Low 1–39 Does not require action on its own; compensating controls exist
Medium 40–69 Should be addressed, but not urgently
High 70–89 Should be addressed as a priority

Severity is a property of the control (its inherent risk), so a check’s Passed and Failed rows carry the same severity. The N/A family is fixed by disposition: not-applicable and advisory findings are Informational; could-not-assess (access-denied / unsupported region) findings are Low. Critical is reserved and not currently emitted.

For the full methodology (matrix, factor definitions, disposition rules) and the authoritative per-finding assignments, see Responsible AI GRC Severity Methodology and the Responsible AI GRC Severity Register. Mappings are preliminary — validate with your MRM/Legal/Compliance teams before relying on them as audit evidence.

Customization

Task How
Add new accounts Add to StackSet deployment targets
Modify assessment runtime permissions Edit the specific Lambda policy in both SAM templates
Modify deployment or cross-account permissions Edit the applicable deployment/*.yaml template
Adjust concurrency Change ConcurrentAccountScans parameter
Add new service checks See Developer Guide

Permissions Required

The deployment uses multiple IAM roles with different trust and permission boundaries. They are not all read-only.

If you need to reduce scope, review the role policies in:

Documentation

Document Description
Changelog User-facing changes and required deployment actions for unreleased work and tagged versions
Security Checks Reference Complete reference for all 208 security checks with severity levels
OWASP Top 10 for LLM Checks Complete OW-01..12 reference: mapping-derived OWASP LLM01..LLM10 rows, native LLM07 checks, source dependencies, references, and status semantics
Responsible AI GRC Scope What Responsible AI GRC is and is not, its relationship to the AWS Well-Architected Responsible AI Lens, the per-bucket source catalog, check-count reconciliation, terminology, and the compatibility policy for preserved identifiers
Responsible AI GRC Checks Complete FS-01..69 reference: shared introduction, severity rubric, upstream-overlap table, compliance framework mapping, and all check definitions (Part 1 infrastructure controls, Part 2 guardrails & content safety, Part 3 app-layer controls & gaps)
Responsible AI GRC Severity Methodology Likelihood × Impact → ASFF severity model, disposition rules, and research basis for FS check severities
Responsible AI GRC Severity Register Authoritative per-finding severity assignments (the single source of truth enforced by the drift-guard test)
Responsible AI GRC Compliance Mappings Preliminary mapping of FS checks to SR 11-7, FFIEC CAT, NYDFS 500, PCI-DSS, DORA, MAS TRM, ISO 27001, ECOA, and OWASP LLM Top 10
Changes Since Last Assessment How each run is compared with the previous run, the change states, the report, and the EnableAssessmentHistory setting
Troubleshooting Guide Common issues, stack identification, upgrade guide, debugging
Developer Guide Architecture details, adding custom checks, and contributing
Cleanup Guide Step-by-step resource removal instructions

CI/CD

GitHub Actions workflows run automatically on pull requests and selected pushes:

Workflow Trigger What It Checks
Python Code Quality PR ruff check and ruff format --check on changed Python files
AI/ML Security Assessment Tests PR, push to main/develop Runs the pytest suite (assessment functions and report pipeline) on Python 3.12
CloudFormation Lint PR Validates deployment and SAM templates with cfn-lint
SAM Validate & Build PR sam validate --lint and sam build on SAM templates
ASH Security Scan PR Scans for secrets, dependency vulnerabilities, and IaC misconfigurations
ASH Full Repository Scan Push to main, monthly Full repository security scan

Contributing

We welcome community contributions! See the Developer Guide for guidelines.

Security

See CONTRIBUTING for reporting security issues.

License

This library is licensed under the MIT-0 License. See the LICENSE file.

This site is open source. Improve this page.