This guide covers common issues, debugging tips, and frequently asked questions for the AI/ML Security Assessment framework.
Symptoms: StackSet instances fail to create in member accounts.
Solutions:
Symptoms: “Access Denied” errors when assuming roles in member accounts.
Solutions:
AIMLSecurityMemberRole exists in target accountsManagementAccountID parameter matches your management accountSymptoms: CodeBuild fails during the SAM deploy phase.
Solutions:
Symptoms: AWS Step Functions show failed state or timeout.
Solutions:
Symptoms: CloudFormation blocks stack creation with this error.
Cause: A resource with the same physical name already exists outside of CloudFormation management, typically from a failed deployment.
Solution:
# Find the orphaned bucket
aws s3 ls | grep aiml-security
# Empty the bucket
aws s3 rm s3://<bucket-name> --recursive
# Delete version markers if versioned
aws s3api delete-objects --bucket <bucket-name> --delete \
"$(aws s3api list-object-versions --bucket <bucket-name> \
--query '{Objects: Versions[].{Key:Key,VersionId:VersionId}}')"
# Delete the bucket
aws s3 rb s3://<bucket-name>
# Re-run the CodeBuild project
Symptoms: CodeBuild job times out or runs slowly when scanning a large number of accounts concurrently.
Cause: The ConcurrentAccountScans parameter controls both the number of parallel account scans and the CodeBuild compute type. Higher concurrency requires a larger (and more expensive) instance:
| ConcurrentAccountScans | Parallel Accounts | CodeBuild Compute Type | Approximate Cost per Build Minute |
|---|---|---|---|
| Three (default) | 3 | BUILD_GENERAL1_SMALL |
$0.005 |
| Six | 6 | BUILD_GENERAL1_MEDIUM |
$0.01 |
| Twelve | 12 | BUILD_GENERAL1_LARGE |
$0.02 |
Solutions:
ConcurrentAccountScans to process more accounts in parallel – but be aware this also increases the per-minute CodeBuild costCodeBuildTimeout parameter (default is 300 minutes for multi-account)MultiAccountListOverride to split assessments into batchesSymptoms: Assessment completes but no HTML/CSV files appear.
Solutions:
{account_id}/ for single-account, consolidated-reports/ for multi-accountAIMLSecurityCodeBuild or AIMLSecurityMultiAccountCodeBuild)# In the member account, check the role trust policy
aws iam get-role --role-name AIMLSecurityMemberRole --query 'Role.AssumeRolePolicyDocument'
The trust policy should allow the central CodeBuild role:
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<management-account-id>:root"
},
"Action": "sts:AssumeRole",
"Condition": {
"ArnEquals": {
"aws:PrincipalArn": "arn:aws:iam::<management-account-id>:role/service-role/MultiAccountCodeBuildRole"
}
}
}
Verify the bucket policy allows cross-account writes for multi-account deployments:
aws s3api get-bucket-policy --bucket <assessment-bucket-name>
AIMLAssessmentStateMachineQ: Does this assessment make any changes to my AWS resources?
A: No. All security checks are read-only. The framework only queries your resources to evaluate their configurations. It does not create, modify, or delete any of your AI/ML workloads or data.
Q: How long does an assessment take to run?
A:
The assessment runs in parallel across accounts to minimize total execution time.
Q: How often should I run security assessments?
A:
You can automate regular assessments using Amazon EventBridge scheduled rules.
Q: What AWS regions are supported?
A: The framework supports all standard AWS commercial regions where Amazon Bedrock, Amazon SageMaker AI, or Amazon Bedrock AgentCore are available. AWS GovCloud and AWS China regions may require template modifications.
Q: Does this work if I don’t have any AI/ML resources deployed yet?
A: Yes. The assessment runs successfully and reports findings with status “N/A” (Not Applicable) for checks where no resources exist to assess. This is useful for establishing a security baseline before deploying AI/ML workloads.
Q: How much does it cost to run this assessment?
A: Estimated cost per assessment: $0.50 - $2.00 for typical single-account usage
Cost breakdown:
Multi-account deployments: AWS Lambda and AWS Step Functions costs scale with the number of accounts. AWS Organizations API calls are free. AWS CodeBuild cost depends on the ConcurrentAccountScans setting, which determines the instance size:
| ConcurrentAccountScans | CodeBuild Compute Type | Approximate Cost per Build Minute |
|---|---|---|
| Three (default) | BUILD_GENERAL1_SMALL |
$0.005 |
| Six | BUILD_GENERAL1_MEDIUM |
$0.01 |
| Twelve | BUILD_GENERAL1_LARGE |
$0.02 |
For example, a 30-minute multi-account assessment at “Twelve” concurrency costs roughly $0.60 in CodeBuild alone, compared to $0.15 at the default “Three.” Choose the concurrency level that balances speed against cost for your organization size.
Q: Are there any ongoing costs when not running assessments?
A: Minimal ongoing costs:
Q: Can I customize which security checks are included?
A: Currently, all 52 checks run by default to provide comprehensive coverage. You can filter results in the generated HTML reports by severity, status, or service. Future versions may support selective check execution.
Q: Can I add custom security checks?
A: Yes! See the Developer Guide for instructions on extending the framework with additional checks. The architecture is designed to be modular and extensible.
Q: Can I export results to other formats (JSON, CSV, SIEM)?
A: Yes. The framework generates:
You can integrate with SIEM tools by processing the CSV or JSON outputs from the Amazon S3 bucket.
Q: Can I schedule automated assessments?
A: Yes. Use Amazon EventBridge to trigger the AWS CodeBuild project on a schedule:
aws events put-rule \
--name "WeeklyAIMLAssessment" \
--schedule-expression "cron(0 2 ? * MON *)"
aws events put-targets \
--rule "WeeklyAIMLAssessment" \
--targets "Id"="1","Arn"="arn:aws:codebuild:region:account:project/your-project"
Q: The assessment completed but I don’t see any reports in my Amazon S3 bucket.
A: Common causes:
{account_id}/ prefix for single-account, consolidated-reports/ for multi-accountQ: I see “Access Denied” errors in the AWS CodeBuild logs.
A: This usually indicates:
AIMLSecurityMemberRole) is not deployed in target accounts through AWS CloudFormation StackSetsSolution: Verify AWS CloudFormation StackSet deployment in Step 1 completed successfully across all target accounts.
Q: The assessment is taking longer than expected.
A: Performance factors:
ConcurrentAccountScans parameter)If assessments consistently timeout, increase the AWS Lambda timeout in the AWS SAM template or reduce concurrent account scans.
Q: Where is my assessment data stored?
A: All assessment data remains entirely within your AWS account:
Q: What IAM permissions does the assessment role need?
A: The framework uses read-only permissions only:
List*, Describe*, Get* actionsSee the main README for the complete permission list.
Q: Is this assessment sufficient for compliance requirements (SOC 2, HIPAA, and similar)?
A: This assessment provides a security evaluation against AWS best practices and can support compliance efforts. However:
Consult with your compliance team to determine how this assessment fits into your overall compliance program.
Q: Does this framework comply with AWS Well-Architected Framework principles?
A: Yes. The assessment checks align with the AWS Well-Architected Framework Security Pillar, specifically: