Security Checks Reference
This document provides a comprehensive reference for all 51 security checks performed by the AI/ML Security Assessment framework.
Table of Contents
Overview
The framework evaluates your AI/ML workloads against AWS security best practices across three services:
| Service |
Number of Checks |
Focus Areas |
| Amazon SageMaker AI |
25 |
Security Hub controls, encryption, network isolation, IAM, MLOps |
| Amazon Bedrock |
13 |
Guardrails, encryption, VPC endpoints, IAM permissions, logging |
| Amazon Bedrock AgentCore |
13 |
VPC configuration, encryption, observability, resource policies |
Check ID Convention
Each security check has a unique identifier with a service prefix:
| Prefix |
Service |
Example |
| SM-XX |
Amazon SageMaker |
SM-01, SM-25 |
| BR-XX |
Amazon Bedrock |
BR-01, BR-13 |
| AC-XX |
Amazon Bedrock AgentCore |
AC-01, AC-13 |
Severity Levels
| Severity |
Description |
Action Required |
| High |
Critical security issues that could lead to data exposure, unauthorized access, or compliance violations |
Immediate remediation recommended |
| Medium |
Important security improvements that strengthen your security posture |
Address in next maintenance window |
| Low |
Minor optimizations and best practice recommendations |
Address when convenient |
| Informational |
Advisory information about your configuration |
No action required |
| N/A |
Check not applicable (no resources to assess) |
No action required |
Status Values
| Status |
Description |
| Failed |
Security issue identified that requires remediation |
| Passed |
Checked resources met the assessed best practice at time of scan |
| N/A |
No resources exist to check (for example, no notebooks, no guardrails configured) |
Amazon SageMaker AI Security Checks (25)
SM-01: Internet Access
- Severity: High
- AWS Security Hub Control: SageMaker.2
- Description: Checks for direct internet access on notebooks and domains.
SM-02: AWS IAM Permissions
- Severity: High
- Description: Identifies overly permissive policies, stale access, and IAM Identity Center configuration.
SM-03: Data Protection
- Severity: High
- AWS Security Hub Control: SageMaker.1
- Description: Verifies encryption at rest and in transit for notebooks and domains.
SM-04: Amazon GuardDuty Integration
- Severity: Medium
- Description: Verifies Amazon GuardDuty runtime threat detection is enabled.
SM-05: MLOps Features
- Severity: Low
- Description: Checks MLOps pipelines, experiment tracking, and model registry usage.
SM-06: Clarify Usage
- Severity: Low
- Description: Validates SageMaker Clarify for bias detection and explainability.
SM-07: Model Monitor
- Severity: Medium
- Description: Checks Model Monitor configuration for drift detection.
SM-08: Model Registry
- Severity: Medium
- Description: Validates model registry usage and permissions.
SM-09: Notebook Root Access
- Severity: High
- AWS Security Hub Control: SageMaker.3
- Description: Validates root access is disabled on notebooks.
SM-10: Notebook Amazon VPC Deployment
- Severity: High
- AWS Security Hub Control: SageMaker.2
- Description: Ensures notebooks are deployed within an Amazon VPC.
SM-11: Model Network Isolation
- Severity: Medium
- AWS Security Hub Control: SageMaker.4
- Description: Checks inference containers have network isolation.
SM-12: Endpoint Instance Count
- Severity: Medium
- AWS Security Hub Control: SageMaker.5
- Description: Verifies endpoints have 2+ instances for high availability.
SM-13: Monitoring Network Isolation
- Severity: Medium
- Description: Checks monitoring job network isolation.
SM-14: Model Container Repository
- Severity: Medium
- Description: Validates model container repository access.
SM-15: Feature Store Encryption
- Severity: High
- Description: Checks feature group encryption settings.
SM-16: Data Quality Encryption
- Severity: Medium
- Description: Validates data quality job encryption.
SM-17: Processing Job Encryption
- Severity: Medium
- Description: Verifies processing job encryption.
- Severity: Medium
- Description: Checks transform job volume encryption.
SM-19: Hyperparameter Tuning Encryption
- Severity: Medium
- Description: Validates hyperparameter tuning job encryption.
SM-20: Compilation Job Encryption
- Severity: Medium
- Description: Checks compilation job encryption.
SM-21: AutoML Network Isolation
- Severity: Medium
- Description: Validates AutoML job network isolation.
SM-22: Model Approval Workflow
- Severity: Medium
- Description: Checks model approval and governance workflow.
SM-23: Model Drift Detection
- Severity: Medium
- Description: Validates model drift monitoring configuration.
SM-24: A/B Testing and Shadow Deployment
- Severity: Low
- Description: Checks for safe deployment patterns.
SM-25: ML Lineage Tracking
- Severity: Low
- Description: Validates experiment tracking and lineage.
Amazon Bedrock Security Checks (13)
BR-01: AWS IAM Least Privilege
- Severity: High
- Description: Identifies roles with AmazonBedrockFullAccess policy.
BR-02: Amazon VPC Endpoint Configuration
- Severity: High
- Description: Validates Bedrock Amazon VPC endpoints exist for private connectivity.
BR-03: Marketplace Subscription Access
- Severity: Medium
- Description: Checks for overly permissive marketplace subscription access.
BR-04: Model Invocation Logging
- Severity: Medium
- Description: Checks invocation logging is enabled.
BR-05: Guardrail Configuration
- Severity: High
- Description: Verifies guardrails are configured and enforced.
BR-06: AWS CloudTrail Logging
- Severity: Medium
- Description: Validates AWS CloudTrail logging for Bedrock API calls.
BR-07: Prompt Management
- Severity: Low
- Description: Validates Bedrock Prompt template usage and variants.
BR-08: Agent AWS IAM Configuration
- Severity: Medium
- Description: Checks agent execution role permissions.
BR-09: Knowledge Base Encryption
- Severity: High
- Description: Checks knowledge base encryption settings.
BR-10: Guardrail AWS IAM Enforcement
- Severity: Medium
- Description: Verifies guardrails are enforced through AWS IAM conditions.
BR-11: Custom Model Encryption
- Severity: High
- Description: Validates custom models use customer-managed AWS KMS keys.
BR-12: Invocation Log Encryption
- Severity: Medium
- Description: Verifies logs are encrypted with AWS KMS.
BR-13: Flows Guardrails
- Severity: Medium
- Description: Validates Bedrock Flows have guardrails attached.
Amazon Bedrock AgentCore Security Checks (13)
AC-01: Runtime Amazon VPC Configuration
- Severity: High
- Description: Validates agent runtimes have proper Amazon VPC settings.
AC-02: AWS IAM Full Access
- Severity: High
- Description: Checks for overly permissive AgentCore AWS IAM policies.
AC-03: Stale Access
- Severity: Low
- Description: Detects unused AgentCore permissions.
AC-04: Observability
- Severity: Medium
- Description: Verifies Amazon CloudWatch Logs and AWS X-Ray tracing configuration.
AC-05: Amazon ECR Repository Encryption
- Severity: High
- Description: Validates Amazon ECR repositories use encryption.
- Severity: Medium
- Description: Checks storage configuration for browser tools.
AC-07: Memory Encryption
- Severity: High
- Description: Checks agent memory encryption with AWS KMS.
AC-08: Amazon VPC Endpoints
- Severity: High
- Description: Validates Amazon VPC endpoints for AgentCore services.
AC-09: Service-Linked Role
- Severity: Medium
- Description: Verifies the AgentCore service-linked role exists.
AC-10: Resource-Based Policies
- Severity: Medium
- Description: Checks runtime and gateway resource policies.
AC-11: Policy Engine Encryption
- Severity: Medium
- Description: Validates policy engine encryption settings.
AC-12: Gateway Encryption
- Severity: Medium
- Description: Verifies gateway encryption settings.
AC-13: Gateway Configuration
- Severity: Medium
- Description: Validates gateway security configuration.
Additional Resources