Source

This page is generated from skills/eks-genai/references/security-and-compliance.md. Edit the source, not this page.

Security & Compliance for GenAI on EKS

Non-negotiable security baseline for every GenAI-on-EKS deployment, plus compliance-regime notes and escalation triggers. Every item in this file must be present in responses — no exceptions, no "we'll add security later."

---

The Non-Negotiable Security Baseline

These controls apply to every GenAI workload on EKS — training, inference, RAG, agentic — regardless of environment or compliance regime.

1. Pod Credentials — EKS Pod Identity / IRSA

Rule: Pods MUST use EKS Pod Identity (preferred, K8s 1.24+) or IRSA for AWS API access. NEVER static AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY in env vars, ConfigMaps, or baked into images.

# EKS Pod Identity association (preferred)
apiVersion: eks.amazonaws.com/v1
kind: PodIdentityAssociation
metadata:
  name: vllm-s3-access
spec:
  serviceAccountName: vllm-inference
  roleArn: arn:aws:iam::ACCOUNT:role/vllm-model-reader

Why for GenAI: model-serving pods need S3 (weights), Bedrock (gateway routing), and Secrets Manager (API keys). Static creds leak in logs, crash dumps, and large cached images.

The GenAI-on-EKS NVIDIA workshop uses pods.eks.amazonaws.com (Pod Identity) for all service accounts.

2. ECR Image Scanning

Rule: Enable ECR enhanced scanning (Inspector-powered) or integrate third-party (Snyk/Aqua/Wiz) in CI/CD. Block deployment of images with critical/high CVEs.

GenAI concern: DLCs and vLLM images carry massive dependency trees (PyTorch, CUDA, cuDNN, Neuron SDK). CVE surface is far larger than typical microservices. Scan on push AND weekly for base-image drift.

3. Secrets — Secrets Store CSI Driver

Rule: Store all secrets (API keys, model registry tokens, Langfuse keys) in AWS Secrets Manager or SSM Parameter Store. Mount via the Secrets Store CSI Driver. Avoid plain Kubernetes Secrets alone — even with KMS envelope encryption enabled (which addresses encryption-at-rest in etcd), a Secret is readable by anyone with namespace get/list secrets RBAC, isn't audited or rotated like Secrets Manager, and is trivially exposed if it lands in Git. The CSI-driver path keeps the source of truth in Secrets Manager (rotation, fine-grained IAM, CloudTrail audit) and never persists a Kubernetes Secret object. Never put secrets in ConfigMaps, env vars in Deployment specs, or baked into images.

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: genai-secrets
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "genai/litellm-master-key"
        objectType: "secretsmanager"
      - objectName: "genai/langfuse-secret-key"
        objectType: "secretsmanager"

4. Model Artifact Provenance

Rule: Verify integrity of every model artifact before serving.

Model source	Verification method
Baked into container image	Image signing via AWS Signer or Sigstore Cosign + admission controller (Kyverno/OPA) to reject unsigned images
Downloaded from Hugging Face	Verify SHA256 checksums against model card; pin exact revision hashes, not branch names
Downloaded from S3	Enable S3 Object Lock (compliance mode) for production artifacts; verify ETag/checksum

A compromised model artifact is the most dangerous GenAI supply-chain vector — it executes arbitrary inference on customer data. Treat model weights with the same rigor as application binaries.

5. Network Isolation

Rule: GPU/Neuron nodes in private subnets with no direct internet. All AWS API calls through VPC endpoints.

Required VPC endpoints for GenAI on EKS:

Endpoint	Why
com.amazonaws.REGION.s3 (Gateway)	Model weights, checkpoints, training data
com.amazonaws.REGION.bedrock-runtime (Interface)	LiteLLM → Bedrock calls
com.amazonaws.REGION.ecr.api + ecr.dkr	Image pull
com.amazonaws.REGION.secretsmanager	Secrets Store CSI
com.amazonaws.REGION.sts	Pod Identity / IRSA token exchange
com.amazonaws.REGION.logs	CloudWatch Logs
com.amazonaws.REGION.monitoring	CloudWatch Metrics / AMP remote-write

Egress to internet (HF download, pip) via NAT Gateway with restrictive SG — or eliminate by pre-caching all artifacts in S3/ECR.

In-cluster isolation (default-deny NetworkPolicy). VPC endpoints only govern traffic leaving the cluster; they do nothing for pod-to-pod reachability inside it. Self-hosted inference backends (vLLM, Triton) typically run without their own auth, so any pod that can reach the model service bypasses the gateway and its rate limiting, cost tracking, and guardrails. Apply a default-deny ingress NetworkPolicy in every standalone-vLLM inference namespace, then allow ingress to the model service only from the gateway namespace/pods:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: genai-inference   # adjust to your inference namespace (must match the gateway target in ai-gateway.md)
spec:
  podSelector: {}          # all pods in the namespace
  policyTypes:
    - Ingress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-gateway-to-vllm
  namespace: genai-inference   # adjust to your inference namespace (must match the gateway target in ai-gateway.md)
spec:
  podSelector:
    matchLabels:
      app: vllm-llama       # the vLLM model service pods
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: ai-gateway   # gateway namespace...
          podSelector:
            matchLabels:
              app: litellm                              # ...and only the gateway pods (least privilege)
      ports:
        - protocol: TCP
          port: 8000
    - from:                                             # allow ONLY the metrics scraper to reach vLLM /metrics
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: monitoring   # namespace running Prometheus (adjust to yours)
          podSelector:
            matchLabels:
              app.kubernetes.io/name: prometheus        # scraper pods only, NOT the whole namespace
      ports:
        - protocol: TCP
          port: 8000                                    # NOTE: vLLM serves the API and /metrics on the SAME port

Requires a NetworkPolicy-enforcing CNI (VPC CNI network policy, Cilium, or Calico). Caveats: (1) a default-deny that omits the scraper silently blocks Prometheus from reaching vLLM /metrics, so keep the scraper allow rule above. Because vLLM serves the API and /metrics on the same port (8000), that rule also grants the scraper API access, so scope it to the scraper pods (as shown). If that shared exposure matters, enable /metrics authentication, or accept it and keep the scraper rule tightly scoped. vLLM serves /metrics on the same port as the API and has no separate-metrics-port flag, so a dedicated metrics port isn't an option. With the CNIs listed (VPC CNI network policy, Cilium, or Calico), kubelet liveness/readiness probes are node-local and not governed by these ingress rules, so they keep working; this assumes DCGM Exporter runs outside this locked-down namespace (its own DaemonSet on a separate port, scraped independently, not through this rule). (2) This policy denies ingress only. If you add a companion default-deny Egress policy it must allow the workload's real dependencies or the pod fails closed: kube-dns (UDP/TCP 53, scoped to the kube-system DNS pods), plus the S3/AWS API endpoints (443) used for weight loading and the L2 cache endpoint (e.g. 6379), not DNS alone. This is the in-cluster complement to the gateway auth caveat in ai-gateway.md. (3) This example is scoped to standalone-vLLM namespaces. If you run vLLM inside Ray Serve / KubeRay in this namespace, the policy blocks Ray traffic (Ray pods don't carry app: vllm-llama), silently halting RayService reconciliation. Additionally allow intra-namespace Ray head↔worker traffic (GCS 6379, dashboard/health 8265, Serve proxy, object/worker ports) and KubeRay-operator→head ingress, or scope this policy to non-Ray namespaces.

6. Audit Logging

Rule: Enable and retain:

• CloudTrail (management + data events for S3 model buckets)
• EKS control plane audit logging (minimum: audit + authenticator log types)
• VPC Flow Logs
• Langfuse traces — full prompt/response audit trail for GenAI-specific compliance

Retention: 90 days hot (CloudWatch), 1 year cold (S3 Glacier) for regulated workloads.

7. Pod Security Admission + CIS-Hardened AMI

Rule: Enforce Pod Security Admission at restricted for all GenAI namespaces. Use namespace-level exceptions for device-plugin DaemonSets — never cluster-wide privileged.

apiVersion: v1
kind: Namespace
metadata:
  name: genai-inference
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/warn: restricted

For regulated workloads: CIS-hardened AMIs (AL2023 CIS Level 1 or Bottlerocket — hardened by design). EKS Auto Mode uses Bottlerocket by default.

---

Compliance-Regime Notes

HIPAA

• Model serving pods processing PHI: HIPAA-eligible EKS cluster (BAA in place).
• PHI in Langfuse traces: Langfuse must be in HIPAA-scoped environment with KMS CMK encryption at rest + access logging.
• Vector stores with PHI embeddings: use Aurora PostgreSQL (HIPAA-eligible) + PGVector; verify Bedrock KB HIPAA status at deployment time.
• Encrypt all EBS volumes and S3 buckets with KMS CMK (not default SSE-S3).

PCI-DSS

• Isolate GenAI workloads in dedicated cluster (or dedicated node group + namespace + network policies) within CDE boundary.
• All pod-to-pod traffic encrypted — mTLS via service mesh or native pod encryption.
• Quarterly vulnerability scans of all DLC/vLLM images; retain scan reports.
• RBAC + IAM identity mapping restricting kubectl to GenAI namespace.

FedRAMP (Moderate / High)

• Deploy in GovCloud or FedRAMP-authorized commercial regions.
• FIPS 140-2 validated modules — FIPS-enabled AMIs; TLS libraries in DLCs support FIPS mode.
• Continuous monitoring: EKS audit logs + CloudTrail → organization SIEM.
• Supply chain: all base images from approved registry (ECR in same boundary); no Docker Hub/HF pulls in production.

GDPR

• RAG stores with EU personal data: deploy in EU region (eu-west-1, eu-central-1).
• Right-to-erasure for vector embeddings — deletion of source documents must cascade to deletion of derived embeddings (design upfront).
• Langfuse traces containing user prompts are personal data — configure retention + ensure same-region deployment.

---

When to Escalate to SpecReq

1. Regulated compliance (HIPAA/PCI/FedRAMP/GDPR) AND GenAI processes regulated data — Specialist compliance review required. GenAI compliance is materially harder (prompts, outputs, embeddings are all data-processing activities).
2. Multi-tenant SaaS with cross-tenant data isolation — KV-cache isolation, prompt leakage, per-tenant vector partitioning need Security + ML TFC joint review.
3. Agentic with autonomous code execution — sandbox escape risk requires Security TFC review.
4. Air-gapped environment with no VPC endpoint path — custom supply chain design.
5. Model distribution to third parties — legal + security review beyond deployment hardening.

---

Quick-Reference Checklist

Include in every GenAI-on-EKS response:

• Pod Identity / IRSA — no static credentials
• ECR image scanning — critical/high CVEs blocked
• Secrets via Secrets Manager + Secrets Store CSI
• Model provenance — image signing or checksum verification
• Private subnets + VPC endpoints (S3, Bedrock, ECR, STS, Secrets Manager)
• Default-deny NetworkPolicy: inference backends reachable only from the gateway
• CloudTrail + EKS audit logs + VPC Flow Logs
• Pod Security Admission restricted + CIS-hardened AMI
• Langfuse traces encrypted + retained per compliance requirement

---

Sources

• EKS AI/ML Best Practices
• EKS Pod Identity
• Secrets Store CSI Driver
• Pod Security Admission
• AWS Signer
• EKS Control Plane Logging
• Container Security Hardening for EKS