EKS Operations Review Report¶

Cluster: eks-review-cluster (us-east-1)¶

Generated: 2026-04-06 Review Type: Comprehensive Best Practices Assessment

Executive Summary¶

The EKS cluster eks-review-cluster was assessed across 6 operational domains with a total of 72 checks evaluated. 23 checks passed and 49 checks failed, yielding an overall compliance score of approximately 31.9%. This represents a decline from the previous assessment (51.4%), primarily due to the addition of two new domains (Resiliency and Security) which surfaced significant gaps.

The cluster has critical operational risks including an unrestricted public API endpoint, no monitoring solution, widespread lack of health probes, missing resource limits, and workloads running as root with privilege escalation allowed. Immediate action is required on security hardening and resiliency improvements.

Issue Breakdown by Category¶

Security: 6 passed, 19 failed
Resiliency: 6 passed, 23 failed
Networking: 4 passed, 5 failed
Autoscaling (Karpenter): 4 passed, 5 failed
Autoscaling (Cluster Autoscaler): 1 passed, 0 failed
Observability: 2 passed, 3 failed

Priority Classification¶

Critical: 0 issues requiring immediate action
High: 27 issues impacting production readiness
Medium: 16 issues for operational improvement
Low: 6 issues for optimization

Quick Wins (fixable in <30 minutes)¶

#	Finding	Fix Type	Effort	Command Preview
1	Check if the EKS Control Plane logs are enabled	AWS CLI	~5 min	`aws eks update-cluster-config --name eks-review-cluster --logging ...`
2	Enable envelope encryption for Kubernetes Secrets	AWS CLI	~10 min	`aws eks associate-encryption-config --cluster-name eks-review-cluster ...`
3	Leverage EKS Pod Identity	AWS CLI	~10 min	`aws eks create-addon --cluster-name eks-review-cluster --addon-name eks-pod-identity-agent`
4	EKS Node Monitoring Agent	AWS CLI	~5 min	`aws eks create-addon --cluster-name eks-review-cluster --addon-name eks-node-monitoring-agent`
5	Configure the Limits in the karpenter node pools	Manifest	~5 min	`kubectl edit nodepool spot-pool`

Requires Planning (maintenance window or manifest changes)¶

Make the EKS Cluster Endpoint private — Requires VPN/Direct Connect connectivity before switching; could lock out users without private network access.
Run the application as a non-root user — 54 workloads (161 pods) need SecurityContext changes; requires testing per application.
Do not allow privileged escalation — 53 workloads (159 pods) need SecurityContext updates; may break workloads relying on elevated privileges.
Configure Resource Requests/Limits — 53 deployments need resource specifications; requires load testing to determine appropriate values.
Use liveness probes / readiness probes — 55 workloads need probe definitions; requires understanding of each application's health endpoints.
Worker nodes spread across multiple AZs — Node group reconfiguration needed; involves draining and rescheduling workloads.

Trend Analysis¶

Metric	Previous (2026-04-03)	Current (2026-04-06)
Overall Compliance	51.4%	~31.9%
Autoscaling — Cluster Autoscaler	64.3%	100.0%
Autoscaling — Karpenter	44.4%	44.4%
Networking	44.4%	44.4%
Observability	40.0%	40.0%
Resiliency	— (new)	31.0%
Security	— (new)	28.0%

Resolved since last report: - ✅ EKS Cluster Endpoint Access Control - ✅ Load Balancer Target Type Configuration - ✅ Optimize scan interval for cluster size

New failures since last report: - ❌ Check if Quotas are defined at the namespace level - ❌ Check if the EKS Control Plane logs are enabled - ❌ Employ least privileged access when creating RoleBindings and ClusterRoleBindings

Check Results Summary¶

S.No	Check Name	Category	Severity	Status	Fix Type	Effort	Impacted Resources
1	Leverage EKS Cluster Access Manager to configure RBAC	Security	—	✅ PASS	—	—	—
2	Ensure IMDSv2 is enforced in the worker node security configuration	Security	—	✅ PASS	—	—	—
3	Check if IRSA is configured	Security	—	✅ PASS	—	—	—
4	Use an OS optimized for running containers	Security	—	✅ PASS	—	—	—
5	Minimize access to worker nodes	Security	—	✅ PASS	—	—	—
6	Isolating tenant workloads to specific nodes	Security	—	✅ PASS	—	—	—
7	Check if the EKS Control Plane logs are enabled	Security	High	❌ FAIL	AWS CLI	~5 min	eks-review-cluster
8	Check if Encryption is enabled in the StorageClass	Security	High	❌ FAIL	Manifest	~10 min	2 StorageClasses (ebs-gp2-unencrypted, gp2)
9	Use an external secrets provider	Security	High	❌ FAIL	Helm/Manifest	~30 min	Cluster-wide
10	Enable envelope encryption for Kubernetes Secrets	Security	High	❌ FAIL	AWS CLI	~10 min	eks-review-cluster
11	Make the EKS Cluster Endpoint private	Security	High	❌ FAIL	AWS CLI	~30 min	eks-review-cluster
12	Don't use a service account token for authentication	Security	High	❌ FAIL	Manifest	~60 min	45 service accounts across 4 namespaces
13	Employ least privileged access when creating RoleBindings and ClusterRoleBindings	Security	Medium	❌ FAIL	Manifest	~60 min	10 roles with wildcard permissions
14	Leverage EKS Pod Identity	Security	Medium	❌ FAIL	AWS CLI	~10 min	eks-review-cluster
15	Run the application as a non-root user	Security	High	❌ FAIL	Manifest	~120 min	54 workloads (161 pods) across 2 namespaces
16	Don't bind ClusterRoles to anonymous or unauthenticated groups	Security	High	❌ FAIL	Manifest	~10 min	1 ClusterRoleBinding
17	Check if the workers nodes are deployed onto private subnets	Security	High	❌ FAIL	Infrastructure	~60 min	1 node group (karpenter-controller)
18	Check if Network Policies are used to restrict communication between namespaces	Security	Medium	❌ FAIL	Manifest	~30 min	Cluster-wide
19	Check if Quotas are defined at the namespace level	Security	Low	❌ FAIL	Manifest	~10 min	1 namespace (default)
20	Check if Pod Security Standards (PSS) and Pod Security Admission (PSA) is configured	Security	High	❌ FAIL	Manifest	~15 min	1 namespace (default)
21	Restrict the use of hostPath or if hostPath is necessary restrict which prefixes can be used and configure the volume as read-only	Security	High	❌ FAIL	Manifest	~30 min	3 workloads (9 pods) in kube-system
22	Check if the application is using the latest version of the image	Security	Medium	❌ FAIL	Manifest	~60 min	50 workloads (150 pods) in default namespace
23	Do not allow privileged escalation	Security	High	❌ FAIL	Manifest	~120 min	53 workloads (159 pods) across 2 namespaces
24	Configure your images with read-only root file system	Security	Medium	❌ FAIL	Manifest	~60 min	52 workloads (156 pods) across 2 namespaces
25	Disable ServiceAccount token mounts	Security	Low	❌ FAIL	Manifest	~60 min	57 workloads (166 pods) across 2 namespaces
26	Avoid running singleton Pods	Resiliency	—	✅ PASS	—	—	—
27	Cluster Authentication	Resiliency	—	✅ PASS	—	—	—
28	Running large clusters	Resiliency	—	✅ PASS	—	—	—
29	Avoid catch-all admission webhooks	Resiliency	—	✅ PASS	—	—	—
30	CoreDNS Configuration	Resiliency	—	✅ PASS	—	—	—
31	Kubelet Reserved Resources	Resiliency	—	✅ PASS	—	—	—
32	Use Vertical Pod Autoscaler	Resiliency	Low	❌ FAIL	Helm/Manifest	~30 min	vpa-controller
33	Use preStop hooks	Resiliency	Medium	❌ FAIL	Manifest	~60 min	54 workloads across 2 namespaces
34	Use a Service Mesh	Resiliency	Low	❌ FAIL	Helm/Manifest	~120 min	Cluster-wide
35	Monitor your applications	Resiliency	High	❌ FAIL	Helm/Manifest	~60 min	Cluster-wide
36	Use centralized logging	Resiliency	Medium	❌ FAIL	Helm/Manifest	~60 min	Cluster-wide
37	Use startup probes	Resiliency	High	❌ FAIL	Manifest	~30 min	4 workloads in kube-system
38	Run multiple replicas	Resiliency	Medium	❌ FAIL	Manifest	~10 min	1 workload (kube-system/karpenter)
39	Use pod anti-affinity	Resiliency	Medium	❌ FAIL	Manifest	~60 min	50 deployments in default namespace
40	Use liveness probes	Resiliency	High	❌ FAIL	Manifest	~120 min	55 workloads across 2 namespaces
41	Use readiness probes	Resiliency	High	❌ FAIL	Manifest	~120 min	55 workloads across 2 namespaces
42	Use Pod Disruption Budgets	Resiliency	Medium	❌ FAIL	Manifest	~60 min	51 workloads across 2 namespaces
43	Run Kubernetes Metrics Server	Resiliency	Medium	❌ FAIL	Helm/Manifest	~15 min	metrics-server
44	Use Horizontal Pod Autoscaler	Resiliency	Medium	❌ FAIL	Manifest	~60 min	50 workloads in default namespace
45	Use custom metrics scaling	Resiliency	Low	❌ FAIL	Helm/Manifest	~60 min	custom-metrics-api
46	Monitor Control Plane Logs	Resiliency	Medium	❌ FAIL	AWS CLI	~5 min	eks-review-cluster
47	EKS Control Plane Endpoint Access Control	Resiliency	High	❌ FAIL	AWS CLI	~10 min	eks-review-cluster
48	Use Kubernetes Cluster Autoscaler or Karpenter	Resiliency	High	❌ FAIL	Helm/Manifest	~30 min	Cluster-wide
49	Worker nodes spread across multiple AZs	Resiliency	High	❌ FAIL	Infrastructure	~60 min	3 nodes across 2 AZs
50	Configure Resource Requests/Limits	Resiliency	High	❌ FAIL	Manifest	~120 min	53 deployments across 2 namespaces
51	Namespace ResourceQuotas	Resiliency	Medium	❌ FAIL	Manifest	~10 min	1 namespace (default)
52	Namespace LimitRanges	Resiliency	Medium	❌ FAIL	Manifest	~10 min	1 namespace (default)
53	Monitor CoreDNS metrics	Resiliency	Medium	❌ FAIL	Manifest	~15 min	coredns-metrics
54	EKS Node Monitoring Agent	Resiliency	Medium	❌ FAIL	AWS CLI	~5 min	eks-review-cluster
55	VPC CNI Managed Add-on Usage	Networking	—	✅ PASS	—	—	—
56	Subnet IP Address Availability	Networking	—	✅ PASS	—	—	—
57	Load Balancer Target Type Configuration	Networking	—	✅ PASS	—	—	—
58	Kube-Proxy Mode for Service Scale	Networking	—	✅ PASS	—	—	—
59	EKS Cluster Endpoint Access Control	Networking	High	❌ FAIL	AWS CLI	~10 min	eks-review-cluster
60	Multi-AZ Node Distribution	Networking	High	❌ FAIL	Infrastructure	~60 min	3 nodes across 2 AZs
61	Private Subnet Node Deployment	Networking	High	❌ FAIL	Infrastructure	~60 min	2 nodes in 2 public subnets
62	VPC CNI Service Account IAM Role	Networking	Medium	❌ FAIL	AWS CLI	~15 min	vpc-cni add-on
63	PreStop Hook Configuration for Ingress Deployments	Networking	Medium	❌ FAIL	Manifest	~15 min	5 deployments in default namespace
64	Check if Self-Managed Karpenter is deployed	Karpenter	—	✅ PASS	—	—	—
65	Use timers (TTL) to automatically delete nodes from the cluster	Karpenter	—	✅ PASS	—	—	—
66	Avoid overly constraining the Instance Types that Karpenter can provision, especially when utilizing Spot	Karpenter	—	✅ PASS	—	—	—
67	Configure the proper disruption settings for a production cluster	Karpenter	—	✅ PASS	—	—	—
68	Lock down AMIs in production clusters	Karpenter	High	❌ FAIL	Manifest	~10 min	1 EC2NodeClass (spot-nodes)
69	Exclude instance types that do not fit your workload	Karpenter	High	❌ FAIL	Manifest	~15 min	1 NodePool (spot-pool)
70	Create NodePools that are mutually exclusive or weighted	Karpenter	High	❌ FAIL	Manifest	~15 min	1 NodePool
71	Configure the Limits in the karpenter node pools	Karpenter	High	❌ FAIL	Manifest	~5 min	1 NodePool (spot-pool)
72	Check if Spot capacity is used in the nodepool and ensure spot to spot consolidation is enabled in the karpenter feature gates	Karpenter	High	❌ FAIL	Manifest	~10 min	1 NodePool (spot-pool)
73	Cluster Autoscaler version matches cluster version	Cluster Autoscaler	—	✅ PASS	—	—	—
74	etcd Storage Size	Observability	—	✅ PASS	—	—	—
75	Admission Webhook Latency	Observability	—	✅ PASS	—	—	—
76	API Server Request Error Rate (5xx)	Observability	High	❌ FAIL	Investigation	~30 min	eks-review-cluster
77	API Server Throttling (429 Rate Limiting)	Observability	High	❌ FAIL	Investigation	~30 min	eks-review-cluster
78	Scheduler Pending Pods	Observability	High	❌ FAIL	Investigation	~30 min	eks-review-cluster

High Priority Issues¶

1. Check if the EKS Control Plane logs are enabled¶

High | Security | AWS CLI Impact: Without control plane logs, security incidents and operational issues cannot be audited or diagnosed. Resources: eks-review-cluster Current State: No control plane logs are enabled

Control plane logging is essential for security auditing, troubleshooting, and compliance. Without API server, authenticator, controller manager, scheduler, and audit logs enabled, there is no visibility into control plane operations. This was flagged as a new failure since the last assessment.

Remediation:

aws eks update-cluster-config \
  --name eks-review-cluster \
  --logging '{"clusterLogging":[{"types":["api","audit","authenticator","controllerManager","scheduler"],"enabled":true}]}'

Considerations: No expected negative impact. Enabling logs incurs CloudWatch Logs costs. Logs can be filtered or retained per your compliance requirements.

2. Check if Encryption is enabled in the StorageClass¶

High | Security | Manifest Impact: Unencrypted EBS volumes expose data at rest to unauthorized access if the underlying storage is compromised. Resources: ebs-gp2-unencrypted, gp2 Current State: Found 2 StorageClasses without encryption enabled

Two StorageClasses are provisioning EBS volumes without encryption. All new PersistentVolumes created using these StorageClasses will have unencrypted data at rest, violating security best practices and potentially compliance requirements.

Remediation:

# Create an encrypted StorageClass (or patch existing ones)
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp2-encrypted
provisioner: ebs.csi.aws.com
parameters:
  type: gp2
  encrypted: "true"
  # Optionally specify a KMS key:
  # kmsKeyId: arn:aws:kms:us-east-1:<ACCOUNT_ID>:key/<KEY_ID>  # TEMPLATE — replace <ACCOUNT_ID> and <KEY_ID> before running
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

Considerations: Existing PVs will not be retroactively encrypted. New PVs will use the updated StorageClass. To encrypt existing data, you must snapshot, encrypt the snapshot, and restore.

3. Use an external secrets provider¶

High | Security | Helm/Manifest Impact: Native Kubernetes Secrets are base64-encoded (not encrypted) and lack rotation, auditing, and fine-grained access control. Resources: Cluster-wide Current State: No external secrets provider found - using native Kubernetes secrets only

The cluster relies solely on native Kubernetes Secrets, which are stored as base64-encoded values in etcd. Without an external secrets provider (e.g., AWS Secrets Manager, HashiCorp Vault), secrets lack automatic rotation, centralized management, and audit trails.

Remediation:

# Install External Secrets Operator via Helm
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
  -n external-secrets --create-namespace

# Or install AWS Secrets and Configuration Provider (ASCP)
kubectl apply -f https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/deployment/aws-provider-installer.yaml

Considerations: Requires IAM roles/policies for the secrets provider to access AWS Secrets Manager or SSM Parameter Store. Existing secrets need to be migrated to the external provider.

4. Enable envelope encryption for Kubernetes Secrets¶

High | Security | AWS CLI Impact: Without KMS envelope encryption, Kubernetes Secrets stored in etcd can be read by anyone with etcd access. Resources: eks-review-cluster Current State: Kubernetes Secrets envelope encryption is not enabled. Secrets are stored in etcd without KMS encryption.

EKS supports envelope encryption of Kubernetes Secrets using AWS KMS. Without this, secrets are stored in etcd in a format that can be decoded. Enabling envelope encryption adds a layer of protection using a customer-managed KMS key.

Remediation:

# Create a KMS key (if not already available)
aws kms create-key --description "EKS Secrets Encryption Key"

# Associate encryption config with the cluster
aws eks associate-encryption-config \
  --cluster-name eks-review-cluster \
  --encryption-config '[{"resources":["secrets"],"provider":{"keyArn":"arn:aws:kms:us-east-1:<ACCOUNT_ID>:key/<KEY_ID>"}}]'
# TEMPLATE — replace <ACCOUNT_ID> and <KEY_ID> before running

Considerations: This is a one-way operation — once enabled, it cannot be disabled. Existing secrets will be encrypted on next update. Ensure the KMS key policy grants the EKS service access.

5. Make the EKS Cluster Endpoint private¶

High | Security | AWS CLI Impact: A public-only API endpoint exposes the Kubernetes control plane to the internet, increasing the attack surface. Resources: eks-review-cluster Current State: Cluster endpoint is public only

The cluster API server endpoint is configured as public-only, meaning it is accessible from the internet. This significantly increases the attack surface. Best practice is to enable private endpoint access and restrict or disable public access.

Remediation:

# Enable private endpoint and restrict public access
aws eks update-cluster-config \
  --name eks-review-cluster \
  --resources-vpc-config endpointPublicAccess=true,endpointPrivateAccess=true,publicAccessCidrs="<YOUR_CIDR>/32"
# TEMPLATE — replace <YOUR_CIDR> with your corporate IP range

# Once private connectivity is verified, disable public access:
# aws eks update-cluster-config \
#   --name eks-review-cluster \
#   --resources-vpc-config endpointPublicAccess=false,endpointPrivateAccess=true

Considerations: Ensure VPN, Direct Connect, or bastion host connectivity to the VPC before disabling public access. Locking out public access without private connectivity will prevent kubectl access.

6. Don't use a service account token for authentication¶

High | Security | Manifest Impact: Auto-mounted service account tokens provide unnecessary API access to pods, expanding the blast radius of a compromise. Resources: 45 service accounts across 4 namespaces — default/default, kube-node-lease/default, kube-public/default, kube-system (attachdetach-controller, aws-cloud-provider, aws-load-balancer-controller, and 39 more) Current State: Found 45 service accounts with automountServiceAccountToken enabled

45 service accounts have automountServiceAccountToken: true, which automatically mounts API credentials into every pod using those accounts. Most workloads do not need direct Kubernetes API access.

Remediation:

# Disable auto-mounting on service accounts that don't need API access
apiVersion: v1
kind: ServiceAccount
metadata:
  name: default
  namespace: default
automountServiceAccountToken: false

# Patch the default service account in each namespace
kubectl patch serviceaccount default -n default -p '{"automountServiceAccountToken": false}'
kubectl patch serviceaccount default -n kube-node-lease -p '{"automountServiceAccountToken": false}'
kubectl patch serviceaccount default -n kube-public -p '{"automountServiceAccountToken": false}'

Considerations: Do not disable token mounting for service accounts used by controllers or workloads that require Kubernetes API access (e.g., aws-load-balancer-controller, karpenter). Test each workload before applying.

7. Run the application as a non-root user¶

High | Security | Manifest Impact: Running containers as root allows processes to modify the container filesystem and potentially escape to the host. Resources: 54 workloads (161 pods) across 2 namespaces — default: 50 workloads (e.g., ReplicaSet/nginx-1-7f679f8d4c, ReplicaSet/nginx-10-768f774ff, ReplicaSet/nginx-11-56df9f6659, and 47 more); kube-system: DaemonSet/aws-node, DaemonSet/ebs-csi-node, DaemonSet/kube-proxy, ReplicaSet/coredns-849f74687b Current State: Found 54 workloads (161 pods) running as root user

A large number of workloads are running without a non-root security context. This violates the principle of least privilege and increases the risk of container breakout attacks. Each workload should specify runAsNonRoot: true and a specific runAsUser.

Remediation:

# Add to each Deployment/DaemonSet pod spec
spec:
  template:
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        runAsGroup: 1000
      containers:
      - name: <container-name>
        securityContext:
          runAsNonRoot: true
          runAsUser: 1000

Considerations: Some system workloads (aws-node, kube-proxy) require root access. Verify application compatibility before enforcing non-root. Nginx default image requires root — use nginxinc/nginx-unprivileged instead.

8. Don't bind ClusterRoles to anonymous or unauthenticated groups¶

High | Security | Manifest Impact: Granting cluster roles to unauthenticated subjects allows anyone to access the API server without credentials. Resources: system:public-info-viewer (subject: system:unauthenticated) Current State: Found 1 ClusterRoleBindings granting access to anonymous/unauthenticated subjects

The system:public-info-viewer ClusterRoleBinding grants access to the system:unauthenticated group. While this is a default Kubernetes binding for discovery endpoints, it should be reviewed to ensure it does not expose sensitive information.

Remediation:

# Review the binding
kubectl describe clusterrolebinding system:public-info-viewer

# If the binding is not needed, remove it:
# kubectl delete clusterrolebinding system:public-info-viewer
# WARNING: This is a default Kubernetes binding — only remove if you understand the impact

Considerations: The system:public-info-viewer binding is a Kubernetes default that allows unauthenticated access to /healthz and /version endpoints. Removing it may break health checks from external systems. Evaluate whether your security posture requires its removal.

9. Check if the workers nodes are deployed onto private subnets¶

High | Security | Infrastructure Impact: Worker nodes in public subnets are directly reachable from the internet, increasing exposure to attacks. Resources: 1 node group (karpenter-controller) Current State: Found 1 node groups in public subnets

The karpenter-controller node group is deployed in public subnets. Worker nodes should be placed in private subnets with internet access provided through NAT gateways to reduce the attack surface.

Remediation:

# Update the EC2NodeClass to use private subnets
kubectl edit ec2nodeclass spot-nodes
# Change subnetSelector to target private subnets:
# spec:
#   subnetSelectorTerms:
#   - tags:
#       kubernetes.io/role/internal-elb: "1"  # Private subnet tag

Considerations: Moving nodes to private subnets requires NAT gateway for outbound internet access (ECR image pulls, etc.). Existing pods will be rescheduled during node rotation. Plan during a maintenance window.

10. Check if Pod Security Standards (PSS) and Pod Security Admission (PSA) is configured¶

High | Security | Manifest Impact: Without PSS/PSA, pods can run with unrestricted privileges, bypassing security boundaries. Resources: 1 namespace (default) Current State: Found 1 namespaces without Pod Security Standards configured

The default namespace does not have Pod Security Standards labels configured. PSA enforcement ensures pods meet baseline or restricted security profiles, preventing privileged containers from being scheduled.

Remediation:

# Apply Pod Security Standards labels to the default namespace
kubectl label namespace default \
  pod-security.kubernetes.io/enforce=baseline \
  pod-security.kubernetes.io/warn=restricted \
  pod-security.kubernetes.io/audit=restricted

Considerations: Applying enforce=baseline will reject pods that don't meet baseline standards. Start with warn mode to identify non-compliant workloads before enforcing. The restricted profile is more stringent and may require workload changes.

11. Restrict the use of hostPath volumes¶

High | Security | Manifest Impact: hostPath volumes allow containers to access the host filesystem, enabling potential container escape and data exfiltration. Resources: kube-system/DaemonSet/aws-node, kube-system/DaemonSet/ebs-csi-node, kube-system/DaemonSet/kube-proxy Current State: Found 3 workloads (9 pods) using hostPath volumes

Three system DaemonSets in kube-system use hostPath volumes. While some system components legitimately require host filesystem access, the volumes should be configured as read-only where possible and restricted to specific paths.

Remediation:

# For workloads that must use hostPath, ensure read-only access:
spec:
  template:
    spec:
      containers:
      - name: <container-name>
        volumeMounts:
        - name: host-volume
          mountPath: /var/run
          readOnly: true
      volumes:
      - name: host-volume
        hostPath:
          path: /var/run
          type: Directory

Considerations: aws-node, ebs-csi-node, and kube-proxy are AWS-managed components that require hostPath access for CNI networking and storage operations. Do not remove hostPath from these without understanding the impact. Focus on ensuring read-only where possible.

12. Do not allow privileged escalation¶

High | Security | Manifest Impact: Privilege escalation allows container processes to gain more privileges than their parent, potentially leading to host compromise. Resources: 53 workloads (159 pods) across 2 namespaces — default: 50 workloads (e.g., ReplicaSet/nginx-1-7f679f8d4c, ReplicaSet/nginx-10-768f774ff, ReplicaSet/nginx-11-56df9f6659, and 47 more); kube-system: DaemonSet/aws-node, DaemonSet/ebs-csi-node, DaemonSet/kube-proxy Current State: Found 53 workloads (159 pods) allowing privilege escalation

The majority of workloads do not explicitly set allowPrivilegeEscalation: false in their security context. This allows container processes to gain additional privileges via setuid binaries or other mechanisms.

Remediation:

# Add to each container's securityContext
spec:
  template:
    spec:
      containers:
      - name: <container-name>
        securityContext:
          allowPrivilegeEscalation: false

Considerations: Some system components (aws-node, kube-proxy) may require privilege escalation. Apply to application workloads first. Test thoroughly — some applications with setuid binaries will break.

13. Monitor your applications¶

High | Resiliency | Helm/Manifest Impact: Without application monitoring, performance degradation and failures go undetected until users are impacted. Resources: Cluster-wide Current State: No monitoring solution found

No monitoring solution (Prometheus, CloudWatch Container Insights, Datadog, etc.) was detected in the cluster. Application and infrastructure monitoring is essential for detecting issues, capacity planning, and maintaining SLAs.

Remediation:

# Option 1: Install Amazon CloudWatch Observability add-on
aws eks create-addon \
  --cluster-name eks-review-cluster \
  --addon-name amazon-cloudwatch-observability

# Option 2: Install Prometheus via Helm
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install prometheus prometheus-community/kube-prometheus-stack \
  -n monitoring --create-namespace

Considerations: Monitoring solutions consume cluster resources (CPU, memory). Size the monitoring stack appropriately. CloudWatch Container Insights incurs AWS costs based on metrics and log volume.

14. Use startup probes¶

High | Resiliency | Manifest Impact: Without startup probes, slow-starting containers may be killed by liveness probes before they are ready, causing restart loops. Resources: kube-system/Deployment: aws-load-balancer-controller, kube-system/Deployment: coredns, kube-system/Deployment: ebs-csi-controller, kube-system/Deployment: karpenter Current State: Found 4 workloads with liveness probes but no startup probes across all namespaces

Four critical system workloads have liveness probes configured but lack startup probes. Startup probes protect slow-starting containers from being killed by liveness probes during initialization.

Remediation:

# Add startup probes to workloads that have liveness probes
spec:
  template:
    spec:
      containers:
      - name: <container-name>
        startupProbe:
          httpGet:
            path: /healthz
            port: 8080
          failureThreshold: 30
          periodSeconds: 10

Considerations: Startup probe configuration should match the application's maximum expected startup time. The failureThreshold * periodSeconds should exceed the worst-case startup duration.

15. Use liveness probes¶

High | Resiliency | Manifest Impact: Without liveness probes, Kubernetes cannot detect and restart containers that are in a deadlocked or unhealthy state. Resources: 55 workloads across 2 namespaces — default: 50 workloads (e.g., Deployment: nginx-1, Deployment: nginx-10, Deployment: nginx-11, and 47 more); kube-system: Deployment: ebs-csi-controller, DaemonSet: aws-node, DaemonSet: ebs-csi-node, DaemonSet: ebs-csi-node-windows, DaemonSet: kube-proxy Current State: Found 55 workloads without liveness probes across all namespaces

55 workloads lack liveness probes, meaning Kubernetes has no mechanism to detect and automatically restart unhealthy containers. This can lead to degraded service with no automatic recovery.

Remediation:

# Add liveness probes to each workload
spec:
  template:
    spec:
      containers:
      - name: nginx
        livenessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 15
          periodSeconds: 10
          timeoutSeconds: 5
          failureThreshold: 3

Considerations: Liveness probe endpoints must return quickly. Misconfigured probes (too aggressive thresholds) can cause unnecessary restarts. Always pair with startup probes for slow-starting applications.

16. Use readiness probes¶

High | Resiliency | Manifest Impact: Without readiness probes, traffic is sent to pods before they are ready to serve, causing errors for end users. Resources: 55 workloads across 2 namespaces — default: 50 workloads (e.g., Deployment: nginx-1, Deployment: nginx-10, Deployment: nginx-11, and 47 more); kube-system: Deployment: ebs-csi-controller, DaemonSet: aws-node, DaemonSet: ebs-csi-node, DaemonSet: ebs-csi-node-windows, DaemonSet: kube-proxy Current State: Found 55 workloads without readiness probes across all namespaces

55 workloads lack readiness probes. Without readiness probes, Kubernetes adds pods to Service endpoints immediately, potentially routing traffic to containers that are not yet ready to handle requests.

Remediation:

# Add readiness probes to each workload
spec:
  template:
    spec:
      containers:
      - name: nginx
        readinessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 5
          periodSeconds: 5
          timeoutSeconds: 3
          failureThreshold: 3

Considerations: Readiness probes should check actual application readiness (e.g., database connections established). Overly strict readiness probes can cause pods to be removed from service during transient issues.

17. EKS Control Plane Endpoint Access Control¶

High | Resiliency / Networking | AWS CLI Impact: Unrestricted public API access allows anyone on the internet to attempt authentication against the cluster. Resources: eks-review-cluster Current State: API server endpoint has unrestricted public access (0.0.0.0/0)

The EKS API server endpoint allows unrestricted public access from any IP address (0.0.0.0/0). This should be restricted to known CIDR ranges or switched to private-only access.

Remediation:

aws eks update-cluster-config \
  --name eks-review-cluster \
  --resources-vpc-config publicAccessCidrs="<YOUR_CORPORATE_CIDR>/32"
# TEMPLATE — replace <YOUR_CORPORATE_CIDR> with your allowed IP ranges

Considerations: Ensure all administrators and CI/CD systems are within the allowed CIDR ranges before restricting. Incorrect CIDR configuration will lock out users.

18. Use Kubernetes Cluster Autoscaler or Karpenter¶

High | Resiliency | Helm/Manifest Impact: Without node autoscaling, the cluster cannot respond to demand changes, leading to pending pods or wasted resources. Resources: Cluster-wide Current State: No node autoscaling solution found

No active node autoscaling solution was detected. While Karpenter is deployed, it may not be functioning correctly. The Scheduler Pending Pods metric shows a peak of 721 pending pods, confirming autoscaling is not working as expected.

Remediation:

# Verify Karpenter is running and healthy
kubectl get pods -n kube-system -l app.kubernetes.io/name=karpenter
kubectl logs -n kube-system -l app.kubernetes.io/name=karpenter --tail=50

# Verify NodePool and EC2NodeClass are configured
kubectl get nodepools
kubectl get ec2nodeclasses

# Check for provisioning errors
kubectl describe nodepool spot-pool

Considerations: Investigate why Karpenter is not provisioning nodes despite being deployed. Check IAM permissions, subnet availability, and instance type availability in the configured AZs.

19. Worker nodes spread across multiple AZs¶

High | Resiliency / Networking | Infrastructure Impact: Uneven AZ distribution means an AZ failure could take down a disproportionate number of workloads. Resources: 3 nodes — us-east-1f: 2 nodes, us-east-1c: 1 node Current State: Uneven node distribution across 2 AZs with 33.3% deviation

The cluster has only 3 nodes spread across 2 AZs with uneven distribution. Best practice is to use at least 3 AZs with balanced node counts for high availability.

Remediation:

# Update Karpenter NodePool to use 3+ AZs with topology spread
kubectl edit nodepool spot-pool
# Ensure subnetSelectorTerms include subnets in 3+ AZs

# Add topology spread constraints to workloads:
# spec:
#   topologySpreadConstraints:
#   - maxSkew: 1
#     topologyKey: topology.kubernetes.io/zone
#     whenUnsatisfiable: DoNotSchedule

Considerations: Adding a third AZ requires subnets in that AZ. Node rotation will be needed to rebalance. Cross-AZ data transfer costs may increase.

20. Configure Resource Requests/Limits¶

High | Resiliency | Manifest Impact: Without resource requests/limits, pods can consume unbounded resources, causing node instability and noisy-neighbor issues. Resources: 53 deployments across 2 namespaces — default: 50 deployments (e.g., nginx-1, nginx-10, nginx-11, and 47 more); kube-system: aws-load-balancer-controller, coredns, ebs-csi-controller Current State: Found 53 deployments without complete resource specifications across all namespaces

53 deployments lack complete resource requests and limits. Without these, the scheduler cannot make informed placement decisions, and pods can starve other workloads of resources.

Remediation:

# Add resource specifications to each deployment
spec:
  template:
    spec:
      containers:
      - name: nginx
        resources:
          requests:
            cpu: "100m"
            memory: "128Mi"
          limits:
            cpu: "500m"
            memory: "256Mi"

Considerations: Resource values should be based on actual usage data. Install VPA in recommendation mode first to get sizing suggestions. Over-provisioning wastes resources; under-provisioning causes OOMKills and throttling.

21. Private Subnet Node Deployment¶

High | Networking | Infrastructure Impact: Nodes in public subnets have public IPs and are directly reachable from the internet. Resources: ip-192-168-40-195.ec2.internal (in public subnet subnet-094bb656e1e610bbc), ip-192-168-26-84.ec2.internal (in public subnet subnet-09c0e890c9302a337) Current State: Found 2 nodes in 2 public subnets

Two worker nodes are running in public subnets, making them directly accessible from the internet. Worker nodes should be deployed in private subnets with NAT gateway for outbound access.

Remediation:

# Update EC2NodeClass to use private subnets
kubectl edit ec2nodeclass spot-nodes
# spec:
#   subnetSelectorTerms:
#   - tags:
#       kubernetes.io/role/internal-elb: "1"

# Cordon and drain existing nodes in public subnets
kubectl cordon ip-192-168-40-195.ec2.internal
kubectl drain ip-192-168-40-195.ec2.internal --ignore-daemonsets --delete-emptydir-data
kubectl cordon ip-192-168-26-84.ec2.internal
kubectl drain ip-192-168-26-84.ec2.internal --ignore-daemonsets --delete-emptydir-data

Considerations: Ensure NAT gateways are configured in private subnets for outbound internet access. Draining nodes will reschedule pods — ensure PDBs are in place first.

22. Lock down AMIs in production clusters¶

High | Karpenter | Manifest Impact: Using @latest AMI aliases means nodes can get different AMI versions during scaling, causing inconsistency and potential breakage. Resources: 1 EC2NodeClass (spot-nodes) Current State: Found 1 EC2NodeClasses using @latest AMI alias (not pinned)

The spot-nodes EC2NodeClass uses the @latest AMI alias instead of a pinned AMI version. In production, AMIs should be pinned to a specific version to ensure consistency and allow controlled rollouts.

Remediation:

# Pin the AMI version in EC2NodeClass
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: spot-nodes
spec:
  amiSelectorTerms:
  - alias: al2023@v20260401  # TEMPLATE — replace with your desired AMI version

Considerations: Pinned AMIs require manual updates for security patches. Establish a process to test and roll out new AMI versions regularly.

23. Exclude instance types that do not fit your workload¶

High | Karpenter | Manifest Impact: Without instance type exclusions, Karpenter may provision unsuitable instances (e.g., GPU, high-memory) that waste cost. Resources: 1 NodePool (spot-pool) Current State: Found 1 NodePools without instance type exclusions

The spot-pool NodePool does not exclude instance types that are unsuitable for the workload. This can lead to Karpenter provisioning expensive or inappropriate instance types.

Remediation:

# Add instance type requirements to the NodePool
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: spot-pool
spec:
  template:
    spec:
      requirements:
      - key: karpenter.k8s.aws/instance-family
        operator: NotIn
        values: ["p", "g", "inf", "trn", "dl", "f", "vt"]  # Exclude GPU/accelerator families
      - key: karpenter.k8s.aws/instance-size
        operator: NotIn
        values: ["metal", "24xlarge", "16xlarge", "12xlarge"]  # Exclude very large instances

Considerations: Review your workload resource requirements to determine appropriate instance families and sizes. Overly restrictive exclusions may reduce Spot availability.

24. Create NodePools that are mutually exclusive or weighted¶

High | Karpenter | Manifest Impact: Overlapping NodePools without weights can cause unpredictable scheduling behavior and resource contention. Resources: 1 NodePool Current State: Found 1 NodePools without proper weight or exclusivity configuration

The NodePool lacks weight configuration. When multiple NodePools exist, weights determine scheduling priority. Without weights or mutual exclusivity, Karpenter's scheduling decisions become unpredictable.

Remediation:

# Add weight to the NodePool
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: spot-pool
spec:
  weight: 50  # Higher weight = higher priority (0-100)
  template:
    spec:
      requirements:
      - key: karpenter.sh/capacity-type
        operator: In
        values: ["spot"]

Considerations: If you have only one NodePool, adding a weight is still recommended for future-proofing. Plan NodePool weights based on cost optimization strategy (e.g., Spot preferred over On-Demand).

25. Configure the Limits in the karpenter node pools¶

High | Karpenter | Manifest Impact: Without resource limits on NodePools, Karpenter can provision unlimited nodes, leading to runaway costs. Resources: 1 NodePool (spot-pool) Current State: Found 1 NodePools without resource limits

The spot-pool NodePool has no resource limits configured. This means Karpenter can scale the cluster indefinitely, potentially causing unexpected AWS costs.

Remediation:

# Add limits to the NodePool
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: spot-pool
spec:
  limits:
    cpu: "100"       # TEMPLATE — adjust based on your expected maximum
    memory: "400Gi"  # TEMPLATE — adjust based on your expected maximum

Considerations: Set limits based on your budget and expected peak workload. Monitor Karpenter metrics to understand typical resource consumption before setting limits.

26. Check if Spot capacity is used in the nodepool and ensure spot to spot consolidation is enabled¶

High | Karpenter | Manifest Impact: Without SpotToSpotConsolidation, Karpenter cannot optimize Spot instance usage, missing cost savings opportunities. Resources: 1 NodePool (spot-pool) Current State: Spot capacity used in 1 NodePools but SpotToSpotConsolidation is not enabled

The spot-pool uses Spot instances but does not have SpotToSpotConsolidation enabled in Karpenter feature gates. This feature allows Karpenter to consolidate workloads across Spot instances for better cost optimization.

Remediation:

# Enable SpotToSpotConsolidation in Karpenter ConfigMap or Helm values
# If using Helm, update values:
# controller:
#   env:
#   - name: FEATURE_GATES
#     value: "SpotToSpotConsolidation=true"

# Or patch the Karpenter deployment directly:
kubectl -n kube-system set env deployment/karpenter FEATURE_GATES=SpotToSpotConsolidation=true

Considerations: SpotToSpotConsolidation may cause more frequent node replacements as Karpenter optimizes for cost. Ensure PDBs are configured to prevent disruption during consolidation.

27. API Server Request Error Rate (5xx)¶

High | Observability | Investigation Impact: High 5xx error rates indicate API server instability, which can affect all cluster operations including deployments and scaling. Resources: eks-review-cluster Current State: 2,993 total 5xx errors over 7 days (threshold: 100). Peak: 69 errors in a single period. Average: 27.7 per period. Latest value: 0.

The API server has returned 2,993 5xx errors over the past 7 days, significantly exceeding the threshold of 100. While the latest value is 0, the sustained error rate indicates intermittent control plane issues that need investigation.

Remediation:

# Enable control plane logging to investigate 5xx errors
aws eks update-cluster-config \
  --name eks-review-cluster \
  --logging '{"clusterLogging":[{"types":["api","audit"],"enabled":true}]}'

# Query CloudWatch Logs for error patterns
aws logs filter-log-events \
  --log-group-name /aws/eks/eks-review-cluster/cluster \
  --filter-pattern "5xx" \
  --start-time $(date -d '7 days ago' +%s000)

Considerations: 5xx errors may correlate with the high pending pod count (721 peak) and throttling. Investigate whether the errors are caused by resource exhaustion or API server overload.

28. API Server Throttling (429 Rate Limiting)¶

High | Observability | Investigation Impact: API throttling means clients are being rate-limited, causing delayed operations, failed deployments, and degraded controller performance. Resources: eks-review-cluster Current State: 109 total 429 throttled requests over 7 days (threshold: 50). Peak: 109 in a single period. Average: 1.0 per period. Latest value: 0.

The API server has throttled 109 requests over the past 7 days. While the latest value is 0, the peak of 109 in a single period suggests burst traffic patterns that overwhelm the API server.

Remediation:

# Investigate which clients are generating the most API calls
# Enable audit logging first:
aws eks update-cluster-config \
  --name eks-review-cluster \
  --logging '{"clusterLogging":[{"types":["audit"],"enabled":true}]}'

# Review client-side configurations:
# - Reduce list/watch frequency in custom controllers
# - Implement exponential backoff in API clients
# - Use informers/caches instead of direct API calls

Considerations: Throttling often correlates with controllers performing excessive list operations. Check Karpenter, custom operators, and CI/CD tools for aggressive API polling patterns.

29. Scheduler Pending Pods¶

High | Observability | Investigation Impact: Pending pods indicate workloads that cannot be scheduled, directly impacting application availability and user experience. Resources: eks-review-cluster Current State: Peak of 721 pending pods over 7 days (threshold: 10). Current: 101 pending pods. Average: 196.9 pending pods.

The cluster has a severe scheduling backlog with 101 pods currently pending and a peak of 721. This indicates insufficient cluster capacity and/or a malfunctioning autoscaler. The average of ~197 pending pods suggests this is a persistent issue.

Remediation:

# Investigate pending pods
kubectl get pods --all-namespaces --field-selector=status.phase=Pending

# Check for scheduling issues
kubectl describe pods -n default --field-selector=status.phase=Pending | grep -A 5 "Events:"

# Verify Karpenter is provisioning nodes
kubectl logs -n kube-system -l app.kubernetes.io/name=karpenter --tail=100

# Check NodePool status
kubectl describe nodepool spot-pool

Considerations: This is likely related to the Karpenter configuration issues identified above (no limits, unpinned AMIs, missing exclusions). Fixing Karpenter configuration should resolve the pending pod backlog.

Medium Priority Issues¶

30. Employ least privileged access when creating RoleBindings and ClusterRoleBindings¶

Medium | Security | Manifest Impact: Wildcard permissions grant unrestricted access to all resources and actions, violating the principle of least privilege. Resources: 10 roles with wildcard permissions — cluster-admin, eks:addon-manager, eks:service-operations, and 7 more Current State: Found 10 roles with wildcard permissions

10 roles use wildcard (*) permissions for resources or verbs. While some system roles (cluster-admin) require broad access, custom roles should follow least-privilege principles.

Remediation:

# Review roles with wildcard permissions
kubectl get clusterroles -o json | jq '.items[] | select(.rules[]?.resources[]? == "*" or .rules[]?.verbs[]? == "*") | .metadata.name'

# Replace wildcards with specific resources and verbs
# Example: Instead of resources: ["*"], use resources: ["pods", "services", "deployments"]

Considerations: Do not modify system roles (cluster-admin, eks:addon-manager, eks:service-operations) as they are managed by EKS. Focus on custom roles that have been granted wildcard permissions.

31. Leverage EKS Pod Identity¶

Medium | Security | AWS CLI Impact: Without Pod Identity, workloads may use overly broad node-level IAM permissions instead of fine-grained pod-level access. Resources: eks-review-cluster Current State: EKS Pod Identity agent addon is not installed

EKS Pod Identity provides a simplified way to assign IAM roles to Kubernetes pods. While IRSA is configured (passing check), Pod Identity is the newer recommended approach with simpler setup and cross-account support.

Remediation:

# Install the EKS Pod Identity Agent add-on
aws eks create-addon \
  --cluster-name eks-review-cluster \
  --addon-name eks-pod-identity-agent

# Create a pod identity association
aws eks create-pod-identity-association \
  --cluster-name eks-review-cluster \
  --namespace <NAMESPACE> \
  --service-account <SERVICE_ACCOUNT> \
  --role-arn arn:aws:iam::<ACCOUNT_ID>:role/<ROLE_NAME>
# TEMPLATE — replace <NAMESPACE>, <SERVICE_ACCOUNT>, <ACCOUNT_ID>, <ROLE_NAME> before running

Considerations: Pod Identity and IRSA can coexist. Migrate workloads incrementally. Pod Identity requires the agent DaemonSet which consumes resources on each node.

32. Check if Network Policies are used to restrict communication between namespaces¶

Medium | Security | Manifest Impact: Without Network Policies, all pods can communicate freely across namespaces, allowing lateral movement in case of compromise. Resources: Cluster-wide Current State: No Network Policies found - namespaces can communicate freely

No Kubernetes Network Policies are defined in the cluster. This means any pod can communicate with any other pod across all namespaces, which violates the principle of network segmentation.

Remediation:

# Create a default deny-all ingress policy per namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: default
spec:
  podSelector: {}
  policyTypes:
  - Ingress

# Then create allow policies for legitimate traffic
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-nginx-ingress
  namespace: default
spec:
  podSelector:
    matchLabels:
      app: nginx
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: ingress-namespace
    ports:
    - port: 80

Considerations: Ensure the VPC CNI Network Policy feature is enabled (ENABLE_NETWORK_POLICY=true on aws-node). Apply deny-all policies carefully — start with audit/monitoring before enforcement to avoid breaking existing traffic flows.

33. Check if the application is using the latest version of the image¶

Medium | Security | Manifest Impact: Mutable image tags (e.g., latest) make deployments non-reproducible and can introduce untested changes. Resources: 50 workloads (150 pods) in default namespace (e.g., ReplicaSet/nginx-1-7f679f8d4c, ReplicaSet/nginx-10-768f774ff, ReplicaSet/nginx-11-56df9f6659, and 47 more) Current State: Found 50 workloads (150 pods) using mutable image tags

50 workloads use mutable image tags instead of immutable digests or specific version tags. This means image content can change without the tag changing, leading to inconsistent deployments.

Remediation:

# Use image digests or specific version tags
spec:
  template:
    spec:
      containers:
      - name: nginx
        image: nginx:1.27.0@sha256:<digest>  # TEMPLATE — replace <digest> with actual image digest
        # Or at minimum use a specific version tag:
        # image: nginx:1.27.0

Considerations: Switching to immutable tags requires updating CI/CD pipelines to produce versioned images. Use imagePullPolicy: IfNotPresent with specific tags to avoid unnecessary pulls.

34. Configure your images with read-only root file system¶

Medium | Security | Manifest Impact: Writable root filesystems allow attackers to modify container binaries, install malware, or tamper with application files. Resources: 52 workloads (156 pods) across 2 namespaces — default: 50 workloads (e.g., ReplicaSet/nginx-1-7f679f8d4c, ReplicaSet/nginx-10-768f774ff, ReplicaSet/nginx-11-56df9f6659, and 47 more); kube-system: DaemonSet/aws-node, DaemonSet/kube-proxy Current State: Found 52 workloads (156 pods) with writable root filesystem

52 workloads do not set readOnlyRootFilesystem: true. A read-only root filesystem prevents runtime modification of container contents, reducing the impact of a container compromise.

Remediation:

# Add readOnlyRootFilesystem to each container
spec:
  template:
    spec:
      containers:
      - name: nginx
        securityContext:
          readOnlyRootFilesystem: true
        volumeMounts:
        - name: tmp
          mountPath: /tmp
        - name: var-cache-nginx
          mountPath: /var/cache/nginx
      volumes:
      - name: tmp
        emptyDir: {}
      - name: var-cache-nginx
        emptyDir: {}

Considerations: Applications that write to the filesystem (logs, temp files, caches) need emptyDir volumes mounted at writable paths. Test each application to identify required writable paths.

35. Use preStop hooks¶

Medium | Resiliency | Manifest Impact: Without preStop hooks, pods are terminated immediately, causing in-flight requests to fail and connections to drop. Resources: 54 workloads across 2 namespaces — default: 50 workloads (e.g., Deployment: nginx-1, Deployment: nginx-10, Deployment: nginx-11, and 47 more); kube-system: Deployment: aws-load-balancer-controller, Deployment: coredns, Deployment: ebs-csi-controller, Deployment: karpenter Current State: Found 54 workloads without preStop hooks across all namespaces

54 workloads lack preStop hooks. PreStop hooks allow graceful shutdown by giving the application time to drain connections and complete in-flight requests before SIGTERM is sent.

Remediation:

# Add preStop hooks to containers
spec:
  template:
    spec:
      containers:
      - name: nginx
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "sleep 15"]
      terminationGracePeriodSeconds: 30

Considerations: The preStop hook duration should be less than terminationGracePeriodSeconds. For load-balanced services, the sleep allows time for the load balancer to deregister the pod.

36. Use centralized logging¶

Medium | Resiliency | Helm/Manifest Impact: Without centralized logging, troubleshooting requires accessing individual pods, and logs are lost when pods are terminated. Resources: Cluster-wide Current State: No centralized logging solution found

No centralized logging solution (Fluent Bit, Fluentd, CloudWatch agent) was detected. Container logs are only available via kubectl logs and are lost when pods are deleted or evicted.

Remediation:

# Option 1: Enable CloudWatch logging via the observability add-on
aws eks create-addon \
  --cluster-name eks-review-cluster \
  --addon-name amazon-cloudwatch-observability

# Option 2: Install Fluent Bit via Helm
helm repo add fluent https://fluent.github.io/helm-charts
helm install fluent-bit fluent/fluent-bit \
  -n logging --create-namespace \
  --set output.type=cloudwatch

Considerations: Centralized logging generates significant data volume. Configure log retention policies and filters to manage costs. Ensure IAM roles are configured for the logging agent to write to the destination.

37. Run multiple replicas¶

Medium | Resiliency | Manifest Impact: Single-replica deployments have no redundancy — a pod failure causes complete service unavailability. Resources: kube-system/Deployment: karpenter Current State: Found 1 workloads (Deployments/StatefulSets) with only 1 replica across all namespaces

The Karpenter deployment runs with only 1 replica. As the node autoscaler, Karpenter is a critical component — its failure would prevent new nodes from being provisioned.

Remediation:

kubectl scale deployment karpenter -n kube-system --replicas=2

Considerations: Karpenter supports leader election, so multiple replicas are safe. Ensure the replicas are spread across AZs using pod anti-affinity or topology spread constraints.

38. Use pod anti-affinity¶

Medium | Resiliency | Manifest Impact: Without anti-affinity, all replicas may be scheduled on the same node, eliminating redundancy benefits. Resources: 50 deployments in default namespace (e.g., nginx-1, nginx-10, nginx-11, and 47 more) Current State: Found 50 multi-replica deployments without pod anti-affinity across all namespaces

50 multi-replica deployments lack pod anti-affinity rules. This means all replicas of a deployment could be placed on the same node, and a single node failure would take down the entire service.

Remediation:

# Add pod anti-affinity to spread replicas across nodes
spec:
  template:
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - nginx
              topologyKey: kubernetes.io/hostname

Considerations: Use preferredDuringSchedulingIgnoredDuringExecution to avoid scheduling failures when nodes are limited. For critical services, use requiredDuringSchedulingIgnoredDuringExecution with topology spread constraints.

39. Use Pod Disruption Budgets¶

Medium | Resiliency | Manifest Impact: Without PDBs, voluntary disruptions (node drains, upgrades) can terminate all replicas simultaneously, causing downtime. Resources: 51 workloads across 2 namespaces — default: 50 workloads (e.g., Deployment: nginx-1, Deployment: nginx-10, Deployment: nginx-11, and 47 more); kube-system: Deployment: aws-load-balancer-controller Current State: Found 51 critical workloads without Pod Disruption Budgets across all namespaces

51 workloads lack Pod Disruption Budgets. During node drains, upgrades, or Karpenter consolidation, all pods could be evicted simultaneously without PDB protection.

Remediation:

# Create PDBs for each deployment
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: nginx-1-pdb
  namespace: default
spec:
  minAvailable: 1  # Or use maxUnavailable: 1
  selector:
    matchLabels:
      app: nginx-1

Considerations: PDBs can block node drains if there aren't enough healthy replicas. Ensure deployments have at least 2 replicas before setting minAvailable: 1. For single-replica deployments, use maxUnavailable: 0 cautiously.

40. Run Kubernetes Metrics Server¶

Medium | Resiliency | Helm/Manifest Impact: Without Metrics Server, HPA cannot function, kubectl top is unavailable, and resource-based scheduling decisions are impaired. Resources: metrics-server Current State: Metrics server is not properly configured or not accessible

The Kubernetes Metrics Server is not functioning correctly. This prevents Horizontal Pod Autoscaler from working and removes visibility into pod/node resource utilization.

Remediation:

# Install Metrics Server via Helm
helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server/
helm install metrics-server metrics-server/metrics-server -n kube-system

# Or install as an EKS add-on (if available)
# Verify after installation:
kubectl top nodes
kubectl top pods --all-namespaces

Considerations: Metrics Server requires port 443 access from the API server to kubelets. Ensure security groups allow this traffic. Metrics Server is lightweight but should be monitored for availability.

41. Use Horizontal Pod Autoscaler¶

Medium | Resiliency | Manifest Impact: Without HPA, workloads cannot scale automatically based on demand, leading to either over-provisioning or service degradation. Resources: 50 workloads in default namespace (e.g., Deployment: nginx-1, Deployment: nginx-10, Deployment: nginx-11, and 47 more) Current State: Found 50 multi-replica workloads without HPA across all namespaces

50 multi-replica workloads lack Horizontal Pod Autoscalers. These workloads cannot automatically scale based on CPU, memory, or custom metrics, requiring manual intervention during traffic spikes.

Remediation:

# Create HPA for each deployment (requires Metrics Server)
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: nginx-1-hpa
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: nginx-1
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

Considerations: HPA requires Metrics Server (currently not working). Fix Metrics Server first. Resource requests must be set on containers for CPU/memory-based HPA to function.

42. Monitor Control Plane Logs¶

Medium | Resiliency | AWS CLI Impact: Without control plane logs, operational issues and security events in the control plane cannot be investigated. Resources: eks-review-cluster Current State: Control plane logging is not enabled

This is the same underlying issue as the Security check for control plane logs. Control plane logging must be enabled for both security auditing and operational monitoring.

Remediation:

aws eks update-cluster-config \
  --name eks-review-cluster \
  --logging '{"clusterLogging":[{"types":["api","audit","authenticator","controllerManager","scheduler"],"enabled":true}]}'

Considerations: Same as finding #1. No expected negative impact. CloudWatch Logs costs apply.

43. Namespace ResourceQuotas¶

Medium | Resiliency | Manifest Impact: Without ResourceQuotas, a single namespace can consume all cluster resources, starving other namespaces. Resources: 1 namespace (default) Current State: Found 1 namespaces without ResourceQuotas

The default namespace lacks ResourceQuotas. This allows unlimited resource consumption, which can impact other namespaces and cluster stability.

Remediation:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: default-quota
  namespace: default
spec:
  hard:
    requests.cpu: "10"        # TEMPLATE — adjust based on your capacity
    requests.memory: "20Gi"   # TEMPLATE — adjust based on your capacity
    limits.cpu: "20"          # TEMPLATE — adjust based on your capacity
    limits.memory: "40Gi"     # TEMPLATE — adjust based on your capacity
    pods: "100"               # TEMPLATE — adjust based on your needs

Considerations: Setting quotas requires all pods in the namespace to have resource requests/limits defined. Apply resource requests/limits to workloads before enabling quotas.

44. Namespace LimitRanges¶

Medium | Resiliency | Manifest Impact: Without LimitRanges, pods can be created without resource specifications, making resource management unpredictable. Resources: 1 namespace (default) Current State: Found 1 namespaces without LimitRanges

The default namespace lacks LimitRanges. LimitRanges set default resource requests/limits for pods that don't specify them, ensuring consistent resource management.

Remediation:

apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
  namespace: default
spec:
  limits:
  - default:
      cpu: "500m"
      memory: "256Mi"
    defaultRequest:
      cpu: "100m"
      memory: "128Mi"
    type: Container

Considerations: LimitRanges apply defaults to new pods only — existing pods are not affected. Set reasonable defaults based on your workload profiles.

45. Monitor CoreDNS metrics¶

Medium | Resiliency | Manifest Impact: Without CoreDNS metrics monitoring, DNS resolution issues go undetected, causing application-level failures. Resources: coredns-metrics Current State: CoreDNS is deployed but metrics monitoring is not configured

CoreDNS is running but its metrics are not being collected. DNS is critical infrastructure — monitoring CoreDNS latency, error rates, and cache hit ratios is essential for detecting resolution issues.

Remediation:

# Create a ServiceMonitor for CoreDNS (if using Prometheus)
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: coredns
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: kube-dns
  endpoints:
  - port: metrics
    interval: 15s

Considerations: Requires a monitoring solution (Prometheus/CloudWatch) to be installed first. Address finding #13 (Monitor your applications) before configuring CoreDNS metrics.

46. EKS Node Monitoring Agent¶

Medium | Resiliency | AWS CLI Impact: Without the Node Monitoring Agent, node-level health issues (disk pressure, kernel problems) are not proactively detected. Resources: eks-review-cluster Current State: EKS Node Monitoring Agent add-on is not installed. Consider enabling it for enhanced node-level health monitoring and diagnostics.

The EKS Node Monitoring Agent provides enhanced node health monitoring, detecting issues like disk pressure, kernel deadlocks, and container runtime problems before they impact workloads.

Remediation:

aws eks create-addon \
  --cluster-name eks-review-cluster \
  --addon-name eks-node-monitoring-agent

Considerations: The agent runs as a DaemonSet and consumes minimal resources. No expected negative impact on existing workloads.

47. VPC CNI Service Account IAM Role¶

Medium | Networking | AWS CLI Impact: Using the node instance profile for VPC CNI grants broader permissions than necessary, violating least-privilege principles. Resources: EKS Add-on: vpc-cni (no dedicated IAM role) Current State: VPC CNI using node instance profile instead of dedicated IAM role

The VPC CNI plugin is using the node's instance profile IAM role instead of a dedicated IAM role. This means the CNI has access to all permissions granted to the node role, which may be broader than needed.

Remediation:

# Create a dedicated IAM role for VPC CNI
# TEMPLATE — replace <ACCOUNT_ID> and <OIDC_ID> before running
aws iam create-role \
  --role-name eks-review-cluster-vpc-cni-role \
  --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Allow",
      "Principal": {"Federated": "arn:aws:iam::<ACCOUNT_ID>:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/<OIDC_ID>"},
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {"StringEquals": {"oidc.eks.us-east-1.amazonaws.com/id/<OIDC_ID>:sub": "system:serviceaccount:kube-system:aws-node"}}
    }]
  }'

aws iam attach-role-policy \
  --role-name eks-review-cluster-vpc-cni-role \
  --policy-arn arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy

# Update the VPC CNI add-on with the dedicated role
aws eks update-addon \
  --cluster-name eks-review-cluster \
  --addon-name vpc-cni \
  --service-account-role-arn arn:aws:iam::<ACCOUNT_ID>:role/eks-review-cluster-vpc-cni-role

Considerations: Updating the add-on IAM role triggers a rolling update of the aws-node DaemonSet. Brief network disruption is possible during the rollout. Plan during a maintenance window.

48. PreStop Hook Configuration for Ingress Deployments¶

Medium | Networking | Manifest Impact: Ingress-backed deployments without preStop hooks will drop in-flight requests during pod termination. Resources: 5 deployments in default namespace — nginx-29, nginx-30, nginx-28, nginx-27, nginx-26 Current State: 5 deployments behind Ingress missing PreStop hooks

Five deployments that are targets of Ingress resources lack preStop hooks. When these pods are terminated, the ALB/NLB may still route traffic to them during the deregistration delay, causing 502/504 errors.

Remediation:

# Add preStop hooks to ingress-backed deployments
spec:
  template:
    spec:
      containers:
      - name: nginx
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "sleep 15"]
      terminationGracePeriodSeconds: 30

Considerations: The sleep duration should match or exceed the ALB deregistration delay (default 300s for ALB, but typically configured lower). Coordinate with target group deregistration settings.

Low Priority Issues¶

49. Check if Quotas are defined at the namespace level¶

Low | Security | Manifest Impact: Without namespace quotas, resource consumption is unbounded, potentially affecting cluster stability. Resources: 1 namespace (default) Current State: Found 1 namespaces without Resource Quotas

The default namespace lacks Resource Quotas. This is the same underlying issue as finding #43 (Namespace ResourceQuotas).

Remediation: See finding #43 for remediation steps.

Considerations: Same as finding #43.

50. Disable ServiceAccount token mounts¶

Low | Security | Manifest Impact: Unnecessary token mounts provide API credentials to pods that don't need them, increasing the attack surface. Resources: 57 workloads (166 pods) across 2 namespaces — default: 50 workloads (e.g., ReplicaSet/nginx-1-7f679f8d4c, ReplicaSet/nginx-10-768f774ff, ReplicaSet/nginx-11-56df9f6659, and 47 more); kube-system: DaemonSet/aws-node, DaemonSet/ebs-csi-node, DaemonSet/kube-proxy, and 4 more Current State: Found 57 workloads (166 pods) with ServiceAccount token mounting enabled

57 workloads have ServiceAccount tokens automatically mounted. For workloads that don't interact with the Kubernetes API, these tokens are unnecessary and represent an attack vector.

Remediation:

# Disable token mounting at the pod level
spec:
  template:
    spec:
      automountServiceAccountToken: false

Considerations: Do not disable for workloads that need Kubernetes API access (controllers, operators, service mesh sidecars). Apply selectively to application workloads.

51. Use Vertical Pod Autoscaler¶

Low | Resiliency | Helm/Manifest Impact: Without VPA, resource requests/limits must be manually tuned, often leading to over- or under-provisioning. Resources: vpa-controller Current State: VPA is not installed or not properly configured

VPA automatically adjusts resource requests based on actual usage. Without it, resource specifications are static and may not reflect actual workload needs.

Remediation:

# Install VPA via Helm
helm repo add fairwinds-stable https://charts.fairwinds.com/stable
helm install vpa fairwinds-stable/vpa -n vpa --create-namespace

# Or install from the Kubernetes autoscaler repository
kubectl apply -f https://github.com/kubernetes/autoscaler/releases/latest/download/vpa-v1-crd-gen.yaml
kubectl apply -f https://github.com/kubernetes/autoscaler/releases/latest/download/vpa-v1-rbac.yaml

Considerations: Start VPA in Off or Initial mode to get recommendations without automatic updates. VPA and HPA should not target the same metric (CPU/memory) simultaneously.

52. Use a Service Mesh¶

Low | Resiliency | Helm/Manifest Impact: Without a service mesh, there is no automatic mTLS, traffic management, or observability for service-to-service communication. Resources: Cluster-wide Current State: No service mesh implementation found

No service mesh (Istio, Linkerd, App Mesh) was detected. A service mesh provides mTLS encryption, traffic management, retries, circuit breaking, and observability for inter-service communication.

Remediation:

# Option 1: Install Istio
istioctl install --set profile=default

# Option 2: Install Linkerd
linkerd install | kubectl apply -f -
linkerd check

Considerations: Service meshes add operational complexity and resource overhead (sidecar proxies). Evaluate whether your microservices architecture benefits from mesh features before adopting. Start with a subset of namespaces.

53. Use custom metrics scaling¶

Low | Resiliency | Helm/Manifest Impact: Without custom metrics, autoscaling is limited to CPU/memory, which may not reflect actual application load. Resources: custom-metrics-api Current State: No custom metrics scaling infrastructure found

No custom metrics API adapter (Prometheus Adapter, KEDA, CloudWatch Metrics Adapter) was detected. Custom metrics enable HPA to scale based on application-specific metrics like request rate, queue depth, or latency.

Remediation:

# Install KEDA for event-driven autoscaling
helm repo add kedacore https://kedacore.github.io/charts
helm install keda kedacore/keda -n keda --create-namespace

# Or install Prometheus Adapter
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install prometheus-adapter prometheus-community/prometheus-adapter -n monitoring

Considerations: Requires a monitoring solution to be in place first. Custom metrics scaling adds complexity — start with CPU/memory-based HPA and add custom metrics for workloads with specific scaling requirements.

Summary¶

Priority	Count	Key Actions
High	27	Enable control plane logs, restrict API endpoint, fix Karpenter configuration, add health probes, configure resource limits
Medium	16	Install monitoring/logging, add PDBs, configure Network Policies, set up HPA
Low	6	Install VPA, disable unnecessary SA token mounts, evaluate service mesh

Recommended Remediation Order¶

Immediate (Day 1): Enable control plane logs, restrict API endpoint CIDR, configure Karpenter NodePool limits
Short-term (Week 1): Install monitoring solution, fix Karpenter AMI pinning and instance exclusions, enable envelope encryption, scale Karpenter to 2 replicas
Medium-term (Week 2-3): Add resource requests/limits, health probes, and PDBs to all workloads; move nodes to private subnets
Ongoing: Implement Network Policies, migrate to non-root containers, adopt read-only root filesystems, configure HPA