Source

This page is generated from skills/eks-genai/references/compute-hardware.md. Edit the source, not this page.

Compute & Hardware — NVIDIA GPU vs AWS Neuron

The single most-impactful decision in any GenAI-on-EKS architecture. Do not reflexively pick NVIDIA GPU. For Transformer-family LLMs, evaluate AWS Neuron (Trainium/Inferentia) first: it often wins on price-performance versus NVIDIA GPU, but validate per workload.

Layer-1 Decision Rule

Right hardware = f(workload type × model family × latency × cost posture × team skill × timeline) — never one dimension alone.

• Default to AWS Neuron (Trn2/Trn1 training, Inf2 inference) when: model is Transformer-family (Llama, Mistral, Qwen, Falcon) + framework is PyTorch/vLLM + cost-conscious + team can absorb a 1–2 week compilation ramp.
• Default to NVIDIA GPU (g6/g6e inference, p5/p5e training) when: fastest time-to-first-success matters, CUDA-only dependencies exist, novel/non-Transformer architectures, or multi-modal models.

GPU vs Neuron Decision Matrix

Workload signal	Recommend NVIDIA GPU when…	Recommend AWS Neuron when…
Training — frontier (70B+) / novel arch	Custom CUDA kernels; H100/H200 ecosystem dependency; cutting-edge research	Model is Transformer-family — Trn2 delivers up to 50% cost-to-train savings vs comparable GPU (trn1 instance page)
Training — fine-tuning (LoRA/PEFT/full)	Fastest time-to-first-success; broadest tooling	Strong fit if base model has Neuron support (most popular FMs do)
Inference — high-throughput LLM (7B–70B)	g6 (L4) or g6e (L40S) — broad vLLM-native ecosystem	inf2 — up to 40% better price-performance for vLLM-supported models (inf2 instance page)
Inference — vision / audio / multi-modal	Required — Neuron support uneven for non-Transformer architectures	Limited — verify model-specific support before committing
Inference — sub-100ms TTFT	Best raw latency on H100/H200	Acceptable on inf2 with TP/PP tuning + ahead-of-time compilation
Customer skill	Existing CUDA / NVIDIA tooling investment	Willing to add Neuron SDK + compilation step (torch-neuronx)
Time-to-first-success	Fastest path; least new tooling	+1–2 weeks ramp; ROI realized at production scale
Production cost optimization	Acceptable but more expensive per token	Recommend for steady-state production once model is stabilized

NVIDIA GPU Instance Quick-Reference

Network/EFA values are the per-instance maximums from the EC2 instance-type pages; EFA generation follows the Nitro version (Nitro v3 = EFA v1, no RDMA; Nitro v4+ = EFA v2/v3, with RDMA).

Instance	Accelerator	GPU Memory	Network / EFA	Best for
p5.48xlarge	8× NVIDIA H100 SXM5	640 GB HBM3	3200 Gbps, EFA v2 (GPUDirect RDMA)	Frontier pre-training, 100B+ models, multi-node FSDP
p5e.48xlarge	8× NVIDIA H200 SXM5	~1.1 TB HBM3e	3200 Gbps, EFA v2 (GPUDirect RDMA)	Same as p5 + larger memory for longer contexts
g6e.48xlarge	8× NVIDIA L40S	384 GB GDDR6	400 Gbps, EFA v2	70B inference, multi-GPU PEFT fine-tune
g6e.2xlarge	1× NVIDIA L40S	48 GB GDDR6	Up to 20 Gbps (no EFA)	Single-GPU inference (7B–13B), dev/test, workshop validated
g6.12xlarge	4× NVIDIA L4	96 GB GDDR6	40 Gbps, EFA v2	7B–30B inference, cost-sensitive production
g6.48xlarge	8× NVIDIA L4	192 GB GDDR6	100 Gbps, EFA v2	Larger multi-GPU L4 inference fleets
g5.48xlarge	8× NVIDIA A10G	192 GB GDDR6	100 Gbps, EFA v1 (no RDMA)	Multi-modal, NVIDIA NIM, dev/test, legacy workloads

Workshop-validated: The GenAI-on-EKS NVIDIA workshop runs on g6e.2xlarge (1× L40S, 48 GB) on EKS Auto Mode (Kubernetes 1.34) with Bottlerocket. GPU capacity reserved via On-Demand Capacity Reservation (ODCR) patched into the Karpenter EC2NodeClass via capacityReservationSelectorTerms.

AWS Neuron Instance Quick-Reference

Note: Inf2 instances do not support EFA — their inf2.48xlarge chip-to-chip interconnect is NeuronLink (intra-instance), and their external networking is standard ENA. EFA (inter-node fabric) is a Trainium-and-GPU feature.

Instance	Accelerator	HBM	Network / EFA	Best for
trn2.48xlarge	16× Trainium2	1.5 TB HBM3	3.2 Tbps, EFA v3 (RDMA)	Pre-training (3× compute vs Trn1), large-scale fine-tuning
trn1.32xlarge	16× Trainium	512 GB HBM2e	800 Gbps, EFA v2	Pre-training, fine-tuning — up to 50% cost-to-train savings
inf2.48xlarge	12× Inferentia2	384 GB HBM2e	100 Gbps ENA (no EFA); NeuronLink chip-to-chip	High-throughput LLM inference (30B–70B)
inf2.8xlarge	1× Inferentia2	32 GB HBM2e	Up to 25 Gbps ENA (no EFA)	Dev/test inference, smaller models (7B–13B)

When Each Accelerator Wins — Summary

Scenario	Winner	Why
Cost-sensitive Transformer LLM training at scale	Trainium (Trn2/Trn1)	Up to 50% cost-to-train savings; EFA for multi-node
Cost-sensitive Transformer LLM inference at scale	Inferentia2 (Inf2)	Up to 40% better price-performance; vLLM+Neuron supported
Fastest time-to-production, any model	NVIDIA GPU (g6e/p5)	Broadest ecosystem; zero compilation ramp
Novel architecture / custom CUDA kernels	NVIDIA GPU	Neuron compilation may not support custom ops
Multi-modal (vision+language, audio)	NVIDIA GPU	Neuron support is model-specific; verify first
Shared dev cluster, multiple small models	NVIDIA GPU (g6/g5)	MIG/time-slicing enable multi-tenant sharing

GPU Optimization Techniques

Multi-Instance GPU (MIG) — H100/A100 only

Partitions a single physical GPU into up to 7 isolated instances, each with dedicated memory, cache, and compute. Use for:

• Multi-tenant dev clusters — isolate teams on a single p5 node
• Small-model inference — run multiple 7B models on one H100 slice
• Requires NVIDIA device plugin configured with MIG strategy (mixed or single)

MIG profiles on H100 (p5.48xlarge):

Profile	GPU Memory	Compute	Use case
1g.10gb	10 GB	1/7 SM	Tiny models, embedding inference
2g.20gb	20 GB	2/7 SM	7B quantized inference
3g.40gb	40 GB	3/7 SM	7B–13B FP16 inference
7g.80gb	80 GB	Full GPU	Single-tenant training/inference

GPU Time-Slicing

Multiplexes multiple pods onto a single GPU via temporal sharing (no memory isolation). Use for:

• Dev/test environments where isolation is not critical
• Low-utilization workloads that don't saturate GPU compute
• Simpler than MIG; no H100/A100 requirement — works on any NVIDIA GPU
• Configure via nvidia.com/gpu.replicas in the device plugin ConfigMap

# Time-slicing ConfigMap for nvidia-device-plugin
apiVersion: v1
kind: ConfigMap
metadata:
  name: nvidia-device-plugin-config
data:
  config.yaml: |
    version: v1
    sharing:
      timeSlicing:
        replicas: 4    # 4 pods share each physical GPU

Caution: Time-slicing provides no memory isolation — one pod's OOM kills the GPU context for all co-located pods. Use only for dev/test.

Dynamic Resource Allocation (DRA): Kubernetes 1.34+

Fine-grained GPU partitioning managed by the Kubernetes scheduler. Use for:

• Clusters that need scheduler-aware GPU sharing (vs static MIG)
• Karpenter static capacity only (no dynamic provisioning); not on Auto Mode. The NVIDIA DRA driver is supported on Karpenter static-capacity NodePools, EKS managed node groups, and self-managed nodes, but does not support dynamic Karpenter provisioning and is not supported on EKS Auto Mode (Manage NVIDIA GPU devices with DRA). On EKS the DRA drivers require Kubernetes 1.34+, where DRA is GA at resource.k8s.io/v1 (used in the example below; the beta resource.k8s.io/v1beta1 API dates to 1.32).

# DRA ResourceClaim example (Karpenter static capacity / managed NG / self-managed; not Auto Mode)
apiVersion: resource.k8s.io/v1
kind: ResourceClaim
metadata:
  name: gpu-slice
spec:
  devices:
    requests:
      - name: gpu
        exactly:                        # v1 (GA in K8s 1.34): deviceClassName/selectors/allocationMode nest under exactly
          deviceClassName: gpu.nvidia.com

Decision: MIG vs Time-Slicing vs DRA

Technique	Isolation	Memory isolation	Karpenter compatible	Best for
MIG	Hardware-level	✅ Yes	✅ Yes	Production multi-tenant on H100/A100
Time-slicing	None (temporal only)	❌ No	✅ Yes	Dev/test density, low-utilization workloads
DRA	Scheduler-managed	✅ Yes	Static capacity only (no dynamic provisioning); not on Auto Mode	Fine-grained sharing on Karpenter static NodePools, managed node groups, or self-managed nodes

Neuron Compilation — The Ramp Cost

Neuron requires ahead-of-time compilation (torch-neuronx or neuronx-distributed-inference). This is the primary adoption friction vs NVIDIA GPU.

What to expect:

• First model compilation: 1–2 weeks (includes learning the SDK, debugging graph breaks, tuning TP/PP)
• Subsequent model versions: hours to days (incremental, reuse compilation cache)
• Ship compiled artifacts via S3 or bake into container image — never compile at pod startup

Mitigation: Pre-compile models offline in a CI pipeline. Store compiled .neff files in S3. Reference them from the serving container at startup. This removes compilation from the critical path entirely.

Warning (SDK version skew): Precompiled Neuron artifacts (.neff) must be compiled with a Neuron SDK version compatible with the runtime version on the serving nodes. A compile-vs-runtime SDK mismatch is a common serve-time failure, so pin and match the Neuron SDK across build and serve.

# Neuron pod requesting NeuronCores (device plugin path)
resources:
  limits:
    aws.amazon.com/neuroncore: "2"     # 2 NeuronCores for this workload
  requests:
    aws.amazon.com/neuroncore: "2"

Capacity Planning Guidance

Workload	Start with	Scale signal
Inference (7B)	1× g6e.2xlarge or 1× inf2.8xlarge	TTFT p99 > target → add replicas
Inference (30B–70B)	4× g6e.12xlarge or 1× inf2.48xlarge	GPU memory utilization > 85% → upsize
Fine-tuning (7B–13B)	1× g6e.48xlarge or 1× trn1.32xlarge	Training throughput (tokens/sec)
Pre-training (70B+)	8–64× p5.48xlarge or 4–32× trn2.48xlarge	EFA all-reduce latency, loss convergence

Rule: Start small, measure, scale. Over-provisioning GPUs is the second-most-expensive mistake (after wrong hardware family).

Cost-to-Train / Price-Performance Claims (AWS-published)

Claim	Source
Trainium: up to 50% cost-to-train savings vs comparable EC2 GPU instances	aws.amazon.com/ec2/instance-types/trn1
Inferentia2: up to 40% better price-performance vs comparable EC2 GPU instances	aws.amazon.com/ec2/instance-types/inf2
Inferentia2: up to 50% better performance/watt vs comparable EC2 GPU	aws.amazon.com/ec2/instance-types/inf2
Capacity Blocks for ML: guaranteed capacity access, upfront market-based pricing (not a guaranteed on-demand discount)	aws.amazon.com/ec2/capacityblocks/pricing

Always provide directional ranges with caveats — actual savings depend on model size, traffic pattern, batch size, sequence length, and configuration. Never give point cost estimates.

EKS Auto Mode + GPU Note

On EKS Auto Mode, the NVIDIA driver and device plugin are embedded in the Bottlerocket AMI — no gpu-operator DaemonSet or separate nvidia-device-plugin DaemonSet is needed. The "install nvidia-device-plugin" step you see in most guides applies to self-managed / standard EKS only. Auto Mode also auto-enables SOCI parallel pull and unpack on G/P/Trn instance families with local NVMe (always on as of Nov 19, 2025 — no configuration required), parallelizing image download/decompression for faster GPU-pod starts. Note this is SOCI parallel pull, distinct from SOCI lazy/index-based loading (which still needs a pre-built SOCI index in ECR).

Sources

• Best Practices for AI/ML Workloads on Amazon EKS
• EKS for AI/ML
• Amazon EC2 Trn1 Instances
• Amazon EC2 Inf2 Instances
• EC2 Capacity Blocks for ML
• EKS Optimized Accelerated AMIs
• Manage Neuron devices on Amazon EKS
• awslabs/ai-on-eks