AI Platform Engineering — Reference¶
Look-up facts for the Kubernetes GPU and LLM-serving stack: component versions, Kubernetes feature stages, MIG profiles, engine flags, metrics, ports, and hardening checklists. For the reasoning behind these components see Explanation; for step-by-step tasks see How-to Guides.
Version snapshot
Versions below were checked on 2026-09-25 against project repositories (tags, README, CHANGELOG), PyPI and release notes. This stack moves monthly. Re-check before pinning. "Date TBD" means the tag exists but the release date could not be confirmed from a primary source during this pass.
Component Version Matrix¶
| Component | Layer | Latest version (date) | License | Governance |
|---|---|---|---|---|
| Kubernetes | Orchestration | 1.37 "Garhwal" (2026-08-26) | Apache-2.0 | CNCF graduated |
| NVIDIA GPU Operator | GPU lifecycle | v26.7.1 (2026-09-23) | Apache-2.0 | NVIDIA |
| NVIDIA device plugin | GPU advertisement | v0.20.1 (date TBD; bundled in GPU Operator v26.7.1) | Apache-2.0 | NVIDIA |
| DRA Driver for NVIDIA GPUs | GPU allocation (DRA) | 0.5.0 Helm chart (date TBD) | Apache-2.0 | Kubernetes SIG project (kubernetes-sigs/dra-driver-nvidia-gpu), donated by NVIDIA at KubeCon EU 2026 |
| DCGM Exporter | GPU metrics | 4.6.1-4.8.4 (image tag in GPU Operator v26.7.1) | Apache-2.0 | NVIDIA |
| Kueue | Job queueing / quota | v0.19.6 (date TBD; README install pin as of 2026-09) | Apache-2.0 | Kubernetes SIG Scheduling subproject |
| Volcano | Batch scheduler | v1.15.2 (latest tag; date TBD) | Apache-2.0 | CNCF incubating |
| KAI Scheduler | AI scheduler | v0.18.0 (2026-09-23) | Apache-2.0 | CNCF sandbox (from Run:ai, open-sourced 2025-04) |
| Kubeflow Trainer | Training operator (TrainJob) | v2.3.0 (2026-08) | Apache-2.0 | Kubeflow (CNCF incubating) |
| KubeRay | Ray operator | v1.7.1 (date TBD) | Apache-2.0 | Ray project |
| Ray | Distributed compute | 2.58.0 (2026-08-23) | Apache-2.0 | Ray project (Anyscale-led) |
| vLLM | Inference engine | 0.30.0 (2026-09-22) | Apache-2.0 | vLLM project |
| vLLM production-stack | Helm reference stack | chart 0.1.12 (date TBD) | Apache-2.0 | vLLM project |
| KServe | Model serving platform | 0.20.0 (2026-08-06); 0.21.0rc1 (2026-09-23) | Apache-2.0 | CNCF incubating |
| Gateway API Inference Extension | Inference routing API | v1.6.2 (latest tag; date TBD) | Apache-2.0 | Kubernetes SIG Network subproject |
| llm-d | Distributed inference stack | v0.7 (2026-05) | Apache-2.0 | CNCF sandbox (joined 2026-03) |
| NVIDIA Dynamo | Datacenter inference orchestration | 1.5.0 (2026-09-19) | Apache-2.0 | NVIDIA (ai-dynamo org) |
Sources: Kubernetes v1.37 release, GPU Operator v26.7.1, Kueue README, KAI Scheduler, Kubeflow Trainer, Ray on PyPI, vLLM on PyPI, KServe on PyPI, llm-d, ai-dynamo on PyPI.
Kubernetes DRA and AI Scheduling Feature Stages¶
Dynamic Resource Allocation (DRA) core (resource.k8s.io/v1) went GA in Kubernetes 1.34 (2025-08). The table tracks the follow-on KEPs that matter for GPUs.
| Feature (KEP) | Alpha | Beta | Stable | Status in 1.37 |
|---|---|---|---|---|
| DRA structured parameters (KEP-4381) | 1.30 | 1.32 | 1.34 | GA, resource.k8s.io/v1 on by default |
| DRA admin access (KEP-5018) | 1.33 | 1.34 | 1.36 | GA |
DRA prioritized list / firstAvailable (KEP-4816) |
1.33 | 1.34 | 1.36 | GA |
| DRA device taints and tolerations (KEP-5055) | 1.33 | 1.36 | 1.37 | GA |
| DRA extended resource mapping (KEP-5004) | 1.34 | 1.36 | 1.37 | GA: a DRA driver can satisfy classic nvidia.com/gpu requests |
| DRA partitionable devices (KEP-4815) | 1.33 | 1.36 | TBD | Beta, on by default (dynamic MIG-style partitions) |
| DRA consumable capacity (KEP-5075) | 1.34 | 1.36 | 1.38 (planned) | Beta |
Resource health in pod status, allocatedResourcesStatus (KEP-4680) |
1.31 | 1.36 | TBD | Beta |
| Gang scheduling, Workload / PodGroup API (KEP-4671) | 1.35 | 1.37 | 1.38 (planned) | Beta (v1beta1); hierarchical CompositePodGroup is alpha |
Sources: KEP metadata in kubernetes/enhancements, Kubernetes v1.34 DRA GA, Kubernetes v1.36 DRA updates, Kubernetes v1.37 DRA updates.
Stages are per Kubernetes minor
Managed offerings (EKS, GKE, AKS, OpenShift) often lag upstream and may keep beta gates off. For example, OpenShift made DRA GA only in 4.21 (2026-03). Check your provider before relying on a beta DRA feature.
Scheduler and Queueing Feature Matrix¶
| Capability | kube-scheduler (1.37) | Kueue | Volcano | KAI Scheduler |
|---|---|---|---|---|
| Role | Default pod scheduler | Admission and quota layer that works with kube-scheduler | Replacement batch scheduler plus job controller | Replacement scheduler for AI clusters |
| Gang scheduling | Workload/PodGroup API (beta) | All-or-nothing via waitForPodsReady |
minAvailable on PodGroup/Job |
PodGroups, hierarchical PodGroups |
| Queues and quotas | ResourceQuota only | ClusterQueue, LocalQueue, cohorts, borrowing | Queue CRD with weights and capacity | Hierarchical queues, over-quota weights |
| Fair sharing | No | Fair sharing, admission fair sharing | DRF, proportion plugin | DRF, time-based fairshare |
| Topology-aware placement | Via DRA and affinity | Topology-Aware Scheduling (TAS) | Network topology-aware plugin | TAS, including hierarchical |
| Fractional GPUs | Via DRA drivers | No (counts resources) | vGPU / HAMi integrations | GPU sharing (fractions) |
| DRA support | Native | KueueDRAIntegration |
Yes | Yes, including GB200/GB300 ComputeDomains |
| Multi-cluster | No | MultiKueue | No (Karmada integration) | No |
Sources: Kueue README, Volcano README, KAI Scheduler README.
NVIDIA GPU Operator Components¶
| Component | Purpose | Version in GPU Operator v26.7.1 | Default |
|---|---|---|---|
| NVIDIA Driver (containerized) | Kernel-level GPU access | 595.91.07 | Enabled |
| NVIDIA Container Toolkit | Container runtime GPU integration, CDI specs | v1.20.1 | Enabled |
| NVIDIA Device Plugin | Advertises nvidia.com/gpu and MIG resources |
v0.20.1 | Enabled |
| GPU Feature Discovery (GFD) | GPU model, memory, driver labels on nodes | v0.20.1 | Enabled |
| Node Feature Discovery (NFD) | Hardware labels (PCI vendor 10de) |
Subchart | Enabled |
| DCGM Exporter | Prometheus GPU metrics | 4.6.1-4.8.4 | Enabled |
| MIG Manager | Applies MIG layouts from the nvidia.com/mig.config node label |
v0.15.1 | Enabled |
| Node Status Exporter | Operator status metrics | - | Disabled |
| GPUDirect Storage (GDS), GDRCopy | RDMA storage and copy paths | 2.29.4 / v2.6 | Disabled |
| vGPU / VFIO / Kata managers | Virtualized and sandboxed workloads | various | Disabled unless sandboxWorkloads.enabled |
CDI (cdi.enabled: true) is the default in current charts. Source: values.yaml at v26.7.1.
MIG Profiles¶
MIG is available on data-center GPUs from Ampere onward: A30, A100/A800, H100/H800 (SXM, PCIe, NVL), H200, GH200, B200/GB200, and RTX PRO 6000 Blackwell. It is not available on consumer or A10/L4/L40-class cards. A GPU supports up to 7 GPU instances.
A100 80GB profiles (H100 80GB uses the same names; SM counts differ):
| Profile | GPU memory | SMs (A100) | Compute slices | Max instances | nvidia-smi profile ID (A100 80GB) |
|---|---|---|---|---|---|
1g.10gb |
10 GB | 14 | 1 | 7 | 19 |
1g.20gb |
20 GB | 14 | 1 | 4 | 15 |
2g.20gb |
20 GB | 28 | 2 | 3 | 14 |
3g.40gb |
40 GB | 42 | 3 | 2 | 9 |
4g.40gb |
40 GB | 56 | 4 | 1 | 5 |
7g.80gb |
80 GB | 98 | 7 | 1 | 0 |
Smallest profiles on other GPUs, from the GPU Operator default mig-parted config: A100 40GB 1g.5gb x7, A30 1g.6gb x4, H100 NVL / GH200 1g.12gb x7, B200/GB200 1g.23gb x7, RTX PRO 6000 1g.24gb x4. Profiles with +me add a media engine to a single instance.
Profile IDs are GPU-specific
Always run nvidia-smi mig -lgip on the target GPU before scripting -cgi IDs. The IDs above apply to A100 80GB only.
Sources: NVIDIA MIG User Guide, GPU Operator state-mig-manager default ConfigMap (v26.7.1).
GPU Sharing Options Compared¶
| Method | Isolation | Granularity | Configured in | Typical use |
|---|---|---|---|---|
Whole GPU (nvidia.com/gpu: 1) |
Full | 1 GPU | Default device plugin | Production inference, training |
| MIG | Hardware (SMs, L2, memory controllers) | Fixed profiles, up to 7 per GPU | MIG Manager / DRA partitionable devices | Multi-tenant inference, QoS |
| Time-slicing (device plugin) | None for memory or faults. Round-robin compute | N replicas per GPU | Device plugin sharing.timeSlicing |
Dev/test, notebooks |
| MPS (device plugin) | Memory limits per client, shared fault domain | N replicas per GPU | Device plugin sharing.mps |
Many small inference processes |
| Fractional GPU (KAI / Run:ai) | Scheduler-enforced memory fraction | Fraction or GiB | KAI Scheduler annotations | Mixed interactive workloads |
| DRA shared claim | Depends on driver config | Per ResourceClaim | ResourceClaim / DeviceClass | Pods or containers sharing one GPU |
vLLM Key Parameters¶
Defaults as of vLLM v0.30.0 (vllm serve, OpenAI-compatible server).
| Parameter | Default | Purpose |
|---|---|---|
--tensor-parallel-size |
1 | GPUs per replica for tensor parallelism |
--pipeline-parallel-size |
1 | Pipeline stages (multi-node) |
--data-parallel-size |
1 | Data-parallel replicas inside one server (useful for MoE with expert parallelism) |
--gpu-memory-utilization |
0.92 since v0.20 (0.9 through v0.19) | Fraction of GPU memory vLLM may use for weights, activations and KV cache |
--max-model-len |
Model config | Maximum context length |
--max-num-seqs |
256 (A100 and GPUs under 70 GiB); 1024 (H100/H200 and GPUs of 160 GiB or more) | Maximum concurrent sequences per step |
--max-num-batched-tokens |
2048 / 8192 / 16384 by GPU class (server) | Token budget per scheduler step (chunked prefill) |
--dtype |
auto |
Weight/activation precision |
--kv-cache-dtype |
auto |
Set fp8 to halve KV cache memory |
--quantization |
Auto-detected from checkpoint | For example awq, gptq, fp8, compressed-tensors, modelopt_fp4, mxfp4 |
--enable-prefix-caching |
On in the V1 engine | Reuses KV blocks for shared prefixes |
--enforce-eager |
false | Disables CUDA graphs (debug, lower memory) |
--api-key |
unset | Static bearer token for the OpenAI API (no per-user authz) |
Source: vllm/engine/arg_utils.py and vllm/config/cache.py at tag v0.30.0 in vllm-project/vllm.
SqueezeLLM and the V0 engine are gone
Older guides list squeezellm as a quantization method and describe the V0 engine. Neither exists in current vLLM releases.
DCGM Exporter Metrics¶
Enabled by default in dcgm-exporter default-counters.csv:
| Metric | Type | Meaning |
|---|---|---|
DCGM_FI_DEV_GPU_UTIL |
gauge | Percent of time one or more kernels ran (coarse; 100% does not mean the SMs are full) |
DCGM_FI_PROF_GR_ENGINE_ACTIVE |
gauge | Ratio of time the graphics/compute engine was active (better utilization signal) |
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE |
gauge | Tensor Core pipe activity (key for LLM workloads) |
DCGM_FI_PROF_DRAM_ACTIVE |
gauge | Memory bandwidth utilization ratio |
DCGM_FI_DEV_MEM_COPY_UTIL |
gauge | Memory copy utilization (%) |
DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_FREE / DCGM_FI_DEV_FB_RESERVED |
gauge | Framebuffer memory (MiB) |
DCGM_FI_DEV_GPU_TEMP / DCGM_FI_DEV_MEMORY_TEMP |
gauge | Temperatures (C) |
DCGM_FI_DEV_POWER_USAGE |
gauge | Power draw (W) |
DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION |
counter | Energy since boot (mJ) |
DCGM_FI_DEV_SM_CLOCK / DCGM_FI_DEV_MEM_CLOCK |
gauge | Clocks (MHz) |
DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL |
gauge | NVLink bandwidth, all lanes |
DCGM_FI_PROF_PCIE_TX_BYTES / DCGM_FI_PROF_PCIE_RX_BYTES |
gauge | PCIe traffic |
DCGM_FI_DEV_PCIE_REPLAY_COUNTER |
counter | PCIe replays (link health) |
DCGM_FI_DEV_XID_ERRORS |
gauge | Last XID error code (driver-reported GPU faults) |
DCGM_FI_DEV_UNCORRECTABLE_REMAPPED_ROWS / DCGM_FI_DEV_ROW_REMAP_FAILURE |
counter / gauge | HBM row remapping health |
Available but commented out by default: DCGM_FI_PROF_SM_ACTIVE, DCGM_FI_PROF_SM_OCCUPANCY, DCGM_FI_PROF_PIPE_FP16_ACTIVE, DCGM_FI_PROF_PIPE_FP32_ACTIVE, DCGM_FI_PROF_PIPE_FP64_ACTIVE. Source: NVIDIA/dcgm-exporter etc/default-counters.csv.
GPU Interconnect Bandwidth¶
| Interconnect | Bandwidth | Use case |
|---|---|---|
| NVLink 5 (B200) | 1.8 TB/s per GPU | Intra-node and NVL72 rack-scale domain |
| NVLink 4 (H100) | 900 GB/s per GPU | Intra-node GPU communication |
| NVLink 3 (A100) | 600 GB/s per GPU | Intra-node GPU communication |
| PCIe Gen5 x16 | ~64 GB/s per direction | Host to GPU, GPUs without NVLink |
| InfiniBand NDR | 400 Gb/s (50 GB/s) per port | Inter-node communication |
| InfiniBand HDR | 200 Gb/s (25 GB/s) per port | Inter-node communication |
| Ethernet (RoCE) | 100-800 Gb/s per port | Cost-effective inter-node |
Inter-node bandwidth per GPU is roughly 10-30x lower than NVLink, which is why topology-aware placement matters. Source: NVIDIA product datasheets (not re-fetched in this pass; figures are the published per-GPU aggregate bandwidths).
Ray Ports and Security Surfaces¶
| Component | Default port | Risk | Mitigation |
|---|---|---|---|
| Ray Dashboard / Jobs API | 8265 | Unauthenticated job submission is remote code execution | NetworkPolicy, authenticating ingress, Ray token auth (KubeRay v1.5+) |
| GCS server | 6379 | Cluster control plane access | Pod-to-pod only, never expose externally, TLS |
| Ray Client | 10001 | Remote code execution | TLS, authentication proxy, or disable |
| Object store (Plasma) | in-node shared memory | Data visible to all tasks in the cluster | Namespace isolation, trusted workloads only |
| Ray Serve HTTP proxy | 8000 | Public inference endpoint | Gateway with authn and rate limits |
Source: Ray cluster key concepts, KubeRay v1.5 announcement.
Encryption Matrix¶
Data in Transit¶
| Path | Protocol | Configuration |
|---|---|---|
| Client → Inference API | HTTPS/TLS | API Gateway with TLS termination |
| Pod → Pod (inference) | mTLS | Service mesh (Istio/Linkerd) |
| Ray inter-node | TLS | Ray TLS configuration (RAY_USE_TLS) |
| GPU node → Storage | HTTPS | Encrypted object store endpoints |
| NVLink (intra-node) | N/A | Physical hardware path, not encrypted by default |
| RDMA/InfiniBand | N/A | Typically trusted fabric, IPsec optional |
Data at Rest¶
| Component | Encryption Method |
|---|---|
| Model weights on PV | Encrypted PersistentVolumes (StorageClass encryption) |
| Training data | Encrypted object storage (SSE-S3, SSE-KMS) |
| KV Cache (GPU memory) | Not encrypted (volatile GPU memory) unless Confidential Computing mode is on (H100 and later) |
| KV cache offload (CPU/SSD tiers, LMCache, KVBM) | Host memory or disk. Encrypt the disk tier |
| Model registry | Application-level encryption + encrypted backend storage |
| Logs/metrics | Encrypted storage backend |
Checklist for Regulated Environments¶
- MIG or dedicated GPU allocation (no time-slicing for cross-tenant workloads)
- NetworkPolicies isolating inference endpoints and Ray ports
- mTLS between all AI platform services
- Model weight encryption at rest and access logging
- GPU utilization monitoring with anomaly detection
- RBAC and ResourceQuota restricting GPU resource creation to authorized teams
- Container image scanning and allowlisting on GPU nodes
- Audit logging for all GPU resource allocation events (Pods, ResourceClaims, queue objects)
- Training data access controls and lineage tracking
- Inference API rate limiting and authentication at the gateway
- Model provenance: checksums or signatures verified before load
Glossary¶
| Term | Meaning |
|---|---|
| CDI | Container Device Interface, the runtime spec for injecting devices into containers |
| ComputeDomain | DRA driver abstraction guaranteeing Multi-Node NVLink (IMEX) reachability between pods |
| DeviceClass | DRA object selecting a class of devices (for example gpu.nvidia.com, mig.nvidia.com) |
| EPP | Endpoint Picker, the ext-proc service that picks a model-server pod for each request in the Gateway API Inference Extension |
| Gang scheduling | All pods of a group are placed together or none are |
| InferencePool | Gateway API Inference Extension resource grouping model-server pods behind one EPP (inference.networking.k8s.io/v1) |
| KV cache | Cached attention keys and values per token, the main consumer of inference GPU memory |
| LLMInferenceService | KServe CRD for LLM serving with router, prefill/decode and llm-d integration |
| MIG | Multi-Instance GPU, hardware partitioning of one GPU into up to 7 instances |
| MPS | CUDA Multi-Process Service, concurrent kernels from several processes on one GPU |
| P/D disaggregation | Running prefill and decode on separate worker pools, moving KV cache between them (NIXL) |
| ResourceClaim / ResourceSlice | DRA request object / driver-published inventory of devices per node |
| TTFT / ITL | Time to first token / inter-token latency |