AI Platform Engineering — Architecture¶

GPU Discovery in Kubernetes¶

Kubernetes does not natively understand GPU hardware. A multi-layer discovery path makes GPUs visible as schedulable resources:

GPU Hardware
      |
NVIDIA Driver (OS-level)
      |
Device Plugin (kubelet integration)
      |
Kubernetes Node (resource advertised)
      |
Pod Requests GPU

Step 1: Physical GPU + Driver¶

The physical GPU is attached to a worker node. The operating system exposes it through vendor-specific drivers. For NVIDIA GPUs, the NVIDIA driver must be installed on the node before the OS can interact with the hardware.

Step 2: Kubernetes Awareness Gap¶

Even with the driver installed, Kubernetes remains unaware of the GPU. Kubernetes natively understands three resource types:

CPU (cpu)
Memory (memory)
Ephemeral storage (ephemeral-storage)

GPUs must be explicitly registered through the Device Plugin API.

Step 3: Device Plugin Registration¶

A Device Plugin is a Kubernetes extension that advertises specialized hardware to the kubelet. The NVIDIA Device Plugin runs as a DaemonSet on each GPU node and reports available GPUs.

Once registered, Kubernetes sees the resource:

# Node capacity after device plugin registration
nvidia.com/gpu: 4

Key Insight

Kubernetes does not schedule GPUs because it understands GPUs. It schedules GPUs because a Device Plugin exposes them as generic extended resources. The same mechanism works for TPUs, FPGAs, SmartNICs, and any other accelerator with a Device Plugin implementation.

Step 4: Pod GPU Requests¶

After registration, workloads request GPUs identically to CPU and memory:

apiVersion: v1
kind: Pod
metadata:
  name: gpu-inference
spec:
  containers:
    - name: model-server
      image: vllm/vllm-openai:latest
      resources:
        limits:
          nvidia.com/gpu: 1

The scheduler matches resource requests with available resources — it has no knowledge of whether the workload is running an LLM, image generator, or training job.

Device Plugin Architecture¶

sequenceDiagram
    participant GPU as GPU Hardware
    participant Driver as NVIDIA Driver
    participant DP as NVIDIA Device Plugin
    participant Kubelet as kubelet
    participant Scheduler as kube-scheduler
    participant Pod as Pod

    GPU->>Driver: Hardware attached
    Driver->>DP: GPU devices available
    DP->>Kubelet: Register via gRPC<br/>ListAndWatch(nvidia.com/gpu: 4)
    Kubelet->>Scheduler: Node capacity updated
    Pod->>Scheduler: Request nvidia.com/gpu: 1
    Scheduler->>Kubelet: Schedule Pod on GPU node
    Kubelet->>DP: Allocate(deviceID)
    DP->>Pod: Mount GPU device + env vars

The Device Plugin communicates with the kubelet via a gRPC interface at /var/lib/kubelet/device-plugins/. The plugin must handle kubelet restarts by monitoring socket deletion and re-registering. The Device Plugin API supports:

ListAndWatch — advertises available devices and reports health changes
Allocate — provisions device access for containers (device nodes, environment variables, mounts, CDI device names)
Health monitoring — marks devices as unhealthy when failures are detected. This reduces the node's allocatable count

Since Kubernetes v1.36 (beta), allocatedResourcesStatus in pod status reports per-device health information including error details and failure reasons.

NVIDIA GPU Operator¶

The NVIDIA GPU Operator automates the full GPU lifecycle on Kubernetes nodes:

Component	Purpose
NVIDIA Driver	Kernel-level GPU access
NVIDIA Device Plugin	GPU advertisement to Kubernetes
NVIDIA Container Toolkit	Container runtime GPU integration
DCGM Exporter	GPU metrics for Prometheus
Node Feature Discovery	Labels nodes with GPU properties
MIG Manager	Multi-Instance GPU partition management
GPU Feature Discovery	Exposes GPU model, memory, driver version as node labels

This eliminates manual driver installation, device plugin deployment, and monitoring setup across the cluster.

GPU Utilization and Resource Fragmentation¶

The Core Problem¶

Standard Kubernetes GPU allocation is binary — a pod requests one or more whole GPUs, and each GPU is allocated exclusively:

Pod A → 1 GPU Requested → 1 GPU Allocated (80 GB)
Actual usage: 10 GB memory, 20% compute
Waste: 70 GB memory, 80% compute

With CPUs, Kubernetes efficiently bin-packs multiple pods onto a single node. GPUs traditionally do not support this — one pod per GPU, regardless of actual utilization.

Cost Impact¶

Consider a cluster with 8x NVIDIA A100 GPUs where every workload uses only 25% of each GPU:

Effective utilization: 2 GPUs worth of useful work
Paid for: 8 GPUs
Waste: 75% of GPU investment

At $30,000+ per A100 GPU, this translates to ~$180,000 in wasted capacity.

Three primary techniques address GPU underutilization:

Time-Slicing¶

Multiple workloads take turns using the same GPU, analogous to CPU time-sharing. NVIDIA's Run:ai implementation provides two modes:

Mode	Behavior	K8s Mapping
Strict	Each workload gets exactly its requested GPU compute fraction	`gpu-compute-request = gpu-compute-limit = gpu-fraction`
Fair	Each workload gets at least its fraction, plus unused slices from idle workloads	`gpu-compute-request = gpu-fraction`, `gpu-compute-limit = 1.0`

Time-slicing operates on a plan/lease cycle. Default configuration:

Lease time: 250ms (exclusive GPU access per workload)
Granularity: 5% precision
Plan (cycle) time: 250ms / 0.05 = 5000ms (5 seconds)

A workload requesting gpu-fraction=0.5 gets 2.5s of runtime per 5s cycle.

Trade-offs

Decreasing lease time makes time-slicing less accurate. Increasing lease time improves accuracy but reduces workload responsiveness. Context switching between workloads adds overhead.

Multi-Instance GPU (MIG)¶

Available on NVIDIA Ampere+ GPUs (A100, H100). MIG partitions a single GPU into up to 7 isolated GPU Instances, each with dedicated:

Streaming Multiprocessors (SMs)
GPU engines (copy engines, decoders)
L2 cache banks
Memory controllers
DRAM address busses

Example partitioning of an 80 GB A100:

Full A100 (80 GB)
├── MIG Instance 1: 10 GB (1g.10gb)
├── MIG Instance 2: 10 GB (1g.10gb)
├── MIG Instance 3: 20 GB (2g.20gb)
└── MIG Instance 4: 40 GB (4g.40gb)

Each instance provides hardware-level isolation — one workload cannot impact the L2 cache or DRAM bandwidth of another. This makes MIG suitable for multi-tenant environments where QoS guarantees are required.

MIG supports:

Bare-metal and containers
GPU passthrough virtualization
vGPU on supported hypervisors

Continuous Batching (Inference)¶

Instead of processing inference requests one-by-one, the serving engine combines multiple requests into a single GPU execution. vLLM's continuous batching dynamically adds new requests as older ones complete. The GPU then stays busy continuously instead of waiting for a full batch to form.

AI Workload Scheduling¶

Why Standard Kubernetes Scheduling Falls Short¶

Traditional applications are loosely coupled — components can start independently and tolerate staggered scheduling. AI training jobs have fundamentally different requirements.

Gang Scheduling¶

Distributed training jobs require all resources allocated simultaneously:

Training Job (requires 8 GPUs)
├── Worker 0: GPU 0
├── Worker 1: GPU 1
├── Worker 2: GPU 2
├── Worker 3: GPU 3
├── Worker 4: GPU 4
├── Worker 5: GPU 5
├── Worker 6: GPU 6
└── Worker 7: GPU 7

If only 6 GPUs are available, the job cannot start. Partial allocation wastes resources — workers wait indefinitely for the remaining GPUs, blocking other jobs.

Gang scheduling rule: Either schedule all required resources together, or schedule none of them.

Topology Awareness¶

GPU placement affects training performance significantly. Same-node GPUs communicate via high-speed interconnects (NVLink at 900 GB/s on H100), while cross-node GPUs use network fabric (InfiniBand at ~400 Gb/s).

graph LR
    subgraph "Node A (Fast: NVLink)"
        GPU0["GPU 0"]
        GPU1["GPU 1"]
        GPU2["GPU 2"]
        GPU3["GPU 3"]
        GPU0 <--> GPU1
        GPU1 <--> GPU2
        GPU2 <--> GPU3
    end

    subgraph "Node B (Fast: NVLink)"
        GPU4["GPU 4"]
        GPU5["GPU 5"]
        GPU6["GPU 6"]
        GPU7["GPU 7"]
        GPU4 <--> GPU5
        GPU5 <--> GPU6
        GPU6 <--> GPU7
    end

    GPU3 <-.->|"Slower: Network Fabric"| GPU4

A topology-aware scheduler prefers placing all GPUs on the same node when possible. If that is not possible, it falls back to nodes with the best inter-node connectivity.

Scheduling Tools Comparison¶

Tool	Type	Key Capabilities
Volcano	Kubernetes-native batch scheduler	Gang scheduling, queue management, fair-share, priority-based preemption
Kueue	Kubernetes SIG job queueing	Admission control, resource quotas, job queuing, cluster queue management
Kubeflow Training Operator	Distributed training CRD	PyTorchJob, TFJob, XGBoostJob — works with Volcano for gang scheduling
NVIDIA GPU Operator	GPU lifecycle manager	Driver management, device plugins, DCGM metrics, MIG management

Ray — Distributed Compute Framework¶

The Two-Layer Model¶

graph TB
    subgraph "Application Layer"
        User["User / Application"]
        RayDriver["Ray Driver"]
    end

    subgraph "Ray Layer (Computation Scheduling)"
        RayHead["Ray Head Node<br/>(GCS, Autoscaler, Dashboard)"]
        RayWorker1["Ray Worker 1"]
        RayWorker2["Ray Worker 2"]
        RayWorker3["Ray Worker 3"]
    end

    subgraph "Kubernetes Layer (Infrastructure Scheduling)"
        K8s["kube-scheduler"]
        Pod1["Pod (Head)"]
        Pod2["Pod (Worker)"]
        Pod3["Pod (Worker)"]
        Pod4["Pod (Worker)"]
    end

    User --> RayDriver
    RayDriver --> RayHead
    RayHead --> RayWorker1
    RayHead --> RayWorker2
    RayHead --> RayWorker3

    K8s --> Pod1
    K8s --> Pod2
    K8s --> Pod3
    K8s --> Pod4

    Pod1 -.- RayHead
    Pod2 -.- RayWorker1
    Pod3 -.- RayWorker2
    Pod4 -.- RayWorker3

Kubernetes schedules infrastructure (which node should this pod run on?). Ray schedules computation (which worker executes which task? how are results collected?).

Ray Architecture¶

Component	Role
Head Node	Runs GCS (Global Control Service), autoscaler, Ray dashboard. Also schedules tasks like worker nodes.
Worker Nodes	Execute Ray tasks and actors. Participate in distributed object storage.
Autoscaler	Scales worker nodes based on task/actor resource requests (not CPU/memory metrics).
GCS	Central metadata store for cluster state, actor locations, and resource availability.

Tasks vs Actors¶

Dimension	Tasks	Actors
State	Stateless	Stateful
Lifecycle	Run once, return result	Long-lived, handle multiple requests
Use case	Data processing, hyperparameter search	Model serving, stateful computation
Invocation	`function.remote(args)`	`actor.method.remote(args)`

Tasks enable embarrassingly parallel workloads (data processing, hyperparameter tuning). Actors enable stateful services (model serving, game environments, RL training).

Ray Libraries¶

Library	Purpose
Ray Train	Distributed training (PyTorch, TensorFlow, XGBoost)
Ray Tune	Hyperparameter optimization
Ray Serve	Model serving and composition
Ray Data	Distributed data processing
Ray RLlib	Reinforcement learning

Ray on Kubernetes (KubeRay)¶

KubeRay provides Kubernetes CRDs for managing Ray clusters:

RayCluster — manages head and worker pods
RayJob — submits jobs to a Ray cluster
RayService — manages Ray Serve deployments with zero-downtime upgrades

The autoscaler in KubeRay v2 runs as a sidecar container in the head pod. It scales worker pods based on pending Ray task/actor resource demands.

vLLM — Inference Engine Architecture¶

Why Naive Model Serving Fails at Scale¶

A naive inference server processes requests sequentially or in static batches:

Request 1 → GPU → Response 1
Request 2 → GPU → Response 2  (waits for Request 1)
Request 3 → GPU → Response 3  (waits for Request 2)

With 100 concurrent users, GPU utilization stays low because the GPU cannot exploit its parallel architecture. This is the GPU utilization problem from Day 3 applied to inference serving.

PagedAttention¶

The key innovation in vLLM. Traditional serving systems allocate GPU memory for the KV cache in large, contiguous chunks. For variable-length sequences, this causes:

Internal fragmentation — allocated blocks larger than needed
External fragmentation — free memory scattered in unusable small chunks
Reservation waste — memory reserved for maximum sequence length even for short sequences

PagedAttention treats the KV cache like virtual memory — memory is managed in fixed-size blocks (pages) that need not be contiguous:

Traditional KV Cache:
[████████████░░░░░░░░] Request 1 (wasted space)
[████████░░░░░░░░░░░░] Request 2 (wasted space)
[░░░░░░░░░░░░░░░░░░░░] Free (fragmented)

PagedAttention:
[████][████][████][██] Request 1 (pages, no waste)
[████][████][██]       Request 2 (pages, no waste)
[████][████]           Free (reusable pages)

Results:

Near-zero memory waste
2-4x more concurrent requests with the same GPU
Dynamic memory allocation as sequences grow

Continuous Batching¶

Traditional batching waits for a full batch before processing:

Traditional:    Wait → Process Batch → Wait → Process Batch
Continuous:     Process ─── Process ─── Process ─── Process
                (new requests added as old ones complete)

vLLM's continuous batching adds new requests to the running batch as existing requests finish generating tokens. The GPU stays busy continuously. This dramatically improves throughput.

vLLM in the Platform Stack¶

Users
  |
vLLM (inference optimization: PagedAttention + continuous batching)
  |
Model Weights (loaded into GPU memory)
  |
GPU (compute)

In a Kubernetes deployment:

Users
  |
Load Balancer / Gateway
  |
vLLM Pods (KServe InferenceService or Ray Serve)
  |
Kubernetes (scheduling, scaling, health checks)
  |
GPU Nodes (NVIDIA Device Plugin, GPU Operator)

vLLM exposes an OpenAI-compatible API. It is a drop-in replacement for existing applications that use the OpenAI API format.

Full Platform Architecture Diagram¶

graph TB
    subgraph "Layer 1: AI Applications"
        App1["Virtual Assistants"]
        App2["Recommendation Systems"]
        App3["Fraud Detection"]
        App4["Content Generation"]
    end

    subgraph "Layer 2: AI Platform Services"
        subgraph "Data Pipeline"
            DP["Data Processing"]
            FE["Feature Engineering"]
        end
        subgraph "Model Lifecycle"
            MT["Model Training<br/>(Kubeflow, Ray Train)"]
            MR["Model Registry<br/>(MLflow)"]
        end
        subgraph "Serving & Monitoring"
            MS["Model Serving<br/>(KServe, vLLM, Ray Serve)"]
            MON["Monitoring<br/>(Prometheus, DCGM)"]
        end
    end

    subgraph "Layer 3: Infrastructure"
        subgraph "Orchestration"
            K8S["Kubernetes"]
            SCHED["Schedulers<br/>(Volcano, Kueue)"]
            RAY["Ray Cluster"]
        end
        subgraph "Compute & Storage"
            GPU["GPUs<br/>(A100, H100)"]
            CPU["CPUs"]
            STORE["Object Storage<br/>(S3, MinIO)"]
        end
        subgraph "GPU Management"
            GPUOP["GPU Operator"]
            DEVPLUGIN["Device Plugin"]
            MIG["MIG Manager"]
        end
    end

    App1 & App2 & App3 & App4 --> MS
    DP --> FE --> MT --> MR --> MS
    MS --> MON
    MS --> RAY
    MT --> RAY
    RAY --> K8S
    SCHED --> K8S
    K8S --> GPU & CPU & STORE
    GPUOP --> DEVPLUGIN --> GPU
    GPUOP --> MIG --> GPU

Benchmarks and Scale Considerations¶

vLLM Performance Characteristics¶

Based on the PagedAttention paper (arXiv:2309.06180):

PagedAttention achieves near-zero KV cache waste vs 60-80% waste in naive allocators
Continuous batching can improve throughput by 2-4x over static batching
Memory savings translate directly to higher concurrent request capacity

GPU Memory Budget (Inference)¶

For an LLM with P parameters at B bytes per parameter:

Model weights:  P × B bytes
KV cache:       Variable (grows with context length × batch size)
Overhead:       ~10-20% for framework, CUDA context

Example: Llama 2 70B at FP16:

Model weights: 70B × 2 bytes = 140 GB
Minimum: 2x A100 80GB (tensor parallelism)
With KV cache headroom: 4x A100 80GB for production batch sizes

GPU Interconnect Bandwidth¶

Interconnect	Bandwidth	Use Case
NVLink (H100)	900 GB/s	Intra-node GPU communication
NVLink (A100)	600 GB/s	Intra-node GPU communication
InfiniBand HDR	200 Gb/s (25 GB/s)	Inter-node communication
InfiniBand NDR	400 Gb/s (50 GB/s)	Inter-node communication
Ethernet (RoCE)	100-400 Gb/s	Cost-effective inter-node

Topology-aware scheduling becomes critical when inter-node bandwidth is 10-30x lower than intra-node NVLink bandwidth.

Security¶

Threat Model Overview¶

AI platform infrastructure introduces security concerns beyond traditional Kubernetes workloads due to the high value of GPU resources, model weights, and training data.

graph TB
    subgraph "Attack Surface"
        A1["Model Weight Theft"]
        A2["GPU Resource Hijacking<br/>(Cryptomining)"]
        A3["Training Data Exfiltration"]
        A4["Inference API Abuse"]
        A5["Supply Chain<br/>(Poisoned Models)"]
        A6["Multi-Tenant Isolation<br/>Bypass"]
    end

    subgraph "Assets at Risk"
        M["Model Weights<br/>(Proprietary IP)"]
        G["GPU Compute<br/>(High $$$ value)"]
        D["Training Data<br/>(PII, proprietary)"]
        I["Inference Endpoints<br/>(Production services)"]
    end

    A1 --> M
    A2 --> G
    A3 --> D
    A4 --> I
    A5 --> M
    A6 --> G & M & D

Authentication and Authorization¶

Kubernetes RBAC for GPU Resources¶

Control who can schedule GPU workloads to prevent unauthorized GPU consumption:

# Role restricting GPU pod creation
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: gpu-user
  namespace: ml-team
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create", "get", "list", "delete"]
  - apiGroups: ["batch"]
    resources: ["jobs"]
    verbs: ["create", "get", "list", "delete"]
---
# ResourceQuota limiting GPU allocation per namespace
apiVersion: v1
kind: ResourceQuota
metadata:
  name: gpu-quota
  namespace: ml-team
spec:
  hard:
    requests.nvidia.com/gpu: "4"
    limits.nvidia.com/gpu: "4"

vLLM API Authentication¶

vLLM's OpenAI-compatible API does not provide built-in authentication. Production deployments must use:

API Gateway (Kong, Envoy, NGINX) for token validation, rate limiting, and request logging
Kubernetes NetworkPolicies to restrict which pods can get to vLLM endpoints
Service mesh (Istio, Linkerd) for mTLS between services

# NetworkPolicy: only allow inference gateway to reach vLLM
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: vllm-access
  namespace: inference
spec:
  podSelector:
    matchLabels:
      app: vllm
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: inference-gateway
      ports:
        - port: 8000

Ray Cluster Security¶

Ray clusters require attention to several security surfaces:

Component	Risk	Mitigation
Ray Dashboard (8265)	Unauthenticated access to cluster management	NetworkPolicy, ingress with auth
GCS Port (6379)	Cluster control plane access	Pod-to-pod mTLS, no external exposure
Object Store	In-memory data accessible between workers	Namespace isolation, trusted workloads only
Ray Client (10001)	Remote code execution	TLS, authentication proxy

# NetworkPolicy: isolate Ray cluster
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: ray-cluster-isolation
  namespace: ray
spec:
  podSelector:
    matchLabels:
      ray.io/cluster: gpu-cluster
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              ray.io/cluster: gpu-cluster
    - from:
        - podSelector:
            matchLabels:
              app: ray-job-submitter
      ports:
        - port: 8265
  egress:
    - to:
        - podSelector:
            matchLabels:
              ray.io/cluster: gpu-cluster
    - to: []  # Allow outbound for model downloads

Multi-Tenant GPU Isolation¶

MIG vs Time-Slicing Security Comparison¶

Dimension	MIG	Time-Slicing
Memory isolation	Hardware-enforced — separate memory controllers and DRAM paths	Software-enforced — relies on CUDA context isolation
Compute isolation	Dedicated SMs, L2 cache banks, GPU engines	Shared GPU with time-based access rotation
Side-channel risk	Low — physically separate paths	Higher — shared cache and memory bus
QoS guarantees	Predictable — dedicated resources	Variable — depends on co-tenant behavior
Suitable for	Multi-tenant production, compliance environments	Development, trusted single-tenant clusters

Multi-Tenant Isolation

Time-slicing does NOT provide hardware-level isolation. Workloads from different tenants sharing a GPU via time-slicing can potentially observe side-channel information through shared L2 cache timing. For regulated or untrusted multi-tenant environments, MIG or dedicated GPU allocation is required.

Namespace-Based GPU Isolation¶

# Dedicated GPU nodes per team using taints and tolerations
# Taint GPU nodes for specific teams
# kubectl taint nodes gpu-node-1 team=ml-production:NoSchedule

apiVersion: v1
kind: Pod
metadata:
  name: production-inference
  namespace: ml-production
spec:
  tolerations:
    - key: "team"
      operator: "Equal"
      value: "ml-production"
      effect: "NoSchedule"
  nodeSelector:
    gpu-pool: production
  containers:
    - name: vllm
      image: vllm/vllm-openai:latest
      resources:
        limits:
          nvidia.com/gpu: 1

Model Weight Protection¶

Securing Model Storage¶

Model weights represent significant IP investment. Protection strategies:

Encrypted storage at rest — Use encrypted PersistentVolumes or encrypted object storage (S3 SSE, MinIO encryption)
Access control — Restrict model registry access (MLflow, HuggingFace Hub) with per-team credentials
Network segmentation — Model downloads should traverse private networks, not public internet
Signed models — Verify model integrity using checksums or signatures before loading

# Secret for model registry credentials
apiVersion: v1
kind: Secret
metadata:
  name: hf-token
  namespace: inference
type: Opaque
data:
  token: <base64-encoded-huggingface-token>
---
# Pod with model credentials mounted as env var
# (avoid mounting as files in shared volumes)
apiVersion: v1
kind: Pod
spec:
  containers:
    - name: vllm
      env:
        - name: HUGGING_FACE_HUB_TOKEN
          valueFrom:
            secretKeyRef:
              name: hf-token
              key: token
        - name: HF_HOME
          value: "/models/cache"
      volumeMounts:
        - name: model-cache
          mountPath: /models/cache
  volumes:
    - name: model-cache
      persistentVolumeClaim:
        claimName: model-cache-pvc

GPU Cryptomining Prevention¶

GPU nodes are high-value targets for cryptomining. Detection and prevention:

# Monitor for unexpected GPU utilization
# Alert on sustained high GPU utilization from non-whitelisted pods
# PromQL alert rule:
DCGM_FI_DEV_GPU_UTIL > 90
  and on (pod)
  kube_pod_labels{label_workload_type!="training", label_workload_type!="inference"}

Prevention measures:

Pod Security Standards — Restrict privileged containers on GPU nodes
Image allowlisting — Only permit approved container images on GPU nodes
Admission webhooks — Validate that GPU-requesting pods match approved workload patterns
Resource quotas — Limit GPU allocation per namespace

Encryption¶

Data in Transit¶

Path	Protocol	Configuration
Client → Inference API	HTTPS/TLS	API Gateway with TLS termination
Pod → Pod (inference)	mTLS	Service mesh (Istio/Linkerd)
Ray inter-node	TLS	Ray TLS configuration
GPU node → Storage	HTTPS	Encrypted object store endpoints
NVLink (intra-node)	N/A	Physical hardware path, no encryption needed
RDMA/InfiniBand	N/A	Typically trusted fabric, IPsec optional

Data at Rest¶

Component	Encryption Method
Model weights on PV	Encrypted PersistentVolumes (StorageClass encryption)
Training data	Encrypted object storage (SSE-S3, SSE-KMS)
KV Cache (GPU memory)	Not encrypted (volatile GPU memory)
Model registry	Application-level encryption + encrypted backend storage
Logs/metrics	Encrypted storage backend

Compliance Considerations¶

GPU Workload Auditing¶

# Kubernetes audit policy for GPU resource events
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  - level: Metadata
    resources:
      - group: ""
        resources: ["pods"]
    verbs: ["create", "delete"]
    # Log who creates/deletes GPU pods
  - level: RequestResponse
    resources:
      - group: "batch.volcano.sh"
        resources: ["jobs"]
    # Full audit for Volcano jobs