Production AI Inference on Kubernetes: The 7 Things It Actually Requires (and Who Should Run It)

Key points:

• Production AI inference on Kubernetes requires seven capabilities: GPU device management, GPU-aware scheduling and sharing, autoscaling on inference metrics instead of CPU, fast model artifact loading, an inference-aware gateway, idle-GPU cost control, and observability on tokens, time-to-first-token (TTFT) and latency percentiles. Skip one and it shows up as a crash loop, a latency spike or a cloud bill.
• The Horizontal Pod Autoscaler on CPU is the wrong signal for LLM serving, because CPU sits near idle while the GPU is saturated. Production setups scale on queue depth (vllm:num_requests_waiting), TTFT or KV cache utilization, wired through KEDA or the Kubernetes custom metrics API.
• A cold GPU pod is not ready in seconds. A 70B-parameter model is roughly 140 GB of weights in BF16, which has to be pulled from object storage and loaded into VRAM, so startup is measured in minutes. That breaks default probes and turns naive scale-to-zero into a latency trap.
• The open-source stack exists but it is assembly work: vLLM or SGLang, KServe or KubeAI, the Gateway API Inference Extension, KEDA, Kueue, Karpenter, the NVIDIA GPU Operator and Dynamic Resource Allocation. That is six to eight projects riding on a Kubernetes minor release supported for roughly 14 months.
• Self-hosted inference beats a per-token API only above a sustained utilization threshold, because you pay the GPU hour whether it serves tokens or not. Reserved instances and committed-use discounts only apply when the capacity sits in your own cloud account, which is why Qovery runs workloads inside your own AWS, GCP, Azure, or Scaleway account - or your existing Kubernetes cluster - without you hiring a platform team to operate it.

Kubernetes was built to schedule stateless pods in seconds. An LLM inference pod takes minutes to start and costs dollars an hour to sit idle. KubeCon NA 2026 giving inference and agents their own track is the community admitting the gap.

I talk to engineering leads every week who got a vLLM container running on a GPU node in an afternoon, then spent the next quarter discovering everything else production needs. This is the checklist I wish they had on day one.

What is the KubeCon NA 2026 AI inference and agentic track, and why did CNCF add it?

KubeCon + CloudNativeCon North America 2026 (November 9-12, 2026, Salt Lake City) added a new CFP track called "AI Inference + Agentic", because inference is now the AI workload Kubernetes operators are asked to run, and Kubernetes' scheduling, autoscaling and routing primitives were designed for stateless services that start in seconds. The track name is listed on the official CFP page, and CNCF announced it with the schedule in August 2026: "CNCF Reveals KubeCon + CloudNativeCon North America 2026 Schedule, Adds New AI Inference + Agentic Track". A co-located Cloud Native AI + Inference Day ran on Monday, November 9.

CNCF has not published a per-topic session count, and I am not going to invent one. The structural signal is enough. AI got a dedicated track in a program of 400+ sessions, and the KubeCon NA 2026 AI track names inference and agents explicitly, with training left to the general program.

That distinction matters operationally. Training is batch, restart-tolerant and throughput-optimized: a job checkpoints, dies, resumes. Inference is an online, latency-sensitive service where an idle H100 bills by the hour regardless of tokens served, at $6.88 per GPU-hour on AWS p5 on-demand in us-east-1 (November 2026).

The demand is already there. The CNCF Annual Cloud Native Survey (January 2026) found that 66% of organizations hosting generative AI models use Kubernetes for some or all of their inference, while 44% do not yet run AI/ML workloads on Kubernetes at all, and only 7% deploy models daily. The top reported adoption challenges were cultural change (47%), lack of training (36%), security (36%) and complexity (34%).

CNCF's own AI working group flagged the gaps two years earlier. Its Cloud Native Artificial Intelligence whitepaper (March 2024) says "GPU allocation with adequate sharing is still evolving", notes that serving brings "load variability and often latency requirements", and calls out "the need to be able to fractionalize expensive accelerators like GPUs."

Two upstream projects prove the gap is real. The Gateway API Inference Extension exists as a Kubernetes SIG project at all, and Dynamic Resource Allocation went GA in Kubernetes v1.34 (August 2025) after years in alpha and beta. Everything below is the operational checklist that track exists to discuss.

What does running production AI inference on Kubernetes actually require?

Production AI inference on Kubernetes requires seven capabilities, each mapped to a different project you install, configure and upgrade: GPU device management, GPU-aware scheduling, GPU sharing, fast model loading, inference-metric autoscaling, an inference-aware gateway, and reliability plus token-level observability. A plain Deployment with an HPA covers roughly one of the seven.

1. GPU device management means running the NVIDIA GPU Operator or device plugin, pinning driver and CUDA versions, and keeping separate node pools per accelerator class (L4, A10G, A100, H100).
2. GPU scheduling on Kubernetes means taints and tolerations on GPU pools, node affinity per accelerator class, Kueue for queuing and quota, and Dynamic Resource Allocation (GA since v1.34, using DeviceClass, ResourceClaim and ResourceSlice in resource.k8s.io/v1) as the successor to the device plugin model.
3. GPU sharing means choosing between MIG, which partitions an A100 or H100 into up to seven instances with dedicated memory and fault isolation, and time-slicing, which has "no memory or fault-isolation between replicas" and is unsafe for latency-sensitive production traffic.
4. Model loading means deciding between weights baked into the image and weights pulled from S3, GCS or Azure Blob at startup, then adding node-local caching, PVCs or preloaded disks so a cold start takes minutes rather than tens of minutes.
5. Autoscaling means scaling on vllm:num_requests_waiting, TTFT or vllm:kv_cache_usage_perc through KEDA or the custom metrics API, because CPU utilization on a GPU pod tells you nothing.
6. Routing and admission means a model-aware gateway (InferencePool, endpoint picker, prefix-cache-aware routing) plus per-tenant token rate limits and request criticality, because round-robin ignores which replica already holds a warm KV cache.
7. Reliability and observability means a startupProbe sized for model load, PodDisruptionBudgets, graceful drain of in-flight SSE streams, and metrics for tokens/sec, TTFT, inter-token latency, VRAM usage and cost per million tokens.

Seven requirements, six to eight projects, and one Kubernetes upgrade cycle underneath all of them. That integration tax is the real cost of self-hosting, and it rarely makes it into the spreadsheet.

Why doesn't a normal Kubernetes Deployment with HPA work for LLM inference?

A standard Deployment plus HPA breaks on LLM inference for five reasons: CPU utilization is a false signal on GPU pods, startup takes minutes because tens of gigabytes of weights must load into VRAM, responses are long-lived token streams, H100 capacity is often unavailable when the autoscaler asks, and an idle GPU replica costs 4x to 34x more per hour than an idle CPU replica.

CPU is a false signal. The tokenizer and HTTP server barely touch the CPU while the GPU sits at 100%, so the HPA sees headroom that does not exist. DCGM and vLLM metrics are invisible to the HPA without a Prometheus adapter or KEDA. LLM inference autoscaling has to start from the inference server's own queue.

Throughput is non-linear. Continuous batching packs requests together, so more requests per second does not mean proportionally more latency until the KV cache fills. The original vLLM announcement (June 2023) measured up to 24x higher throughput than HuggingFace Transformers and up to 3.5x higher than TGI, and the PagedAttention paper reports 2-4x over FasterTransformer and Orca at the same latency. Requests-per-second thresholds cannot model that curve.

Startup takes minutes. Llama 3.1 70B has 70 billion parameters at 2 bytes each in BF16, so about 140 GB has to travel from object storage to disk to VRAM. Even at a sustained 1 GB/s that is more than two minutes before the model serves a single token. Google's Hyperdisk ML docs claim weight loading "up to 11.9X" faster than pulling directly from a model registry, which tells you how slow the default path is. Default probe timings kill the pod long before that, so give the model a startupProbe budget:

containers:
  - name: vllm
    image: vllm/vllm-openai:latest
    ports:
      - containerPort: 8000
    startupProbe:
      httpGet:
        path: /health
        port: 8000
      periodSeconds: 10
      failureThreshold: 90   # up to 15 minutes to load weights
    readinessProbe:
      httpGet:
        path: /health
        port: 8000
      periodSeconds: 5
    livenessProbe:
      httpGet:
        path: /health
        port: 8000
      periodSeconds: 15
      failureThreshold: 4

Streams break rollouts. A response is a server-sent event stream that can run for 30 seconds or more. A rolling update with default terminationGracePeriodSeconds: 30 and no preStop hook cuts live generations mid-sentence. You need a PodDisruptionBudget, a preStop hook that stops accepting new requests, and a grace period longer than your longest generation.

GPU capacity is not on tap. All three clouds sell reservation products for exactly this reason: AWS Capacity Blocks for ML reserve GPU instances up to 8 weeks ahead, GCP's Dynamic Workload Scheduler queues GPU requests, and Azure requires a per-VM-family quota request. Karpenter's wait time for a node stacks on top of model load time.

Scale-to-zero is a trade. A g6.xlarge (1x L4) costs $0.8048/hour on demand, about $587/month if it never sleeps. Keep one warm replica whenever the cost of a multi-minute cold start (abandoned users, missed SLOs) exceeds that number. For internal tools and batch it usually does not, and for customer-facing chat it almost always does.

The fix list, in one line: KEDA on inference metrics, a startupProbe, PDB plus preStop, a pre-warmed node pool or capacity reservation, and a node-local weight cache.

Which open-source and CNCF projects should you use for inference on Kubernetes in 2026?

The practical 2026 stack for AI inference on Kubernetes is vLLM or SGLang as the serving engine, KServe or llm-d as the serving layer, the Gateway API Inference Extension for model-aware routing, KEDA for scaling, Kueue for quota, Karpenter or cluster-autoscaler for GPU nodes, and the NVIDIA GPU Operator plus DRA at the device layer. None of them overlaps enough to let you drop one.

Serving engines. vLLM is the default with the broadest model support. SGLang (Apache-2.0, hosted by LMSYS) is strong on structured output and prefix reuse. TensorRT-LLM squeezes the most out of NVIDIA hardware at the cost of a build step. Ollama is great on a laptop and wrong for multi-tenant production. HuggingFace TGI is now in maintenance mode and its README points users to vLLM and SGLang, so do not start a new deployment on it.

Routing. The Gateway API Inference Extension reached v1.0 in September 2025, promoting InferencePool to v1 and adding an endpoint picker that routes on queue depth and KV cache state instead of round-robin. In v1.6 (August 2026) the project moved InferenceObjective and the full endpoint picker into llm-d and kept a lightweight picker upstream, so check which pieces live where before you pin versions.

Scaling and queuing. KEDA's Prometheus scaler takes a serverAddress, a query and a threshold, which is all you need to scale on vLLM's queue (I covered the tuning in KEDA queue-depth autoscaling for inference pods):

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-llama-8b
spec:
  scaleTargetRef:
    name: vllm-llama-8b
  minReplicaCount: 1
  maxReplicaCount: 8
  cooldownPeriod: 600
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring.svc:9090
        query: sum(vllm:num_requests_waiting{model_name="meta-llama/Llama-3.1-8B-Instruct"})
        threshold: "4"

Kueue handles GPU quota and fair sharing across teams (our Kueue walkthrough), and Karpenter provisions GPU nodes on demand.

Observability. Prometheus, Grafana and the NVIDIA DCGM Exporter cover the hardware. vLLM exposes vllm:time_to_first_token_seconds, vllm:inter_token_latency_seconds and vllm:kv_cache_usage_perc. The OpenTelemetry GenAI semantic conventions define gen_ai.usage.input_tokens, gen_ai.usage.output_tokens and gen_ai.server.time_to_first_token, all still at Development stability as of November 2026, so expect renames.

Six to eight of these, each with its own release cadence and CRDs, ride on a Kubernetes minor release that gets roughly 14 months of patches. Kubernetes is at 1.37 today, which means at least one minor upgrade a year across every one of those projects.

What changes when the workload is agentic rather than a single inference call?

Agentic workloads add four requirements on top of inference serving: long-lived stateful sessions, bursty fan-out of parallel tool calls, strict egress and credential scoping, and per-agent or per-tenant isolation. These stress Kubernetes networking, storage and RBAC far more than GPU scheduling.

Traffic flows outward. Agents call tools, so most of their traffic leaves the pod toward third-party APIs, internal services and databases. Egress NetworkPolicies and narrowly scoped, short-lived credentials become your primary control surface.

Load arrives in bursts. One agent run can fan out into dozens of parallel model and tool calls in a second. Queue-based admission (Kueue, per-tenant token rate limits at the gateway) absorbs that better than replica-count autoscaling, which reacts minutes too late.

State matters. Conversation context, memory and intermediate artifacts need durable storage and session affinity, which conflicts with the interchangeable stateless pods Kubernetes assumes.

Isolation is a security boundary. Namespace-per-tenant, default-deny NetworkPolicies and tight RBAC keep a prompt-injected agent away from production data. OWASP lists this as LLM01:2025 Prompt Injection and LLM06:2025 Excessive Agency in the current Top 10 for LLM Applications.

Evaluation needs throwaway environments. Every branch or experiment needs its own full stack, created and destroyed on demand, otherwise eval runs compete with production for the same GPUs. Every agent action that touches infrastructure should also be scoped, logged and reviewable.

Is it cheaper to run inference on your own Kubernetes cluster than to use a per-token API?

Self-hosted inference on Kubernetes becomes cheaper than a per-token API only above a sustained utilization threshold: a single L4 at $0.8048/hour has to process about 745 tokens per second, around the clock, to match a $0.30-per-million-token API. The break-even depends on your utilization and your committed-use discount, and that discount only exists if the capacity sits in your own cloud account.

The reusable formula for self-hosted LLM cost vs API:

Self-hosted cost per million tokens = (GPU hourly price × hours × replicas) ÷ (tokens actually served ÷ 1,000,000)

Compare that to the API's per-million-token price for an equivalent model class.

The inputs, all checked on November 10, 2026:

• GPU prices (on-demand, Linux): AWS p5.48xlarge (8x H100) is $55.04/hour in us-east-1, or $6.88 per GPU-hour, and g6.xlarge (1x L4) is $0.8048/hour. Azure Standard_ND96isr_H100_v5 (8x H100) is $98.32/hour in East US. GCP publishes A3 (H100) and G2 (L4) rates on its GPU pricing page; check your region, since they vary.
• API prices (input / output per million tokens): Together AI lists Qwen2.5 7B Instruct Turbo at $0.30 / $0.30 and gpt-oss-120B at $0.15 / $0.60. OpenAI lists gpt-4o-mini at $0.15 / $0.60. Anthropic lists Claude Haiku 4.5 at $1 / $5. Cloud managed endpoints such as AWS Bedrock price per model and region, so pull the exact row for the model you would self-host.
• Commitment discounts: AWS Savings Plans go up to 72% (EC2 Instance) or 66% (Compute), GCP committed use discounts up to 55% for most machine series (70% for memory-optimized), and Azure Reserved VM Instances up to 72%. Treat these as ceilings and price the exact GPU SKU and term.

Worked example (assumptions stated, redo it with your numbers):

• Small model on one L4. $0.8048/hour ÷ $0.30 per million tokens = 2.68 million tokens per hour, or about 745 tokens/second sustained, 24/7, to match Together's 7B price. If your traffic only keeps the GPU busy 30% of the time, the card has to hit about 2,480 tokens/second while busy. Benchmark your model with vLLM's own benchmark tool before you believe either number is reachable.
• Larger model on one H100. $6.88/hour ÷ $0.60 per million output tokens = 11.5 million output tokens per hour, or about 3,190 output tokens/second per GPU, to match gpt-oss-120B's output price. With an assumed 40% committed discount (well under the 72% ceiling), the bar drops to about 1,910 tokens/second.

The pattern is the same at every size: utilization decides the winner, and the discount moves the bar. Both levers belong to whoever owns the cloud account, which is why BYOC matters so much for inference. GPU reserved instances, Savings Plans and committed-use discounts only lower your inference cost when the capacity sits in your own AWS, GCP, or Azure account.

Idle GPU time is the dominant waste line. Flexera's 2026 State of the Cloud puts wasted cloud spend at 29%, and the FinOps Foundation's State of FinOps 2026 found 98% of respondents now manage AI spend, up from 31% in 2024. The same waste pattern hurts far more on GPUs: on AWS, an idle g6.xlarge costs 4x an idle m7i.xlarge ($0.2016/hour), and one idle H100 GPU-hour costs 34x. The levers are scale-to-zero where latency allows, auto-stop for non-production environments, bin-packing, MIG partitioning, and spot capacity for eval workloads.

Cost is not the only reason to self-host. Data residency, regulated workloads, fine-tuned or proprietary weights, free model choice and protection from token-price changes all push toward your own cluster even when the per-token math is close.

Who should operate the cluster: a platform team, a managed inference service, or an internal developer platform?

Every requirement in this article lands on a human, and you have three realistic options: dedicate a platform team to assemble and maintain the stack, buy a managed inference service that runs the workload in someone else's account, or use an internal developer platform like Qovery that operates Kubernetes inside your own cloud account so you keep the bill, the GPU discounts and the data.

If you already have a platform team of five or more engineers, building KServe, the Gateway API Inference Extension, KEDA, Kueue and Karpenter yourself is a reasonable, well-trodden path. You get maximum control and no vendor in the request path.

Without a dedicated DevOps engineer, the assembly and upgrade cost dominates. You will track six project release cycles plus Kubernetes itself, forever. The Stack Overflow 2025 Developer Survey puts the US median for a DevOps engineer at $165,000 and a cloud infrastructure engineer at $189,000. If keeping this stack current eats 30% of one DevOps engineer (my assumption, adjust it), that is about $49,500 a year before you serve a token.

This is where Qovery fits. Qovery deploys and operates workloads inside your own AWS, GCP, Azure, or Scaleway account - or your existing Kubernetes cluster - so the cloud bill and every GPU reserved-instance or committed-use discount stays in your name. The capabilities that matter here: git-push deployments, preview and ephemeral environments per pull request for model and agent evaluation, auto-stop for non-production GPU environments, managed cluster upgrades, per-environment RBAC, and databases backed by managed cloud services.

To be precise about the boundary, Qovery is the platform and operations layer. You still choose the serving engine (vLLM, SGLang, KServe, Ray Serve) and the model. Qovery does not ship a model server, token router, inference gateway or GPU autoscaler of its own; you bring KEDA, vLLM and the rest, and Qovery runs the cluster and environments underneath them.

The alternatives are good at different things. Modal and Baseten are excellent when you want someone else to own the GPUs and you accept the workload running outside your account. Anyscale is strong for Ray-based pipelines. Bedrock and Vertex AI are the right answer when you just want an API. Red Hat OpenShift AI fits organizations already standardized on OpenShift. Northflank and Porter compete directly on the run-in-your-own-account angle, so compare them with Qovery on your workload.

The decision rule I give founders: use a per-token API if usage is spiky and the data is unregulated, a managed GPU platform if you have no cloud account preference, and your own cluster plus an internal developer platform if the data or the discounts have to stay with you.

How do I deploy and operate this with Qovery from Claude Code or Cursor?

Install the Qovery Agent Skills once with curl -fsSL https://skill.qovery.com/install.sh | bash, and Claude Code, Cursor, Codex, OpenCode or Gemini CLI can deploy and operate services in your own cloud account from plain-language instructions. The Qovery MCP Server operates what already exists and is read-only unless you explicitly enable writes.

Install globally, or scope the skills to a single project:

curl -fsSL https://skill.qovery.com/install.sh | bash
curl -fsSL https://skill.qovery.com/install.sh | bash -s -- --project

Add the Qovery MCP Server to Claude Code (OAuth, read-only by default):

claude mcp add --transport http qovery https://mcp.qovery.com/mcp --callback-port 4242

Enable write access only when you want it. This also requires turning on write access in the Qovery Console:

claude mcp add --transport http qovery "https://mcp.qovery.com/mcp?read_write=true" --callback-port 4242

The skills include qovery-onboard, qovery-deploy, qovery-troubleshoot, qovery-optimize, qovery-speedup, qovery-preview and qovery-terraform. Then ask in plain language: "deploy this vLLM service to my GCP account with Qovery and give me a preview environment per pull request." The agent shows you the plan and asks you to approve each step, and nothing is written to your cloud account without that approval.

Start with the AI agent quickstart or the Claude Code integration.

Frequently asked questions

What is the KubeCon NA 2026 AI inference and agentic track about?

"AI Inference + Agentic" is a new CFP track at KubeCon + CloudNativeCon North America 2026, held November 9-12, 2026 in Salt Lake City. It covers running model serving and AI agents on Kubernetes: GPU sharing, model-aware routing, token-based autoscaling and multi-tenant isolation. CNCF announced it with the schedule in August 2026, alongside a co-located Cloud Native AI + Inference Day on November 9.

What does running production AI inference on Kubernetes actually require?

Production AI inference on Kubernetes requires seven capabilities: GPU device management, GPU-aware scheduling, GPU sharing (MIG or time-slicing), fast model loading, autoscaling on inference metrics, an inference-aware gateway, and reliability plus token-level observability. In practice that means six to eight projects, such as vLLM, KServe, KEDA, Kueue, Karpenter, the Gateway API Inference Extension and the NVIDIA GPU Operator.

Why doesn't the Horizontal Pod Autoscaler work well for LLM inference?

The HPA scales on CPU or memory by default, and an LLM pod's CPU stays near idle while its GPU is saturated. The HPA sees headroom that does not exist and never adds replicas while latency climbs. Continuous batching also makes requests-per-second a poor proxy for load.

What metric should I scale LLM inference pods on instead of CPU?

Scale vLLM pods on vllm:num_requests_waiting, the number of queued requests, using KEDA's Prometheus scaler. Time to first token (vllm:time_to_first_token_seconds) and KV cache utilization (vllm:kv_cache_usage_perc) are good secondary signals. Set the threshold to the queue depth one replica can drain within your latency SLO.

How long does an LLM pod take to cold start on Kubernetes?

An LLM pod usually takes several minutes to cold start. A 70B model is about 140 GB in BF16, so even at a sustained 1 GB/s the weights alone take more than two minutes to load, plus GPU node provisioning if no node is free. Node-local caches, preloaded disks and a startupProbe with a long failureThreshold keep that from breaking deploys.

Do I need KServe, or is a plain Deployment with vLLM enough?

A plain Deployment running vLLM is enough for one model with steady traffic, provided you add a startupProbe, a PodDisruptionBudget and KEDA scaling yourself. KServe (CNCF Incubating since September 2025) or llm-d becomes worth it when you serve several models, need standardized storage initialization, or want model-aware routing across replicas.

Is it cheaper to run inference on my own Kubernetes cluster than to use a per-token API?

Self-hosting is cheaper only above a sustained utilization threshold. One AWS g6.xlarge L4 at $0.8048/hour must process about 745 tokens per second, 24/7, to match a $0.30-per-million-token API (November 2026 prices). Committed-use discounts of up to 72% lower that bar, but only when the GPUs sit in your own cloud account.

Can I run AI inference and agent workloads on Kubernetes without a dedicated DevOps engineer?

Yes, if something else operates the cluster. Managed APIs and GPU platforms like Modal or Baseten remove the cluster entirely but run outside your account. Qovery operates Kubernetes inside your own AWS, GCP, Azure, or Scaleway account - or your existing cluster - with managed upgrades and git-push deployments, while you choose the serving engine and model.

Inference is where Kubernetes earns or loses its place in the AI stack, and the KubeCon track is the community agreeing the work is worth doing. Run it in your own account, measure tokens instead of CPU, and keep the GPU discounts in your name.