llm-d hit 1.0 and is a CNCF Sandbox project: what Kubernetes-native disaggregated inference means if you have no ML infra team

Key points:

• llm-d is a Kubernetes-native serving layer that runs on top of vLLM and SGLang. It adds an LLM-aware router (prefix-cache and load-aware scheduling), prefill/decode disaggregation, KV-cache offloading and autoscaling. llm-d v1.0 shipped on October 20, 2026, and CNCF accepted it into the Sandbox on March 12, 2026. It is Apache 2.0.
• 1.0 is about Day 2, and Sandbox still means early. The release brings a v1 configuration API, multi-tenant flow control, zero-downtime rollouts and session-aware routing for agents. CNCF itself describes Sandbox projects as "not yet widely tested in production".
• Disaggregation is the part most small teams do not need yet. llm-d's own docs say P/D pays off for medium-to-large models with long inputs (around 10k input tokens per 1k output), and it needs RDMA (InfiniBand, RoCE or EFA) between pods. The smallest P/D topology uses 5 GPUs, and the wide expert-parallel guide uses 32 H200s.
• Start with the Optimized Baseline path. It is plain vLLM replicas behind the llm-d router with prefix-cache-aware routing, no RDMA, and a reference setup of 4 H100s (2 replicas at TP=2). On OVHcloud's published prices that is about $8,700 a month; a P/D-ready 8x H200 node on AWS is about $46,200.
• llm-d does not remove the infrastructure work. You still own GPU quotas and node pools, NVIDIA drivers compatible with its CUDA 12.9.1 images, a Gateway API provider, the LeaderWorkerSet controller, weight caching, monitoring and upgrades. That is the part a platform like Qovery runs for you in your own cloud account, while llm-d and vLLM serve the model.

Every platform lead I talk to this month asks me the same thing after KubeCon: "llm-d is 1.0 and in the CNCF, so can we finally self-host a big open-weight model without hiring an ML infrastructure team?" The honest answer is "partly." llm-d removes the hardest software problem in distributed inference. It does not remove the GPUs, the network fabric, or the cluster underneath.

I read the 1.0 release, the guides in the llm-d repository and their hardware tables, then priced each topology on public cloud price lists. Here is what Kubernetes-native disaggregated inference means in practice, which path a small team should start on, and where the bill lands.

What is llm-d, and what actually shipped in 1.0?

llm-d is an open-source, Kubernetes-native distributed inference framework that sits between your clients and vLLM or SGLang model servers. CNCF describes it as a framework "built on vLLM and the Kubernetes Gateway API Inference Extension, providing intelligent inference scheduling, prefix-cache-aware routing, prefill/decode disaggregation, hierarchical KV offloading, and traffic- and hardware-aware autoscaling across NVIDIA, AMD, Intel, and Google TPU accelerators" (CNCF project page).

The "1.0" and the "CNCF" headlines are two different events:

• May 20, 2025: llm-d is announced by engineers from Red Hat, Google and IBM, combining vLLM, Kubernetes and the Inference Gateway.
• March 12, 2026: CNCF accepts llm-d at the Sandbox level.
• October 20, 2026: llm-d v1.0 ships at PyTorch Conference North America.
• November 10, 2026: CNCF's KubeCon announcement lists llm-d among the projects that "joined CNCF and reached its 1.0 release", in a portfolio that now counts 231 projects.

Most of what 1.0 adds is operational. The release post groups it into three themes:

• Multi-tenant quality of service: priority- and fairness-aware flow control with a queue per flow, and a probabilistic admission controller that sheds load gracefully.
• Zero-downtime operations: router high availability with a hot standby, and a DisaggregatedSet that upgrades prefill and decode pools as one versioned unit.
• Agent-aware serving: session-aware routing that pins a conversation to the replica holding its KV cache, turn-priority scheduling for agent loops, and native state for the OpenAI Responses API (previous_response_id).

The configuration API also graduates to v1, which is the change I care about most as an operator: a stable, versioned surface you can pin. The project counts more than 500 contributors across 25-plus organizations and is licensed Apache 2.0.

The numbers in the release post come from adopters, on their hardware and their workloads, so read them as ceilings:

Source for every row: the llm-d v1.0 release post. The biggest wins come from routing and caching rather than from disaggregation.

What does "disaggregated inference" actually mean?

Disaggregated inference splits one LLM request into its two phases and runs them on different pods: a prefill pool that processes the prompt and builds the KV cache, and a decode pool that pulls that KV cache over the network and generates the output tokens. llm-d's router schedules each request twice, once onto a prefill pod and once onto a decode pod (P/D guide).

The split exists because the two phases want different hardware behavior:

• Prefill is compute-bound. A 50,000-token coding-agent prompt saturates the GPU's math units for a burst.
• Decode is memory-bandwidth-bound and latency-sensitive. Each output token is a small step that has to come back fast.

When both phases share a GPU, a long prefill stalls every decode stream on that server, and your users see the output "freeze." Separating them removes that interference, lowers inter-token latency, and lets you size each pool independently. llm-d's architecture doc gives a concrete ratio: 8 prefill pods at TP=1 and 2 decode pods at TP=4 (16 GPUs) suit a 5k-input, 250-output workload on gpt-oss-120b.

The catch is the KV transfer. The decode pod has to fetch the prefill pod's KV cache in milliseconds, and llm-d is blunt about it: "Disaggregated Serving requires high performance (RDMA) interconnects between nodes for efficient KV transfer. Without RDMA, NIXL falls back to TCP for transfer which is not efficient and should only be used for testing and development" (llm-d docs).

That one sentence decides which clouds and instance types are realistic. On AWS it means EFA-capable instances: the P/D guide's aws overlay targets p5en.48xlarge nodes with the EFA device plugin, and AWS published its own walkthrough of disaggregated inference powered by llm-d that calls EFA "essential." On GKE it means RoCE through managed DRANET. On a GPU instance with a 25 Gbit/s Ethernet NIC and no RDMA, it means you should not run P/D in production.

Does CNCF Sandbox status mean llm-d is production-ready?

No, and it does not mean the opposite either. CNCF Sandbox is the entry level of the foundation, described as "experimental projects not yet widely tested in production on the bleeding edge of technology". Sandbox status describes governance and how widely a project is adopted, and says little about code quality. For comparison, KServe was accepted at the Incubating level on September 29, 2025, with a first commit in 2019.

The project's own testing tells you more than the badge. llm-d's guides mark each accelerator and engine combination as either "validated" (covered by a nightly end-to-end workflow) or "community" (maintained by the hardware vendor, not covered by nightly E2E). The release matrix runs the core guides nightly on IBM, CoreWeave, GKE, AMD and Intel environments.

My reading for a team without ML infrastructure engineers:

• The "validated" NVIDIA paths are safe to build on. Optimized Baseline, Precise Prefix Cache Routing and P/D on NVIDIA GPUs with vLLM all run in nightly CI.
• Anything marked "community" or "experimental" needs your own testing. Coordinator disaggregation, turn-priority fairness and the stateful Responses API guide are labeled experimental in the guides index.
• Provider docs are uneven. The repository has infrastructure provider docs for GKE, AKS, CoreWeave, DigitalOcean, OpenShift and minikube. EKS is covered by overlays inside individual guides, and Scaleway or OVHcloud use the provider-neutral base overlays. You will be adapting manifests yourself on those.

Which llm-d well-lit path should a small team start with?

Start with the Optimized Baseline path and stop there until your metrics say otherwise. It deploys normal vLLM (or SGLang) replicas behind the llm-d router with prefix-cache affinity and load-aware scoring, needs no RDMA, and its reference setup is Qwen3-32B on H100 80 GB with 2 replicas at TP=2, 4 GPUs in total (Optimized Baseline guide).

llm-d calls its tested recipes "well-lit paths." The guides index lists more than twenty. These are the ones that matter for self-hosting decisions:

Hardware columns come from each guide's support table: Optimized Baseline, Tiered Prefix Cache, P/D, Wide EP.

llm-d's own advice on P/D is unusually candid: "P/D disaggregation is not a target for all workloads." It suggests exploring it for medium-large models, longer input sequences ("10k ISL | 1k OSL, not 200 ISL | 200 OSL") and sparse MoE models that can use wide expert parallelism (architecture doc). Coding agents do produce 10:1 traffic. A team of ten engineers usually does not produce enough of it to keep two specialized pools busy.

How many GPUs, and how much money, before disaggregation pays off?

Disaggregation starts at a full 8-GPU node in practice, so it only makes sense once a single aggregated node is saturated. The Optimized Baseline reference fits in two 2-GPU instances; the smallest P/D topology needs 5 GPUs with RDMA between them, which on AWS means an 8x H200 p5en.48xlarge; the model-specific recipes for frontier MoE models start at 16 H200s.

Here is what each topology costs on public on-demand prices, computed at 730 hours a month:

AWS prices are from the public on-demand price list for US East (N. Virginia) as published on November 14, 2026. Topologies come from the GLM-5.2 and Nemotron 3 Ultra recipes. None of this includes storage, egress, the control plane or the people.

Two things stand out. First, going from "baseline" to "disaggregated" is roughly a 5x jump in monthly spend on the cheapest example. Second, the frontier MoE recipes are datacenter-scale on purpose. They show the ceiling, and nobody should treat them as a starter kit.

The hosted alternative sets the bar. If a provider sells the same open-weight model through an API, every month your GPUs cost $46,206 they need to replace $46,206 of token spend. For most teams under that line, the right move is the Optimized Baseline on a few GPUs for the models you must keep private, and an API for the rest. I covered the break-even math in more detail in production AI inference on Kubernetes.

What do you still have to run yourself if nobody on the team does ML infrastructure?

llm-d removes the scheduling and KV-cache logic you would otherwise write. It does not remove the cluster. Here is the list I would hand to a team before they promise anyone a self-hosted model:

• GPU capacity and quotas. H100 and H200 quota is gated on every hyperscaler, and the AWS P/D overlay requires p5en.48xlarge specifically. Request quota before you design anything.
• Drivers that match the images. llm-d currently ships images on the CUDA 12.9.1 runtime, which supports drivers from 525.60.13 up to but not including 580, and recommends the 575.x branch. A planned move to CUDA 13.0.2 will require 580.65.06 or later (accelerator docs). A node image that auto-upgrades its NVIDIA driver can break your serving pods.
• RDMA plumbing, if you go P/D. EFA device plugin on AWS, DRANET for RoCE on GKE, an rdma/ib device plugin on InfiniBand clouds. Each is its own operator to install and upgrade.
• A Gateway API provider. The router integrates with GKE's gateway, Istio or agentgateway, and the guide warns that with the wrong provider setting "requests fail with 503s" (Optimized Baseline guide).
• The LeaderWorkerSet controller v0.11.1 or newer with the DisaggregatedSet API enabled, for P/D and multi-node pods (P/D guide prerequisites).
• Model weights. A frontier checkpoint is hundreds of gigabytes to over a terabyte. You need a cache on node-local NVMe or a shared volume, or every scale-up is minutes of paid, idle GPU. llm-d's ModelExpress and pod snapshot guides help here, but you still own the storage.
• Observability and upgrades. Prometheus metrics for queue depth, cache hit rate and time-to-first-token, plus a plan for upgrading vLLM, the router and Kubernetes itself without dropping long-running requests.
• Access control. Who can deploy a new model, who can change the router config, and who can read prompts in logs.

Some of these need recent Kubernetes. Accelerators that are allocated through Dynamic Resource Allocation require "Kubernetes 1.34 with the resource.k8s.io/v1 DRA APIs" in the llm-d docs. If your cluster is pinned to an old version, that is a blocker before llm-d is.

llm-d turns distributed inference from a research project into configuration, but it assumes a well-run GPU Kubernetes cluster underneath, and someone on your team has to own that cluster.

How does llm-d compare with KServe, NVIDIA Dynamo, plain vLLM and a hosted API?

Pick by what you are optimizing for: plain vLLM for one model on one node, llm-d when you have several replicas and want smarter routing on vanilla Kubernetes, KServe when you need a general model-serving platform with many frameworks, NVIDIA Dynamo when you are all-in on NVIDIA and want its own orchestration stack, and a hosted API when your volume does not justify GPUs at all.

I like llm-d's positioning here. It keeps the existing engines and extends the Gateway API, which every Kubernetes platform team already understands. That is why it can run on any conformant cluster, including the one you already have.

Where does Qovery fit with llm-d?

Qovery runs the layer llm-d assumes is already there: a production Kubernetes cluster in your own AWS, GCP, Azure or Scaleway account, or on an existing cluster you bring. Qovery does not serve models. vLLM serves them, and llm-d routes to them.

Concretely, on a Qovery-managed cluster you get:

• GPU node pools per service on Qovery-managed clusters on any supported cloud, and Karpenter on AWS so GPU nodes appear when a model server needs them and disappear when it does not.
• Helm services, so the llm-d router chart and your vLLM model servers deploy and roll back like any other service, from Git.
• Managed cluster upgrades, so the Kubernetes version stays recent enough for DRA-based paths without a weekend of manual work. Qovery-managed EKS currently runs Kubernetes up to 1.35.
• RBAC and audit logs, so changing a router config or swapping a model is attributed to a person or an agent token.
• Cost visibility and environment auto-stop, which matters because an idle 8x H200 node costs the same as a busy one.

Two limits to be clear about. Qovery does not configure RDMA fabrics for you: EFA, RoCE or InfiniBand device plugins stay part of your cluster setup, deployable as Helm charts. And cross-cloud networking is not built in; if your model runs on one cloud and your apps on another, you connect them with a VPN, Cilium Cluster Mesh or Tailscale.

For teams without an ML infrastructure engineer, I see one split work. You keep the model in your own account, Qovery keeps the cluster healthy, and llm-d keeps the GPUs busy. If you are weighing where to run the GPUs in the first place, our comparison of the best ways to self-host AI inference on cloud GPUs and the guide to running several open-weight models on one GPU cluster cover that decision.

How do I set this up with Qovery from Claude Code or Cursor?

Install the Qovery Agent Skills, then ask your agent in plain language to create the cluster and deploy the llm-d router and your vLLM model servers in your own cloud account.

1. Install the skills once. They work with Claude Code, Cursor, Codex, OpenCode and Gemini CLI:

curl -fsSL https://skill.qovery.com/install.sh | bash

1. Or scope them to the current project only:

curl -fsSL https://skill.qovery.com/install.sh | bash -s -- --project

1. Connect the Qovery MCP Server to Claude Code so the agent can read your clusters, environments and logs. It uses OAuth and is read-only by default:

claude mcp add --transport http qovery https://mcp.qovery.com/mcp --callback-port 4242

1. Only if you want the agent to make changes, add write access. This also requires enabling write access in the Qovery Console:

claude mcp add --transport http qovery "https://mcp.qovery.com/mcp?read_write=true" --callback-port 4242

The skills (qovery-onboard, qovery-deploy, qovery-troubleshoot, qovery-optimize, qovery-speedup, qovery-preview, qovery-terraform) do the deploying; the MCP Server operates what already exists. A good first prompt: "set up a Kubernetes cluster with a GPU node pool in my AWS account with Qovery, then deploy two vLLM replicas of Qwen3-32B behind the llm-d router as Helm services." The agent asks you to approve each action. Swap AWS for GCP, Azure, Scaleway or your existing cluster. Start from the AI agent quickstart or the Claude Code integration.