vLLM dropped CUDA 12.9: an 11-step migration checklist for self-hosted LLMs on Kubernetes GPU nodes

Key points:

• vLLM v0.33.0 ships CUDA 13.0 only, so this is a node-level migration. The CUDA 13.0 runtime needs an NVIDIA Linux driver at version 580.65.06 or higher on every GPU node, so the node image and the driver have to move before the workload does.
• The order that avoids an outage, in 11 steps: inventory GPUs and drivers, pin the last CUDA 12.9 image by sha256 digest, gate on GPU compute capability, create a NEW node pool at the new driver floor, update the GPU Operator or device plugin, rebuild your own layers on the CUDA 13 base, smoke test, canary real traffic, re-benchmark and retune gpu_memory_utilization, drain the old pool, write the rollback down.
• CUDA 13.0 removed offline compilation and library support for Maxwell, Pascal and Volta. If a pool still runs V100 cards (compute capability 7.0), the new vLLM release is a hard stop: replace the hardware or stay pinned to v0.32.0's CUDA 12.9 build.
• Never upgrade drivers in place on a live inference pool. Run blue/green GPU node pools behind one endpoint, raise terminationGracePeriodSeconds well above Kubernetes' 30-second default so streaming completions are not killed mid-stream, and confirm you have the GPU quota to run both pools at once.
• Teams self-hosting vLLM on AWS, GCP, Azure, Scaleway or an existing Kubernetes cluster can let Qovery handle cluster upgrades, node pools and canary environments, while GPU instances, reservations and committed-use discounts stay on their own cloud bill and in their own name.

Bumping CUDA by a major version is a node migration, not an image tag change. Most teams find that out when every vLLM pod goes into CrashLoopBackOff with CUDA driver version is insufficient for CUDA runtime version. On November 2, 2026, vLLM v0.33.0 stopped publishing CUDA 12.9 artifacts. With a project of 96.5k GitHub stars and more than 2,000 contributors behind it, a lot of GPU fleets are about to hit that error.

This is the checklist I would hand to any team running vLLM on Kubernetes GPU nodes this week.

What actually changed in vLLM when CUDA 12.9 support was dropped?

vLLM v0.33.0 publishes wheels and container images built against CUDA 13.0 only. That moves the PyTorch build, the NCCL version and the NVIDIA driver floor all at once, so the new artifacts will not start on a node still running a CUDA 12.9-era driver. The release notes put it in one line: "CUDA 12.9 dropped: release wheels and images are CUDA 13.0 only."

Here is what exists now and what disappeared:

• Published for v0.33.0: CUDA 13.0 Python wheels for x86_64 and arm64 (pip install vllm), the vllm/vllm-openai:v0.33.0 image (CUDA 13.0, default), and vllm/vllm-openai:v0.33.0-efa (CUDA 13.0 plus AWS EFA). Every image is also available under the shorter vllm/vllm name.
• No longer published: "The CUDA 12.9 wheels and the -cu129 / -cu129-ubuntu2404 images." v0.32.0 (October 18) was the last release with vllm/vllm-openai:v0.32.0-cu129.
• Also in the same release: PyTorch 2.14.0, torchvision 0.29.0 and Triton 3.8.0. That matters for Step 6 below.

The dependency chain you actually have to reason about is longer than the image tag:

1. vLLM wheel
2. PyTorch build (now 2.14.0, CUDA 13.0)
3. CUDA runtime inside the image
4. NVIDIA driver on the host node
5. GPU compute capability of the physical card
6. NCCL, for anything using tensor or pipeline parallelism

What does not change: your model weights, the OpenAI-compatible API surface your clients call, and most of your Kubernetes manifests apart from the image reference and node targeting.

Why a major bump hurts more than a minor one: NVIDIA's CUDA Compatibility guide says minor version compatibility "allows applications built within the same major CUDA release family to run on a sufficiently new driver." Crossing from 12.x to 13.x leaves that family. In NVIDIA's CUDA 13.5 release notes, the 12.x family covers drivers ">= 525" and "< 580", while 13.x requires ">= 580".

The two failures teams hit first, verbatim, so you can search your logs for them:

• CUDA driver version is insufficient for CUDA runtime version: the node driver is below the CUDA 13 floor. The pod crashes on startup and loops in CrashLoopBackOff.
• no kernel image is available for execution on the device: the image has no compiled kernels for your GPU's architecture. This is what an old card looks like when it slips through the driver check.

To confirm a mismatch in seconds, run nvidia-smi on the node (or in the driver pod). The header shows both the driver version and the maximum CUDA version that driver supports. If it says anything below 13.0, the v0.33.0 image will not run there.

Which GPUs and NVIDIA drivers still work with vLLM on CUDA 13?

CUDA 13.0 supports Turing through Grace Blackwell and needs a Linux driver at 580.65.06 or newer. Maxwell, Pascal and Volta are gone for good, so a V100 cannot run the new vLLM release whatever driver you install. The CUDA 13.0 release notes say "offline compilation and library support for these architectures have been removed in CUDA Toolkit 13.0" and that "newer toolkits will be unable to target these architectures."

For comparison, CUDA 12.9 GA needed Linux driver >=575.51.03 and CUDA 12.9 Update 1 needed >=575.57.08 (same release notes, Table 3). Compute capabilities below come from NVIDIA's CUDA GPUs and legacy GPUs pages.

NVIDIA's page lists the A10, not the A10G. The A10G is the AWS variant of the same Ampere chip, and the EC2 accelerated computing specs list it as "NVIDIA A10G GPU".

Audit your real state in one pass before you plan anything:

nvidia-smi --query-gpu=name,driver_version,compute_cap --format=csv

# Cluster-wide, using GPU Feature Discovery labels
kubectl get nodes -L nvidia.com/gpu.product -L nvidia.com/cuda.driver.major

What managed Kubernetes ships right now, checked today:

• Amazon EKS: AMI release v20261031 of the AL2023 NVIDIA AMI ships driver 580.178.04 as default and 595.91.07 as latest, with nvidia-container-toolkit 1.20.1. Fresh EKS GPU nodes clear the CUDA 13 floor. Nodes on an AMI from before the 580 branch do not.
• Google GKE: the GKE GPU guide offers gpu-driver-version=default, latest or disabled. Its driver table lists R535 as the default, with R580 available as an option on Container-Optimized OS only. If you took GKE's default driver, you are below the CUDA 13 floor.
• Azure AKS: AKS-managed GPU node pools install and maintain the driver, device plugin and DCGM exporter for you. If you need a specific driver branch, create the pool with --gpu-driver none and run the NVIDIA GPU Operator. The --gpu-driver value cannot be changed on an existing pool.

All three lag upstream NVIDIA by design, so read the driver version off your nodes rather than assuming.

The forward-compatibility escape hatch is real but narrow. NVIDIA's cuda-compat packages let newer CUDA runtimes run on older drivers, but only on "NVIDIA Data Center GPUs", select NGC Server Ready RTX SKUs and Jetson, only on supported driver branches, and never on a removed architecture. It buys you days on a T4 or A100. It does nothing for a V100.

Instance mapping so you can act today:

• AWS: g4dn (T4), g5 (A10G), g6 (L4), g6e (L40S), p4d and p4de (A100), p5 (H100), p5e and p5en (H200), p6-b200 (B200). V100s live on P3, which AWS now files under previous generation.
• GCP: N1 with attached T4 or V100, G2 (L4), A2 (A100), A3 Mega/High/Edge (H100), A3 Ultra (H200), A4 (B200).
• Azure: the NC T4 v3 series (AKS docs use Standard_NC4as_T4_v3 as the minimum GPU size), plus the A100 and H100 families in Azure's GPU VM sizes.
• Scaleway: check the GPU model on each GPU Instance type in Scaleway's catalog, then apply the same table.

What is the step-by-step vLLM CUDA 13 migration checklist?

Here is the full ordered checklist, 11 steps, that takes a vLLM fleet from CUDA 12.9 to CUDA 13 without downtime: inventory, pin the escape route by digest, gate on hardware, build a new node pool, update the GPU stack, rebuild your layers, smoke test, canary, re-benchmark, drain, document the rollback. Copy it into the ticket as-is.

1. Inventory every GPU node pool. Record GPU model, compute capability, driver version, GPU Operator or device plugin version, container toolkit version, and the sha256 digest of the vLLM image currently serving. Why: every later decision keys off these numbers. Effort: 30 to 60 minutes.
2. Pin the escape route by digest. Pin vllm/vllm-openai:v0.32.0-cu129 by sha256 digest, not by tag, and mirror it to your own registry before touching anything. Why: tags can move and upstream registries can prune, and your rollback cannot depend on either. Effort: 15 minutes.
3. Gate on hardware. If any pool runs compute capability below 7.5 (V100 and older), stop and decide: replace the hardware or stay pinned. Why: no driver fixes a removed architecture. Effort: 15 minutes to decide, weeks if hardware has to be procured.
4. Create a NEW GPU node pool at the CUDA 13 driver floor (580.65.06 or newer). Do not upgrade the live pool, and cordon nothing yet. Why: the old pool is your rollback target until Step 10. Effort: 1 to 2 hours, plus quota approval if needed.
5. Update the NVIDIA GPU Operator, device plugin and container toolkit on the new pool. Use a GPU Operator release that lists the R580 branch; in the GPU Operator release notes, v25.3.1 is the first to list 580.65.06. Scope it to the new pool with node labels. Why: an old operator can reconcile the wrong driver onto new nodes. Effort: 1 to 2 hours.
6. Rebuild your own layers on the CUDA 13 base. Re-pin every CUDA-linked dependency to builds made for CUDA 13.0 and PyTorch 2.14.0: flash-attn, xformers, bitsandbytes, AutoAWQ, Triton and any custom kernels. Why: a wheel built for CUDA 12 imports fine and fails at the first kernel launch. Effort: 2 hours to 2 days, depending on how many custom kernels you own.
7. Smoke test on the new pool. Measure model load time, time-to-first-token, output tokens per second, KV cache headroom, and tensor-parallel init over NCCL for multi-GPU models. Why: "Running" does not mean "serving". Effort: 1 to 2 hours.
8. Canary real traffic. Route a small slice of production traffic to the new pool behind the same endpoint, with both versions serving. Why: synthetic prompts never cover your real context lengths. Effort: 1 hour to set up, 24 to 72 hours to observe.
9. Re-benchmark and retune gpu_memory_utilization. Compare against your Step 7 baseline at real concurrency. Why: the vLLM engine arguments now default it to "1.0 with the extensible KV cache (sized from measured memory) and 0.92 otherwise", so a value you tuned on an older release may no longer fit. Effort: 2 to 4 hours.
10. Drain the old pool with a long terminationGracePeriodSeconds, then delete it. Why: in-flight streaming completions need to finish, and the old pool should only go once nothing points to it. Effort: 1 to 2 hours.
11. Write the rollback down. Which digest, which pool, which driver, who decides, and the trigger thresholds. Why: rolling back after Step 10 means re-imaging nodes, which is exactly why the old pool survives until then. Effort: 30 minutes.

Total: roughly two working days of hands-on time for a single model on modern GPUs, plus the canary window. Add procurement time if Step 3 fails.

How do you upgrade GPU node drivers on Kubernetes without taking inference down?

You do not upgrade drivers in place on a live inference pool. You create a second node pool at the new driver version, run both vLLM versions side by side behind one endpoint, and drain the old pool only after the canary holds. NVIDIA's own procedure takes GPU workloads off the node anyway: the GPU Operator driver upgrade guide requires you to "disable all clients to the GPU driver" and unload the kernel modules, and its upgrade controller cordons the node and deletes "pods allocated with GPUs" before reloading.

In-place rollouts through the GPU Operator's driver DaemonSet are fine in one narrow case: batch or non-user-facing work on a single pool, where a node going dark for a few minutes costs nothing. For anything answering users, blue/green wins because the old pool is your instant rollback.

Raise the grace period. Kubernetes gives a pod a termination window that "defaults to 30 seconds". A long-context streaming completion can run for minutes, and the default kills it mid-stream. Set terminationGracePeriodSeconds to your longest realistic generation time plus margin (I start at 300 to 600 seconds), and add a preStop hook that takes the pod out of the load balancer before vLLM stops. The docs note that if preStop needs longer than the grace period, you "must modify terminationGracePeriodSeconds to suit this."

The primitives that steer pods during the transition:

• nodeSelector or node affinity on nvidia.com/gpu.product or a custom pool label, so the v0.33.0 Deployment only lands on the new pool.
• Per-pool taints and tolerations, so nothing unrelated lands on scarce GPUs.
• A PodDisruptionBudget on each vLLM Deployment, so a drain never takes all replicas at once.
• A preStop hook that stops accepting new requests and lets in-flight ones finish.

Model weights make new pools slow. Llama 3.1 70B Instruct is 30 safetensors shards adding up to roughly 141 GB. Every cold pod on a fresh node pulls that before it serves a token. Bake weights into the image, mount a shared read-only volume, or keep a node-local model cache warm before you shift traffic.

Do the capacity math before the window. Kubernetes exposes GPUs as a "custom schedulable resource such as nvidia.com/gpu", requested as whole units, so two pools mean two sets of physical GPUs. On AWS, the default On-Demand quota for "Running On-Demand G and VT instances" and for P instances is 0 vCPUs until you request more. Quota is usually the binding constraint on AWS, GCP, Azure and Scaleway alike, so file the increase days ahead.

One more rule: do not combine a Kubernetes version upgrade with a driver floor change in the same window. When something breaks, you want one suspect, not two.

How do you validate that the new vLLM release is actually working after the migration?

Validation is a numeric before/after on four metrics, never "the pod is Running": model load time, time-to-first-token, output tokens per second at your real concurrency, and peak GPU memory. Measure all four with vLLM's own benchmark tools against the same dataset and request rate.

Use the vLLM benchmark CLI on both pools with identical inputs:

vllm bench serve --backend vllm --model <your-model> \
  --endpoint /v1/completions --dataset-name random \
  --num-prompts 500 --request-rate inf --max-concurrency 32

# Offline throughput, no separate server
vllm bench throughput --model <your-model> --dataset-name random --num-prompts 500

The docs call --request-rate=inf --max-concurrency=<limit> the "most commonly used configuration for production benchmarking." Set the limit to your real peak concurrency and paste both runs into the ticket. Older teams may still call benchmark_serving.py; vllm bench serve is the entry point the docs now lead with.

Then watch the Prometheus metrics vLLM serves at /metrics during the canary:

• vllm:time_to_first_token_seconds and vllm:e2e_request_latency_seconds for latency
• vllm:num_requests_running versus vllm:num_requests_waiting for queueing
• vllm:kv_cache_usage_perc for KV cache pressure ("1 means 100 percent usage")
• vllm:num_preemptions for memory-driven preemption

Check metric names after the upgrade too. vLLM hides deprecated metrics in the next minor version, so a dashboard that goes blank may be a renamed series rather than a broken pod.

Run a correctness check with fixed-seed prompts, old versus new, and pay extra attention to FP8, AWQ and GPTQ models, where kernel changes bite hardest. For multi-GPU models, confirm NCCL initializes and run a tensor-parallel sanity request; NCCL moves with the CUDA build, and a hang at init is the classic symptom.

Agree on the rollback trigger before the canary starts: a concrete p95 latency ceiling and an error-rate threshold, written down, with a named owner. And remember a vLLM release changes more than CUDA (defaults, scheduler behavior, memory accounting), so a throughput delta is not automatically a CUDA problem.

Should you pin the old CUDA 12 vLLM release instead of migrating now?

Pinning vLLM v0.32.0's CUDA 12.9 image by digest is a legitimate short-term answer, but it has an expiry date. New model architectures, kernel performance work and security fixes land in new releases, and the backport burden becomes yours from the day you pin.

Valid reasons to pin: V100s or older in the fleet, a node image you cannot change this quarter, or an open compliance or certification window. The costs are real, though. New model families ship in new vLLM releases first, nobody backports fixes to your frozen stack, and CVEs in the CUDA, PyTorch and vLLM layers stay open until you move.

Put a date and a named owner on every pin. A pinned digest with no owner is how a cluster ends up three majors behind.

The fair alternative: if you do not need the weights inside your own account, a managed inference API is cheaper than owning this compatibility matrix at all.

Why are CUDA version migrations so painful, and how does a platform layer reduce the blast radius?

The pain is almost never vLLM itself. The node image, NVIDIA driver, container toolkit, device plugin, CUDA runtime and framework wheel are six separately versioned things, and most teams self-hosting LLMs have no dedicated DevOps engineer who owns that matrix.

Change one link in node image, driver, container toolkit, device plugin, CUDA runtime, framework wheel and you revalidate all six. AI teams get hit hardest because ML engineers stand up inference clusters fast to ship a feature, and then nobody owns the upgrade path until a release like v0.33.0 breaks it.

This is where I think a platform layer earns its keep. Qovery runs the cluster and node pools inside your own AWS, GCP, Azure or Scaleway account, or connects to your existing Kubernetes cluster. It brings managed Kubernetes upgrades (each version gets a month of internal testing before rolling out), Karpenter-based autoscaling on AWS, per-environment RBAC, and preview and cloned environments, so the blue/green pool and the rollback become repeatable instead of tribal knowledge.

BYOC, said plainly: your GPU instances, reservations, Savings Plans and committed-use discounts stay on your own cloud bill and in your own name.

The honest boundary: Qovery does not replace the NVIDIA GPU Operator and does not choose your driver version. It makes the cluster, node pool and environment lifecycle around that choice repeatable.

Managed inference APIs are the right call for plenty of teams. You trade the weights-in-your-account control for someone else owning CUDA entirely. If you need that control, someone has to own the matrix, and the question becomes how much of the surrounding lifecycle you want to hand-build.

How do I run this migration with Qovery from Claude Code or Cursor?

You can hand the mechanical parts of this migration to a coding agent: install the Qovery Agent Skills once, add the Qovery MCP Server, then describe the migration in plain language and approve each change the agent proposes.

Install the skills globally for a compatible agent such as Claude Code or Cursor:

curl -fsSL https://skill.qovery.com/install.sh | bash

Or scope them to the current project:

curl -fsSL https://skill.qovery.com/install.sh | bash -s -- --project

Add the Qovery MCP Server to Claude Code. It uses OAuth and is read-only by default, which is what you want while you inspect what is running:

claude mcp add --transport http qovery https://mcp.qovery.com/mcp --callback-port 4242

Enable write access only when you want the agent to change things. This also requires enabling Copilot and write access in the Qovery Console:

claude mcp add --transport http qovery "https://mcp.qovery.com/mcp?read_write=true" --callback-port 4242

How the pieces split, in Qovery's words: "The Skill deploys. The MCP Server operates." The skills (qovery-deploy, qovery-preview, qovery-troubleshoot and qovery-optimize, among others) ship things; the MCP Server queries and operates what already exists.

Example prompts for this migration:

• "Show me every environment referencing the old vLLM image digest."
• "Create a canary environment running the pinned v0.32.0 CUDA 12.9 vLLM image alongside the new one."
• "Roll back this environment to the previous deployment."

On consent: the skill asks before touching your code. The MCP Server cannot write until you flip both switches, every action runs within your RBAC, and every action is audited in the Qovery Console. Start read-only, and switch to write only for the steps you have reviewed. Setup details live in the AI agent quickstart and on the Claude Code integration page.