NVIDIA NIM 2.1.5 Fixed 81 Vulnerabilities, and 2.1.6 Fixed 99 More: A Safe Upgrade Runbook for Self-Hosted Models on Kubernetes GPU Nodes
NIM 2.1.5 fixed 81 unique security issues (3 critical) across NVIDIA's vLLM and SGLang images, and 2.1.6 fixed 99 more a week later. Here is the eight-step runbook I use to upgrade self-hosted NIM containers on Kubernetes GPU nodes: inventory, bulletin triage, scanning, digest pinning, config diffs, cache warm-up, a one-pool canary and rollback.
The numbers check out. NVIDIA's release notes say NIM 2.1.5 "fixes 81 unique security issues across the vLLM and SGLang images (3 critical, 18 high, 59 medium, and 1 low)": 62 in the vLLM images and 60 in the SGLang images (NIM 2.1.5 release notes).
2.1.5 is already superseded. NIM 2.1.6 fixes "99 unique security issues" (10 critical, 30 high, 45 medium, 14 low) and upgrades vLLM to 0.32.0 and SGLang to 0.5.22 (NIM release notes). On NGC, the nim-llm Helm chart shows 2.1.5 on 27 October and 2.1.6 on 3 November 2026 (NGC). If you are still on 2.1.4 or older, plan for 2.1.6.
Exposure depends on your flags. The 2.1.5 critical in lmcache sits in a server "which NIM LLM and VLM does not start". One vLLM CVE applies "only when the server runs with --trust-remote-code". Read the notes against your own configuration before you rank urgency.
The upgrade breaks things. Each release changes engine flags, API response shapes and environment variables. 2.1.6 also invalidates the cached runtime manifest when the image changes, which matters if you run air-gapped.
The runbook: inventory every running NIM image by digest, triage the release notes, scan with Trivy or Grype, pin the new image by digest, diff your config, warm the model cache, canary on one GPU node pool, keep the previous digest ready for rollback.
A NIM image bundles an inference engine, a Python stack, Ray, nginx, FFmpeg, a Go binary inside Mooncake and an Ubuntu base. Every one of those layers ships CVEs. NVIDIA's 2.1.5 notes list fixes in at least a dozen packages, and the image you pull is around 13 GB compressed (NGC catalog). Patching it is closer to a database major-version upgrade than to bumping a web app's base image.
I am a Staff Solutions Architect at Qovery, and I help teams run inference on GPU nodes in their own cloud accounts. The pattern I see after a security release is always the same: someone bumps the tag, the pod restarts, the model takes ages to load, one engine flag no longer parses and the GPU node sits idle while the team debugs. This runbook is how I avoid that. If you have not deployed NIM yet, start with our guide to the model-free vLLM and SGLang containers on Kubernetes; this post assumes you already run them.
What exactly did NIM 2.1.5 fix, and are the counts right?
NIM 2.1.5 fixed 81 unique security issues across the vLLM and SGLang images: 3 critical, 18 high, 59 medium and 1 low. The vLLM images account for 62 (3 critical, 15 high, 43 medium, 1 low) and the SGLang images for 60 (2 critical, 16 high, 41 medium, 1 low), with overlap between them. These figures come straight from NVIDIA's release notes.
NVIDIA publishes product security bulletins on its security portal, but for NIM container fixes the release notes are the primary record. Each release has a "Security Fixes" section with the package, the old and new version, the issue count and an expandable list of CVE and GHSA IDs.
The 2.1.5 fixes that matter most, from the release notes:
Component
Change
Issues fixed
Images
Who is exposed
Go stdlib and golang.org/x/net in Mooncake's libetcd_wrapper.so
Go 1.26.8 to 1.26.9, x/net v0.58.0 to v0.60.0
13 (2 critical: CVE-2026-56857, CVE-2026-78663)
vLLM and SGLang
Anyone running the image
lmcache
0.5.5 to 0.5.6
1 critical (CVE-2026-105192), unauthenticated RCE
vLLM
Only if you start the LMCache multiprocess server yourself; NIM does not
Multimodal requests; CVE-2026-114053 only with --trust-remote-code
aiohttp, including Ray's vendored copy
3.14.3 to 3.14.5 / 3.14.4
8 (3 high)
vLLM and SGLang
Anyone running the image
FFmpeg used by PyAV
Rebuilt from 9.0.2 with the RTSP redirect fix
1 high (CVE-2026-107696)
vLLM and SGLang
Audio and video inputs
nltk and cosmos-guardrail
Removed
2 high
SGLang
Not used at runtime, per NVIDIA
PyJWT, libexpat1, curl, sudo, nginx and others
Patch updates
Medium-severity bulk
Both
Anyone running the image
Two lines from NVIDIA's notes change how you should rank this upgrade. On lmcache: "The flaw allows unauthenticated remote code execution through the network transport of the LMCache multiprocess server, which NIM LLM and VLM does not start." On vLLM: "CVE-2026-114053 applies only when the server runs with --trust-remote-code." NVIDIA adds that some Qwen3.5 and Nemotron omni profiles run with --trust-remote-code and says to "Serve these NIMs only to trusted clients."
Should you upgrade to 2.1.5, or go straight to 2.1.6?
Go straight to 2.1.6 unless you have a validated reason to stop at 2.1.5. 2.1.6 fixes 99 more unique issues, including 10 critical, and it is the release NVIDIA's "latest" documentation now describes. Stopping at 2.1.5 means doing this whole runbook twice.
Step 1: How do you find every NIM image running on your clusters?
Query the running pods for their image reference and the resolved image ID, which includes the digest, then filter on nvcr.io/nim. The tag tells you what someone intended to deploy; the digest tells you what is running.
BASH
kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{range .status.containerStatuses[*]}{.image}{"\t"}{.imageID}{"\n"}{end}{end}' \
| grep 'nvcr.io/nim' | sort -u
Run it against every cluster, including the staging and research clusters nobody remembers. Then check the other places a NIM can hide:
Helm releases:helm list -A and look for the nim-llm chart and its version.
NIM Operator resources: if you use the NVIDIA NIM Operator, the image lives in NIMService resources (with weights in NIMCache), not in a Deployment you wrote.
Custom tags: NVIDIA ships variant tags such as 2.1.4-minimax-m3 or 2.1.6-qwen3.8-2.4t. A grep for :2.1.5 misses them.
Mirrors: if you copied images into ECR, Artifact Registry or ACR, the running image reference no longer contains nvcr.io. Grep for model-free-nim and your model names too.
Write the result into a table with cluster, namespace, workload, tag, digest, GPU type and the engine flags in use. You need the flags for Step 2.
Step 2: How do you read NVIDIA's notes as a security bulletin?
Read the "Security Fixes" section of every release between your current version and the target, then check each critical and high item against three questions: is the vulnerable component started in my configuration, is the vulnerable feature enabled by my flags, and is the endpoint reachable by untrusted clients. That turns 81 or 99 line items into a short list.
For 2.1.5 and 2.1.6 the questions map to concrete checks:
Do you pass --trust-remote-code? Grep your NIM_PASSTHROUGH_ARGS and profile settings. If yes, CVE-2026-114053 applies to you, and NVIDIA's advice to serve only trusted clients applies regardless of version.
Do you accept images, audio or video? The vLLM multimodal CVEs and the FFmpeg RTSP fix matter for VLM workloads. Text-only LLMs are less exposed.
Does NIM terminate TLS? The nginx CVE-2026-97629 fixed in 2.1.6 applies when NIM "serves TLS or mutual TLS". If TLS ends at your ingress or service mesh, NIM's nginx is not your TLS endpoint.
Who can reach port 8000? An inference endpoint on a public LoadBalancer with no authentication turns every "unauthenticated" CVE into an emergency. A ClusterIP Service behind an authenticated gateway buys you time.
Whatever the answers, the Go and aiohttp fixes apply to every deployment, so "we do not use that feature" never removes the upgrade from the list. It only changes whether you do it this week or tonight.
Step 3: How do you scan NIM images with Trivy or Grype?
Scan both the image you run and the target image, gate on critical and high findings that have a fix available, and keep the SBOM. Expect your scanner's numbers to differ from NVIDIA's, because Trivy and Grype use their own vulnerability databases and count per package and per CVE differently.
BASH|Trivy: fail on fixed critical/high findings; large images need a longer timeout
trivy image --severity CRITICAL,HIGH --ignore-unfixed --exit-code 1 --timeout 30m \
nvcr.io/nim/nvidia/vllm-model-free-nim:2.1.6
# Keep an SBOM next to the release record
trivy image --format cyclonedx --output nim-2.1.6.cdx.json \
nvcr.io/nim/nvidia/vllm-model-free-nim:2.1.6
# Grype: same gate, from the image or from the SBOM
grype nvcr.io/nim/nvidia/vllm-model-free-nim:2.1.6 --only-fixed --fail-on critical
Scan in CI, not on a laptop. The pull alone takes minutes. Cache the scanner DB and run the scan once per digest, not once per deploy.
Compare old and new. The useful output is the diff: what the new digest fixes and what it introduces. A new critical in the target image is a reason to pause.
Triage unfixed findings with NVIDIA's reasoning. For example, NVIDIA removed uv because NIM "does not use uv at runtime". Use the same logic for packages your scanner flags but your workload never loads, and record the decision.
Verify the image's origin too. NVIDIA says "Every NIM is digitally signed to detect and prevent tampering, enabling enterprises to verify authenticity through the NVIDIA public key published on the NGC Catalog" (NVIDIA Technical Blog). Verify with cosign and that key before you mirror the image. Model weights from NGC are signed separately with OpenSSF Model Signing, and NIM "automatically performs internal checksum verification of downloaded files" (NIM docs).
Step 4: Why should you pin NIM images by digest?
Pin by digest because a tag can move and a digest cannot. Kubernetes documents that "Tags can be moved to point to different images, but digests are fixed," and that with tags "you might end up with a mix of Pods running the old and new code" (Kubernetes docs). On a GPU fleet that scales nodes up and down, a mixed fleet means two engine versions answering the same endpoint.
BASH|Resolve the digest of the tag you validated
crane digest nvcr.io/nim/nvidia/vllm-model-free-nim:2.1.6
# Copy that exact digest to your own registry, then deploy the mirror by digest
crane copy nvcr.io/nim/nvidia/vllm-model-free-nim@sha256:<digest> \
<your-registry>/nim/vllm-model-free-nim:2.1.6
In the Deployment, reference image: <your-registry>/nim/vllm-model-free-nim@sha256:<digest>. Mirroring has a second benefit for GPU pools: when Karpenter or a cluster autoscaler adds a node during a traffic spike, the 13 GB pull comes from a registry in your region instead of from nvcr.io.
Keep the previous digest in the same registry and in the same release record. Step 8 depends on it.
Patch GPU inference without a platform team.
Qovery runs your NIM containers on GPU node pools in your own AWS, GCP, Azure or Scaleway account, or your existing cluster, with cloned staging environments, rollback and an audit log of every deploy.
Step 5: What should you test in staging before touching production?
Test the new digest against a clone of production's configuration: same model, same profile, same engine flags, same GPU type. Confirm it starts, selects the profile you expect, passes your evaluation set and returns responses your clients can parse.
My staging checklist:
Profile selection. Run list-model-profiles on the new image and confirm the profile ID you pin in NIM_MODEL_PROFILE still exists. 2.1.5 warns that a model-store profile ID "might have changed in this release".
Flags. Start the container with your exact NIM_PASSTHROUGH_ARGS. 2.1.6 now fails fast on bad parser names and on conflicting SGLang data-parallel flags, which is better than failing at request time, but only if you catch it in staging.
Client contracts. Replay a sample of real requests, including tool calls, structured output, streaming /v1/responses and multimodal inputs if you serve them. The table above lists the shape changes.
Quality. Run your evaluation set. Engine upgrades change kernels and defaults; 2.1.5 notes that the SGLang FlashInfer fused MoE finalize step "is off by default, which can change NVFP4 MoE performance".
Latency and memory. Compare time to first token, throughput and peak GPU memory against the old digest on the same GPU. The engine's Prometheus metrics at /v1/metrics are enough for a side-by-side.
Teams self-hosting MoE models should repeat the check for each model and GPU pair they serve. Our post on running GLM-5.3 and Qwen3.8-Flash-Next with NIM has the support-matrix caveats per model.
Step 6: How do you handle the model cache and warm-up during an upgrade?
Keep the weights cache on a persistent volume so the upgrade does not re-download hundreds of gigabytes, expect the first start on a new image to regenerate NIM's runtime manifest, and turn on the startup warm-up so the pod only reports ready after the first request has compiled its kernels.
Cached runtime manifests. For remote NIM_MODEL_PATH values (ngc://, hf://), NIM now saves the manifest in NIM_CACHE_PATH with a nim_runtime_manifest.key.json key file. "If the container image or its tuned recipes changed," NIM regenerates the manifest first. Every image upgrade therefore needs network access on its first start.
Air-gapped clusters. NVIDIA's instruction is explicit: "After you upgrade the container image, run one start with network access to refresh the cached manifest and its key before you deploy in an air-gapped environment." Do that in a connected staging environment, then ship the refreshed cache with the image.
Moving references. A NIM_MODEL_PATH that names a :latest tag or a Hugging Face branch "keeps serving the files that it first resolved to." NVIDIA's advice is to "Pin NIM_MODEL_PATH to a fixed version." Pin the weights the same way you pin the image.
Warm-up. Set NIM_STARTUP_WARMUP=1. NIM then sends one short chat request after the backend reports healthy and "reports ready only after the request returns, fails, or times out." NIM_STARTUP_WARMUP_TIMEOUT_SECS defaults to 900. This keeps a cold pod out of the load balancer until it can answer in normal time.
Two Kubernetes details I add every time. First, a startupProbe on /v1/health/ready with a failureThreshold long enough to cover a cold load, so the kubelet does not kill the pod mid-download. Second, a separate cache volume for the canary, so the old and new images never write manifests into the same directory while both are live. The second one is my own precaution; NVIDIA does not document concurrent writers either way.
Step 7: How do you canary a NIM upgrade on one GPU node pool?
Run the new digest as a separate Deployment pinned to one GPU node pool, send it a small share of traffic, compare error rates and latency against the old version, then widen. A canary on its own pool keeps a bad image from evicting healthy replicas and gives you a clean place to look at GPU metrics.
GPU capacity decides the strategy. A default rolling update with maxSurge needs a spare GPU to start the new pod before the old one stops. If your GPUs are fully booked, the rollout stalls or you set maxSurge: 0 and drop capacity during the upgrade. Decide which one you accept before you start.
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: llm
spec:
parentRefs:
- name: inference-gateway
rules:
- backendRefs:
- name: nim-2-1-4 # current digest
port: 8000
weight: 90
- name: nim-2-1-6 # canary digest, on the canary node pool
port: 8000
weight: 10
Watch these during the canary, side by side for both versions:
HTTP 4xx and 5xx rates, split by endpoint. A jump in 400s usually means a client hit one of the request-shape changes.
Time to first token and inter-token latency from /v1/metrics.
GPU memory and OOM restarts. 2.1.5 makes memory estimates "advisory in profile selection", and NVIDIA warns that "the inference backend can still run out of memory at startup".
Pod restarts and readiness flaps on the canary pool.
Give it at least one full daily traffic cycle before you go past 50%.
Step 8: How do you roll back a NIM upgrade safely?
Roll back by routing traffic to the previous digest, which you kept running or can redeploy from your own registry, and remember that a rollback puts the old vulnerabilities back. Pair it with compensating controls until you can retry the upgrade.
Practical rollback rules:
Keep the old Deployment alive until the canary reaches 100% and survives a traffic peak. Rolling back is then a route change that takes seconds instead of a full model load.
For in-place updates, kubectl rollout undo deployment/<name> returns to the previous ReplicaSet (Kubernetes Deployment docs). It only works quickly if the old image is still in a registry your nodes can reach.
Expect a manifest regeneration when the old image starts on a cache that the new image touched, since the key file records which image generated it. Use the separate canary cache from Step 6 to avoid this.
Compensate while you are on the old version. Remove --trust-remote-code where you can, restrict who can reach port 8000, terminate TLS outside NIM, and disable multimodal inputs for endpoints that do not need them.
Write down why you rolled back and which release note line caused it. The next attempt starts from that note.
Where does Qovery fit in a NIM upgrade?
Qovery runs NIM containers as services on GPU node pools inside your own AWS, GCP, Azure or Scaleway account, or on your existing Kubernetes cluster, so the upgrade path above becomes environment operations instead of hand-written manifests. Qovery deploys and operates the workload; NVIDIA's image, weights and patches stay exactly what they are.
What that looks like for this runbook:
Inventory: each NIM is a service with a visible image tag per environment, so Step 1 is a list, not a cluster-wide jsonpath query.
Staging: clone the production environment, change the image tag to the validated release, and test with the same variables, GPU node pool and volumes. A preview environment per pull request does the same for config changes.
Rollout and rollback: deploy the new version to one environment first, and redeploy the previous version if the canary fails. Every deploy, and who or which agent triggered it, lands in the audit log.
Agents: an AI agent can run the staging steps through the Qovery API with an API Policy Token that allows deploys on staging and denies production.
Your account: the GPU bill, reservations, model weights and prompts stay in your cloud account. Qovery never hosts the model.
To be fair, if you already run the NIM Operator with Argo CD and a mature platform team, you can implement every step here without Qovery. The gap we fill is teams who self-host models in their own cloud without a dedicated DevOps engineer and still need a patch process an auditor will accept. The model-free NIM container guide covers the first deployment end to end.
How do I run this upgrade with Qovery from Claude Code or Cursor?
Install the Qovery Agent Skills and the MCP Server, then ask your agent to clone the production environment, switch the NIM service to the new image and report what fails.
BASH|1. Agent Skills: one install for Claude Code, Cursor, Codex, OpenCode, Gemini CLI
curl -fsSL https://skill.qovery.com/install.sh | bash
# 2. Or scoped to the current project only
curl -fsSL https://skill.qovery.com/install.sh | bash -s -- --project
# 3. MCP Server for Claude Code over OAuth, read-only by default
claude mcp add --transport http qovery https://mcp.qovery.com/mcp --callback-port 4242
# 4. Write access, only when you explicitly want it (also enable write access in the Qovery Console)
claude mcp add --transport http qovery "https://mcp.qovery.com/mcp?read_write=true" --callback-port 4242
Your agent asks you to approve each command, and the MCP Server opens an OAuth login in your browser and acts with your user permissions. Start read-only and ask: "list every service running an nvcr.io/nim image and its tag". Then, with write access: "clone the production inference environment, set the NIM service to the 2.1.6 image and deploy it", and qovery-troubleshoot helps when the first start fails. See the AI agent quickstart and the Claude Code integration.
Frequently asked questions
How many vulnerabilities did NVIDIA NIM 2.1.5 fix?
NIM 2.1.5 fixed 81 unique security issues across its vLLM and SGLang images: 3 critical, 18 high, 59 medium and 1 low. The vLLM images account for 62 of them and the SGLang images for 60, with overlap between the two. The source is the Security Fixes section of NVIDIA's 2.1.5 release notes.
Should I upgrade NIM to 2.1.5 or 2.1.6?
Upgrade to 2.1.6 unless you have a validated reason to stop at 2.1.5. NIM 2.1.6 fixes 99 more unique security issues, including 10 critical in Pillow, nginx, pip and the removed uv package, and moves to vLLM 0.32.0 and SGLang 0.5.22. Read both releases' behavior changes, because jumping two versions means two sets of flag and API changes.
What does a safe upgrade and migration checklist look like for self-hosted NIM on Kubernetes GPU nodes?
Inventory every running NIM image by digest, triage the release notes against your flags and exposure, scan old and new images with Trivy or Grype, and pin the validated image by digest in your own registry. Then test the exact production configuration in staging, warm the model cache with network access and NIM_STARTUP_WARMUP=1, canary on one GPU node pool with weighted routing, and keep the previous digest ready for rollback.
Why do Trivy and Grype report different numbers than NVIDIA?
Trivy, Grype and NVIDIA each use their own vulnerability data and count findings differently, for example per package copy or per advisory alias. Scanners also flag packages that are present in the image but never loaded by NIM at runtime. Use NVIDIA's notes for what was fixed and your scanner for what is still present, and gate on fixed critical and high findings.
Does upgrading NIM require re-downloading the model weights?
No, as long as the weights cache lives on a persistent volume mounted at NIM_CACHE_PATH and the model path and profile stay the same. From 2.1.6, NIM regenerates its cached runtime manifest when the container image changes, so the first start on a new image needs network access. Air-gapped teams should run that first start in a connected environment and ship the refreshed cache with the image.
Is it safe to stay on an older NIM version if my endpoint is internal?
An internal endpoint lowers the risk but does not remove it, because fixes in the Go, aiohttp and Ubuntu layers apply to every deployment. If you must delay, keep the endpoint behind authenticated routing, avoid --trust-remote-code, terminate TLS outside NIM and disable multimodal inputs you do not use. Schedule the upgrade within your normal patch window for critical findings.
Guillaume is a Staff Solutions Architect at Qovery. He helps customers troubleshoot production infrastructure and writes about Kubernetes operations, AI agents, and reliable cloud platforms.
Next step
Patch GPU inference without a platform team.
Qovery runs your NIM containers on GPU node pools in your own AWS, GCP, Azure or Scaleway account, or your existing cluster, with cloned staging environments, rollback and an audit log of every deploy.