How to Self-Host Any Open-Weight Model with NVIDIA NIM's Model-Free vLLM and SGLang Containers on Kubernetes

Key points:

• Model-free NIM containers ship the inference engine (vLLM or SGLang) with no weights inside. You point one image at any supported checkpoint with NIM_MODEL_PATH (hf://, ngc://, s3://, gs://, modelscope:// or a local path) and NIM generates the serving manifest at startup.
• The images are nvcr.io/nim/nvidia/vllm-model-free-nim and nvcr.io/nim/nvidia/sglang-model-free-nim. Both expose an OpenAI-compatible API on port 8000, health checks at /v1/health/live and /v1/health/ready, and the engine's native Prometheus metrics at /v1/metrics. Eligible public tags pull without an NGC API key; Hugging Face downloads need HF_TOKEN.
• Five things decide whether the pod ever runs: GPU quota and a GPU node pool, a GPU driver and device plugin (GPU Operator or your cloud's managed add-on), registry credentials when your tag needs them, model-source credentials, and a persistent volume mounted at /opt/nim/.cache.
• Budget 2 bytes per parameter at BF16/FP16, 1 at FP8, 0.5 at INT4, plus KV cache. Llama 3.1 8B fits one 24 GB GPU; a 70B model at BF16 is about 141 GB of weights on disk and needs tensor parallelism across several GPUs.
• Against cheap hosted open-weight APIs, the token-price break-even is in the billions of tokens per month. Self-host for data residency, model control or high sustained utilization, and attack idle GPU hours first.
• Qovery runs these containers in your own AWS, GCP, Azure or Scaleway account, or on your existing Kubernetes cluster including on-prem GPU nodes, with preview environments, per-environment RBAC and idle auto-stop on top.

For two years, self-hosting an open-weight model on Kubernetes meant picking your pain. Either you hand-built and tuned your own vLLM image, or you used a NIM image welded to exactly one model and waited for NVIDIA to ship the next one. The model-free NIM containers end that trade-off: NVIDIA maintains the engine, you choose the weights.

I have run enough GPU workloads on Kubernetes to know where this goes wrong, so this guide is the version I wanted: verified image names and environment variables, YAML you can apply, a troubleshooting matrix, and honest cost math. Image tags, prices and engine versions below were checked on November 4-5, 2026. They move fast, so treat the NGC catalog as the source of truth.

What are NVIDIA NIM's model-free vLLM and SGLang containers?

Model-free NIM containers are NVIDIA-maintained builds of the vLLM and SGLang inference engines that ship with no model weights inside, so one image serves any open-weight model its engine supports, downloaded at startup from Hugging Face, NGC, S3, Google Cloud Storage, ModelScope or a local directory. Classic NIM microservices work the other way around: each image carries a built-in manifest for exactly one model.

NVIDIA's docs put it plainly: "Model-free NIM lets you run any supported model in NIM LLM and VLM without a model-specific container image" (NVIDIA NIM docs). Operationally, swapping models becomes an environment variable change instead of a new image pull and a new security review.

There are two flavours in the Certified NIM: Model-Free collection:

• vLLM Model-Free NIM (nvcr.io/nim/nvidia/vllm-model-free-nim): PagedAttention, continuous batching and the broadest architecture coverage. NIM 2.1.5 bundles vLLM 0.31.0 (NIM release notes).
• SGLang Model-Free NIM (nvcr.io/nim/nvidia/sglang-model-free-nim): RadixAttention prefix caching, strong structured output, and a good fit for multi-turn agent loops. NIM 2.1.5 bundles SGLang 0.5.21 (same release notes).

Inside the image you get an OpenAI-compatible server on port 8000 (/v1/chat/completions, /v1/completions, /v1/responses, /v1/models), liveness and readiness at /v1/health/live and /v1/health/ready, and the engine's native Prometheus metrics at /v1/metrics with a vllm: or sglang: prefix (NIM observability docs). The vLLM image is 12.89 GB compressed (NGC catalog).

"Any model" has one hard limit. NVIDIA states that if a model's "architecture is unsupported by the inference backend in your container (vLLM or SGLang), it is also unsupported by NIM." Check the vLLM supported models list, which covers well over 100 text-generation architectures, and the SGLang supported models list, which covers dozens of families including DeepSeek, Kimi, Qwen, Llama and GPT-OSS.

On entitlement, quote NVIDIA rather than a blog. To pull NIM containers you need "a free NVIDIA Developer Program membership" or "an NVIDIA AI Enterprise license" (NIM prerequisites). NIM Certified Feature Branch images "are available at no charge under the applicable license terms", while the Production Branch "requires an active NVIDIA AI Enterprise subscription" (NIM offerings). The container itself is governed by the NVIDIA Software License Agreement and the Product-Specific Terms for NVIDIA AI Products, so have legal read those before production.

Why self-host an open-weight model instead of calling a hosted inference API?

Self-host when data residency, model control, predictable latency or very high sustained volume outweigh zero-ops convenience, and keep using a hosted API when volume is low or spiky. A GPU node bills the same per hour at 5% utilization as at 95%, so the break-even is a tokens-per-month number you get by dividing the node's monthly cost by the API's blended per-million-token price.

Here is the worked example with prices checked on November 4, 2026. gpt-oss-120b is 65.25 GB of weights (Hugging Face), so it fits one 80 GB H100.

• Self-hosted: AWS p5.4xlarge (1x H100) is $6.88/hour on demand in us-east-1 (AWS EC2 pricing). Azure Standard_NC40ads_H100_v5 is $6.98/hour in East US (Azure retail prices API). For GCP A3 and Scaleway H100 instances, plug in the current number from Google Cloud GPU pricing or Scaleway GPU pricing.
• Monthly node cost: $6.88 x 730 hours = $5,022.
• Hosted: Amazon Bedrock charges $0.15 per million input tokens and $0.60 per million output tokens for gpt-oss-120b in us-east-1 (Amazon Bedrock pricing), and Together AI lists the same $0.15 / $0.60 (Together AI pricing).
• Blended price at a 3:1 input:output mix: 0.75 x $0.15 + 0.25 x $0.60 = $0.26 per million tokens.
• Break-even: $5,022 / $0.26 = about 19 billion tokens a month, or roughly 7,300 tokens per second, every second, all month.

That number surprises people, and it should. Against aggressively priced open-weight APIs, raw token cost rarely justifies self-hosting on its own. The math flips when you serve a fine-tuned or LoRA model no API hosts, when prompts and logs cannot leave your VPC, when a hosted provider deprecates or silently changes the model you pinned, or when you already own reserved GPU capacity.

The downsides are real: GPU quota approvals take days, cold starts are measured in minutes, CUDA and driver versions drift, and somebody has to own the cluster. My rule of thumb: under a few billion tokens a month, spiky traffic, no GPU quota or no platform owner means stay on a hosted API. I compared the serving engines themselves in vLLM vs TGI vs NVIDIA NIM.

What do you need on your Kubernetes cluster before you deploy a NIM container?

You need five things: GPU quota plus a GPU node pool, an NVIDIA driver and device plugin (the NVIDIA GPU Operator or your cloud's managed GPU add-on), an nvcr.io pull secret using the literal username $oauthtoken when your image tag requires an NGC key, a secret with HF_TOKEN (and NGC_API_KEY if needed), and a persistent volume for /opt/nim/.cache. Miss one and the pod stays Pending, hits ImagePullBackOff, or re-downloads tens of gigabytes on every restart.

GPU nodes and quota. Name the instance families up front: AWS EKS (g5, g6, g6e, p4d, p5), GCP GKE (G2, A2, A3), Azure AKS (NC and ND series), Scaleway Kapsule (L4, L40S, H100), or your own on-prem GPU nodes. Request quota first. AWS tracks GPU quotas per purchasing option under buckets such as "Running On-Demand G and VT instances" and "Running On-Demand P instances" (AWS EC2 quotas). On Google Cloud, "new accounts and projects have a global GPU quota that applies to all regions", and you request per-region quota per GPU model on top (Google Cloud quotas).

Drivers. NIM 2.1.x needs GPU driver 580 or later and CUDA 12.9 or later on the host (NIM prerequisites), so check what your managed node image ships before anything else. The GPU Operator platform support matrix lists Kubernetes 1.33 to 1.37, containerd 2.0 to 2.3 and CRI-O, and validates EKS, GKE and AKS. Use the GPU Operator on on-prem and self-managed clusters, or when you want one driver story across clouds. Use the managed path when you want the cloud to own driver upgrades.

Secrets. Eligible public-catalog model-free tags pull without an NGC key. You still need one for Production Branch images, NIMs older than 2.0.10, private or gated images, and NGC model artifacts (NIM prerequisites). Llama and other gated Hugging Face repos also require you to accept the license on the model page before your token works (Hugging Face gated models).

kubectl create namespace inference

# Only needed if your tag or model source requires an NGC key
kubectl -n inference create secret docker-registry ngc-secret \
  --docker-server=nvcr.io \
  --docker-username='$oauthtoken' \
  --docker-password="${NGC_API_KEY}"

kubectl -n inference create secret generic nim-secrets \
  --from-literal=NGC_API_KEY="${NGC_API_KEY}" \
  --from-literal=HF_TOKEN="${HF_TOKEN}"

VRAM sizing. NVIDIA's own heuristic is weight_memory_per_gpu = total_parameters x bytes_per_parameter / TP, with 2 bytes for BF16/FP16, 1 for FP8 and 0.5 for INT4, NVFP4 and MXFP4 (NIM GPU memory troubleshooting). Llama 3.1 8B at BF16 is 16 GB per GPU and fits a 24 GB card; Llama 3.3 70B at BF16 across four GPUs is 35 GB each, and FP8 halves that to two GPUs. KV cache comes on top and grows with context length and concurrency. Tensor parallel size must equal the GPU count you give the pod.

Storage. Mount a PVC at /opt/nim/.cache (the default NIM_CACHE_PATH, per the NIM environment variables reference), sized above your largest checkpoint. On disk, Llama 3.1 8B Instruct is 16.06 GB of safetensors and Llama 3.3 70B Instruct is 141.11 GB (Hugging Face). A warm cache is the single biggest cold-start win.

Scheduling. Taint GPU nodes with nvidia.com/gpu=present:NoSchedule and add the matching toleration only to inference pods, so no CPU workload squats on a $6.88/hour node.

How do you deploy the model-free vLLM NIM container on Kubernetes step by step?

Deploy it in five steps: smoke-test the image with docker run, create the secrets, apply a Deployment that sets NIM_MODEL_PATH and requests nvidia.com/gpu, expose a ClusterIP Service on port 8000, then verify with curl against /v1/models and /v1/chat/completions. The first boot downloads the weights and takes minutes, so use a generous startupProbe instead of a tight readiness probe.

Step 1: smoke test locally. Run it on any GPU box first. Auth errors, gated-repo errors and unsupported architectures are far easier to read here than in a CrashLoopBackOff. The image tag below comes from NVIDIA's quickstart; check the catalog for newer ones.

export NIM_IMAGE=nvcr.io/nim/nvidia/vllm-model-free-nim:2.1.5
export LOCAL_NIM_CACHE=~/.cache/nim && mkdir -p "$LOCAL_NIM_CACHE"

# See which profiles (TP/PP combinations) NIM generates for your GPUs
docker run --rm --gpus=all -e HF_TOKEN \
  -e NIM_MODEL_PATH=hf://Qwen/Qwen3-8B \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  $NIM_IMAGE list-model-profiles

docker run --rm --gpus=all --shm-size=16g -e HF_TOKEN \
  -e NIM_MODEL_PATH=hf://Qwen/Qwen3-8B \
  -e NIM_SERVED_MODEL_NAME=qwen3-8b \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  -p 8000:8000 $NIM_IMAGE

Step 2: secrets and the Deployment. Create the secrets from the previous section, then apply this. Recreate avoids needing a spare GPU during rollouts, which NVIDIA also recommends for its Helm chart when GPU capacity is tight (NIM Helm docs). The memory-backed /dev/shm matters because vLLM uses PyTorch shared memory, "particularly for tensor parallel inference" (vLLM Docker docs).

apiVersion: apps/v1
kind: Deployment
metadata: {name: nim-vllm, namespace: inference}
spec:
  replicas: 1
  strategy: {type: Recreate}
  selector: {matchLabels: {app: nim-vllm}}
  template:
    metadata: {labels: {app: nim-vllm}}
    spec:
      imagePullSecrets: [{name: ngc-secret}]  # drop if your tag pulls keyless
      tolerations: [{key: nvidia.com/gpu, operator: Exists, effect: NoSchedule}]
      containers:
      - name: nim
        image: nvcr.io/nim/nvidia/vllm-model-free-nim:2.1.5
        ports: [{containerPort: 8000}]
        env:
        - {name: NIM_MODEL_PATH, value: "hf://Qwen/Qwen3-8B"}
        - {name: NIM_SERVED_MODEL_NAME, value: "qwen3-8b"}
        - {name: NIM_TENSOR_PARALLEL_SIZE, value: "1"}
        - name: HF_TOKEN
          valueFrom: {secretKeyRef: {name: nim-secrets, key: HF_TOKEN}}
        resources: {limits: {nvidia.com/gpu: 1}}
        startupProbe: {httpGet: {path: /v1/health/ready, port: 8000}, periodSeconds: 10, failureThreshold: 180}
        readinessProbe: {httpGet: {path: /v1/health/ready, port: 8000}, periodSeconds: 10}
        livenessProbe: {httpGet: {path: /v1/health/live, port: 8000}, periodSeconds: 20}
        volumeMounts: [{name: cache, mountPath: /opt/nim/.cache}, {name: shm, mountPath: /dev/shm}]
      volumes:
      - {name: cache, persistentVolumeClaim: {claimName: nim-cache}}
      - {name: shm, emptyDir: {medium: Memory, sizeLimit: 16Gi}}

Step 3: probes. The startupProbe above allows 180 x 10 seconds, so 30 minutes, for image pull, download and warmup before liveness kicks in. Size it for your largest model and slowest link. Readiness and liveness then use NIM's own /v1/health/ready and /v1/health/live endpoints.

Step 4: cache volume and Service. Keep the Service internal. An unauthenticated public inference endpoint is an open bill and a data-exfiltration path; if something outside the cluster must call it, put an Ingress with authentication in front.

apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: nim-cache, namespace: inference}
spec:
  accessModes: [ReadWriteOnce]   # ReadWriteMany if replicas share one cache
  resources: {requests: {storage: 100Gi}}
---
apiVersion: v1
kind: Service
metadata: {name: nim-vllm, namespace: inference}
spec:
  type: ClusterIP
  selector: {app: nim-vllm}
  ports: [{name: http, port: 8000, targetPort: 8000}]

Step 5: verify. Port-forward, then hit the API. Any OpenAI SDK client works by changing only base_url to http://nim-vllm.inference:8000/v1 and passing a dummy api_key.

kubectl -n inference port-forward svc/nim-vllm 8000:8000 &
curl -s localhost:8000/v1/models
curl -s localhost:8000/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"model":"qwen3-8b","messages":[{"role":"user","content":"Say hi in five words."}],"max_tokens":32}'

Before optimizing anything, write down four numbers from this first boot: image pull time (kubectl describe pod events), weight download time and engine warmup time (container logs), and time to the first successful completion. Those are your baseline.

How does the SGLang NIM container differ from vLLM, and which engine should you choose?

The deployment shape is nearly identical (same registry, same secrets, same NIM_MODEL_PATH, same port 8000, same OpenAI-compatible API), so choose on workload: vLLM for the broadest architecture coverage and high-throughput batch serving, SGLang for prefix-heavy multi-turn chat, agent loops and strict structured output. If you cannot tell, run both against the same model on the same GPU for an hour and let time-to-first-token and output tokens per second at your real concurrency decide.

What carries over: NIM_MODEL_PATH, NIM_SERVED_MODEL_NAME, NIM_TENSOR_PARALLEL_SIZE, NIM_PIPELINE_PARALLEL_SIZE and the canonical --tensor-parallel-size flag, which NIM "translates" into each engine's arguments (NIM model-free docs). What diverges, per NVIDIA's docs rather than my assumption:

• Engine flags. Backend-specific arguments differ: the memory knob is --gpu-memory-utilization on vLLM and --mem-fraction-static on SGLang (NIM memory docs).
• Served model names. On SGLang the name must not contain :, which separates a base model from a LoRA adapter.
• Metrics. NIM passes native engine metrics through unchanged, so dashboards "do not transfer between engines."
• Profiles. list-model-profiles returns vllm-tp1-pp1-... or sglang-tp1-pp1-... depending on the image.

SGLang's advantage shows up when every request resends a long system prompt or a growing agent transcript. SGLang enables prefix caching by default (NIM Helm docs), and its radix tree reuses the shared prefix across requests. vLLM also has prefix caching (--enable-prefix-caching) and exposes vllm:prefix_cache_hits so you can measure it.

I am not quoting a vLLM-vs-SGLang benchmark here. Both projects ship releases every few weeks, so any dated public comparison describes engine versions you are not running, and results swing with model, GPU and prompt shape. Measure your own.

To A/B honestly, run two Deployments and two Services with the same model, the same GPU type and the same load. Use SGLang's bench_serving (SGLang docs) or vLLM's vllm bench serve against both endpoints, since both speak the OpenAI API. Decide on three numbers: p95 time-to-first-token, output tokens per second at your real concurrency, and KV-cache headroom left (vllm:kv_cache_usage_perc vs sglang:token_usage).

How do you keep GPU cost and operations under control in production?

Self-hosted inference bills are dominated by idle GPU hours, not tokens, so cut them in this order: right-size and quantize the model, keep weights on a warm persistent cache, autoscale on queue depth instead of CPU, run batch and preview workloads on spot GPUs, and stop non-production GPU environments when nobody uses them.

Quantize before you scale out. NVIDIA's own example: Llama 3.3 70B at FP8 runs on two GPUs instead of four at BF16 (NIM memory docs). That halves the node bill on day one.

Scale on the right signal. CPU-based HPA is meaningless for GPU inference: the GPU saturates long before CPU moves. Scale on vllm:num_requests_waiting (vLLM metrics) or sglang:num_queue_reqs through the KEDA Prometheus scaler. Point Prometheus at metrics_path: /v1/metrics on port 8000. I went deeper in KEDA queue-depth autoscaling for AI inference.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata: {name: nim-vllm, namespace: inference}
spec:
  scaleTargetRef: {name: nim-vllm}
  minReplicaCount: 1
  maxReplicaCount: 4
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus.monitoring:9090
      query: sum(vllm:num_requests_waiting{model_name="qwen3-8b"})
      threshold: "5"

Do the cold-start math. A new replica pays three costs. The image is 12.89 GB compressed. The weights are 16.06 GB for Llama 3.1 8B and 141.11 GB for Llama 3.3 70B. Then the engine warms up. At an assumed 1 Gbit/s sustained download, the 8B weights take about 2 minutes and the 70B weights about 19 minutes. A pre-warmed PVC removes the download entirely, and node image caching or a pre-pull DaemonSet removes most of the image pull. The new replica is then left with warmup only.

Use spot for the right workloads. AWS advertises Spot at "up to 90%" off on demand (AWS) and Google Cloud Spot VMs "up to 91%" (Google Cloud). On Azure, NC40ads_H100_v5 Spot was $1.42/hour vs $6.98 on demand in East US on November 4, 2026. Put evaluation, batch and preview environments there. Keep production serving on on-demand or committed capacity.

Find the quiet leak. One p5.4xlarge staging node left on outside a 10-hour, 22-day working month idles for 510 of 730 hours. At $6.88/hour that is $3,509 a month of nothing. Multiply by the number of teams with a "temporary" GPU environment.

Instrument on day one. Scrape /v1/metrics and track TTFT, output tokens per second, KV-cache usage and GPU utilization (DCGM exporter). Alert on sustained low utilization as a cost alarm, not a performance one.

How do you make GPU inference repeatable across teams and clouds with Qovery?

Qovery deploys these NIM containers into your own AWS, GCP, Azure or Scaleway account, or onto your existing Kubernetes cluster including on-prem GPU nodes, so the GPU bill, reservations and data stay in your name while developers get a self-service path to a running inference endpoint instead of hand-written GPU manifests.

Bring-your-own-cloud is the point. Qovery operates the cluster inside your cloud account, so Savings Plans, committed-use discounts and GPU reservations stay yours, and prompts never transit our infrastructure. If you already run a cluster, Qovery deploys onto it.

In practice the NIM image becomes a container service. You set the nvcr.io registry credentials once, NGC_API_KEY and HF_TOKEN as secret environment variables, NIM_MODEL_PATH as a plain variable, a GPU node pool, a persistent volume for the cache, and port 8000 exposed internally. Then you add the parts I would otherwise script myself:

• Preview environments per pull request, so an app team tests against a real inference endpoint before merging.
• Environment auto-stop, so the $3,509-a-month staging leak from the previous section stops at night.
• Per-environment RBAC, so data scientists redeploy the inference service with a new model without cluster-admin on a GPU cluster.
• Managed cluster upgrades, so Kubernetes and the GPU node pool stay current without a dedicated platform team.

To be fair, if you already have a mature platform team, the NVIDIA NIM Operator, Helm with Argo CD, KServe, KubeAI or Ray Serve are all good paths, and several of them sit nicely under Qovery. The gap we fill is teams who need GPU inference in their own cloud account without a dedicated DevOps engineer.

How do I deploy a NIM container with Qovery from Claude Code or Cursor?

Install the Qovery Agent Skills once, then ask your coding agent in plain language to deploy the NIM container into your own cloud account. The agent builds the service, environment variables and GPU configuration, and asks you to approve each action before it runs.

Install globally for Claude Code, Cursor, Codex, OpenCode and Gemini CLI:

curl -fsSL https://skill.qovery.com/install.sh | bash
# or scope it to the current repository
curl -fsSL https://skill.qovery.com/install.sh | bash -s -- --project

Add the Qovery MCP Server to Claude Code (OAuth, read-only by default). Enable write access only when you want the agent to change things; it also needs write access enabled in the Qovery Console:

claude mcp add --transport http qovery https://mcp.qovery.com/mcp --callback-port 4242
claude mcp add --transport http qovery "https://mcp.qovery.com/mcp?read_write=true" --callback-port 4242

Use qovery-onboard to connect the cluster, qovery-deploy to ship the NIM container, qovery-troubleshoot for Pending or CrashLoop GPU pods, qovery-optimize for GPU cost, and qovery-preview for pull request environments. Then ask: "deploy the model-free vLLM NIM container serving Qwen3-8B to my GPU node pool on GCP with Qovery", review the plan and approve. The same prompt works with AWS, Azure, Scaleway or your own cluster. Setup details are in the AI agent quickstart and the Claude Code integration.