How to Self-Host Open-Weight Models With NVIDIA's Model-Free NIM Container (One Image, Any Supported Architecture)

Key Points:

• A model-free NIM container is one NVIDIA image with no weights inside. You pass the model at start time with NIM_MODEL_PATH (for example hf://openai/gpt-oss-20b, an s3:// bucket or a local directory). NVIDIA publishes two of them: vllm-model-free-nim and sglang-model-free-nim (NVIDIA NIM docs).
• "Any model" means "any architecture the bundled engine supports". NIM 2.1.5 ships vLLM 0.31.0 and SGLang 0.5.21 (NIM release notes). Read the architectures field in the checkpoint's config.json, then check it against that engine version before you rent a GPU.
• Kimi-K3 and MiniMax-M3 are on NVIDIA's validated list today. Each has a dedicated tag of the same model-free image (2.1.4-kimi-k3, 2.1.4-minimax-m3) (NIM support matrix). Kimi K3.1 Pro, released three days ago, is not on the list yet.
• Size memory from total parameters, not active ones. Weights = total params x bytes per param (2 at BF16, 1 at FP8, about 0.5 at FP4). Add KV cache and 10-20% overhead on top. Kimi-K3 has 2.8T total parameters but activates 104B per token, and NVIDIA's smallest validated layout is one 8x B300 node.
• Self-hosting only beats a token API at high sustained utilization. $/M tokens = hourly node price / (tokens per second x 0.0036). On a $55.04/hour AWS p5.48xlarge at SGLang's published 22,282 output tokens/s per node, that is $0.69/M at 100% use and $6.86/M at 10%.
• The economics hold up only when the GPUs sit in your own cloud account. That is where your Savings Plans, committed-use discounts, capacity reservations and data residency apply. Qovery deploys and operates the workload in your own AWS, GCP, Azure or Scaleway account, or on your existing Kubernetes cluster, and auto-stops idle environments.

A new open-weight model drops on a Friday. Your team wants it in staging by Monday. Until recently, NIM users had to wait for NVIDIA to publish a per-model image on NGC, or give up on NIM and wire vLLM themselves.

The model-free NIM container removes that wait for any model whose architecture the bundled engine already understands. I've spent the last few weeks reading the docs, the support matrix and the model cards to work out what that means in practice.

This guide covers how to self-host open-weight models with it end to end: the support check, the VRAM math, the Docker commands, the Kubernetes wrapper and the cost break-even.

What is NVIDIA's model-free NIM container, and how is it different from a per-model NIM image?

A model-free NIM container is a single NVIDIA NIM image that ships the inference engine (vLLM or SGLang), NVIDIA's profile-selection logic and an OpenAI-compatible API server, with no model weights inside. You point it at a model at start time with NIM_MODEL_PATH, and NIM generates a runtime manifest, picks a profile and serves the model (NVIDIA NIM docs).

The limit, in NVIDIA's own words: "if its architecture is unsupported by the inference backend in your container (vLLM or SGLang), it is also unsupported by NIM." So the NVIDIA NIM model-free container serves any model whose architecture the bundled engine version supports. It does not serve every model on Hugging Face.

The classic packaging model: one container per model, versioned with that model, published on NGC (for example nvcr.io/nim/meta/llama-3.1-8b-instruct:2.1.5). The manifest, curated weights and optimized profiles are baked in. That works well for models NVIDIA has already packaged, and it blocks you the week a new one ships.

The model-free approach:

• One image per engine: nvcr.io/nim/nvidia/vllm-model-free-nim:2.1.5 or nvcr.io/nim/nvidia/sglang-model-free-nim:2.1.5 (NIM installation).
• Weights come from hf://, ngc://, s3://, gs://, modelscope:// or an absolute local path.
• Tensor and pipeline parallelism are runtime settings (NIM_TENSOR_PARALLEL_SIZE, NIM_PIPELINE_PARALLEL_SIZE).
• Profiles are generated per model in the form vllm-tp2-pp1-<hash>. NIM still auto-selects one based on the GPUs it detects.

What stays the same as per-model NIM:

• The OpenAI-compatible routes: /v1/chat/completions, /v1/completions, /v1/responses, /v1/models.
• The health endpoints /v1/health/live and /v1/health/ready.
• Prometheus metrics at /v1/metrics (NIM API reference).

The maintenance argument is the one I care about most. Ten models used to mean ten images to mirror into your private registry, scan, patch and upgrade. With model-free NIM you run one image per engine, with one CVE-patching surface and one upgrade path. That is also how NVIDIA pitches it: "One container image can pass security review and serve any supported model."

Licensing, read before production:

• NVIDIA's NIM FAQ says the free NVIDIA Developer Program covers self-hosted NIM "for research, application development, and experimentation on up to 16 GPUs" (NVIDIA NIM FAQ).
• For production, NVIDIA points you to NVIDIA AI Enterprise, which has a free 90-day trial (developer.nvidia.com/nim).
• The current docs split NIM into two offerings, and NIM Certified Production Branch "requires an active NVIDIA AI Enterprise subscription" (NIM offerings).
• Every container exposes GET /v1/license. Read it, and get your legal team to read it too.

Which open-weight models can you actually serve, and how do you check in 60 seconds?

To check whether the model-free NIM container can serve a model, open the checkpoint's config.json on Hugging Face and read the architectures field. Then confirm that the vLLM or SGLang version bundled in your NIM image supports that exact class name and the checkpoint's quantization format. NIM 2.1.5 bundles vLLM 0.31.0 and SGLang 0.5.21.

The 60-second check:

1. Fetch the config: curl -s https://huggingface.co/<org>/<model>/raw/main/config.json | jq .architectures.
2. Look for that string on NVIDIA's Model-Free NIM support matrix. If it is listed, use the exact image tag NVIDIA gives you.
3. If it is not listed, search the bundled engine version's model registry. For vLLM 0.31.0 that is registry.py at v0.31.0, or use the vLLM supported models and SGLang supported models pages.
4. Confirm the precision: BF16, FP8, MXFP4, NVFP4, GPTQ or AWQ. It must match a kernel your GPU generation has. NVFP4 needs Blackwell.
5. Check the context length (max_position_embeddings) and whether the repo ships a standard tokenizer and chat template.
6. Run list-model-profiles with the same NIM_MODEL_PATH you plan to deploy. If NIM lists no runnable profiles for your GPUs, stop there.

A real example from this week: Moonshot AI published Kimi K3.1 Pro on Hugging Face on November 2, 2026. Its architecture is KimiK31ForConditionalGeneration. That class is not in vLLM 0.31.0's registry, so model-free NIM 2.1.5 cannot serve it yet. vLLM shipped a day-0 image on November 4. Community engines usually get there first, because model-free NIM can only be as new as the engine it bundles. Use upstream vLLM or SGLang for that gap and move to NIM once NVIDIA ships a new tag.

The licence check is separate from the technical one. Apache 2.0 (gpt-oss, Qwen3-235B), MIT (DeepSeek) and bespoke licences (Kimi-K3, MiniMax-M3, both marked other on Hugging Face) differ on commercial use, attribution and redistribution. Check each model, not each family.

Gated repos: none of the checkpoints below are gated today. For those that are, Hugging Face says you must request access "from your browser" while logged in, and scripts must "provide a user token" (Hugging Face gated models). A 401 at download time almost always means the token's account never got access.

Pin the revision. NIM accepts hf://{org}/{model}:{revision} (NIM model download), and Hugging Face requires "the full-length hash" for commits (huggingface_hub docs). Write hf://openai/gpt-oss-120b:b5c939de8f754692c1647ca79fbf85e8c1e70f8a, not just the repo name. If the upstream repo is re-uploaded, your production model stays the same.

The chat-template trap: a mismatched chat template still returns HTTP 200, with quietly worse answers. Tool calls break, reasoning tags leak into the output, and the model ignores the system prompt. Use the template that ships with the checkpoint, and override it only through NIM's custom parsers and templates when the model card tells you to.

Parameter counts come from each model card. The Hugging Face file browser shows wrong totals for 4-bit checkpoints, because packed integer tensors get counted as single parameters.

How much GPU memory and what node shape do you need to self-host these models?

Estimate GPU memory as total parameters x bytes per parameter (2 at BF16, 1 at FP8, about 0.5 at FP4), plus the KV cache, plus 10-20% runtime overhead. For Mixture-of-Experts models, all experts must sit in GPU memory even though only a few run per token. That is why large MoE checkpoints need 8-GPU nodes or multi-node serving.

Step 1, weights: NVIDIA's memory guide uses the same arithmetic: Llama 3.3 70B at FP8 is "70 billion x 1 byte / 2 GPUs" = 35 GB per GPU (NIM memory troubleshooting).

Step 2, KV cache: 2 x layers x kv_heads x head_dim x bytes per element x context length x concurrent sequences.

• Qwen3-235B uses GQA with 94 layers, 4 KV heads and head_dim 128 (config.json).
• At BF16 that is about 192 KB per token: 6.3 GB for one 32K-token sequence, and 101 GB for 16 of them.
• GQA, MLA (DeepSeek, Kimi K2) and the hybrid linear-attention designs in Kimi-K3 and MiniMax-M3 shrink this a lot compared with full multi-head attention. Read the model card before you assume the worst case.

Step 3, overhead: add 10-20% for activations, CUDA graphs and communication buffers. If startup fails during KV allocation, NVIDIA's fix is to cut NIM_MAX_MODEL_LEN before you buy more GPUs.

The MoE rule: active parameters set compute cost, total parameters set memory cost. So how much VRAM does a 100B MoE model need? Budget for 100B, not for the 5-10B it activates. gpt-oss-120b activates 5.1B parameters, yet its card says it fits "into a single 80GB GPU (like NVIDIA H100 or AMD MI300X)" only because its MoE weights ship in MXFP4 (gpt-oss-120b model card).

Check your arithmetic against an official layout:

• Kimi-K3: 2.8T params x 0.5 byte is about 1.4 TB of weights before any KV cache. One 8x B200 node holds 1,440 GB in total (NVIDIA DGX B200), so NVIDIA's validated B200 layout uses two nodes, and B300 fits on one 8-GPU node (NIM support matrix).
• MiniMax-M3 NVFP4: about 250 GB of weights. NVIDIA validates it at TP4 on H200 (4 x 141 GB) and TP8 on H100 80GB. BF16 needs TP8 on B200.

Tensor parallelism has two hard constraints:

• TP must divide the GPUs you have (1, 2, 4, 8).
• TP must divide the attention-head count. vLLM refuses to start otherwise: "Total number of attention heads (...) must be divisible by tensor parallel size" (vLLM source).

MiniMax-M3 and gpt-oss have 64 heads, so TP=6 on a six-GPU box fails. Once you go past one GPU, NVLink/NVSwitch matters: PCIe-only nodes lose a lot of throughput on large models, and NVIDIA states that MiniMax-M3 is not supported in VMs with PCIe GPU passthrough in 2.1.4.

Per-GPU memory: L40S 48 GB (NVIDIA), H100 SXM 80 GB (NVIDIA), H200 141 GB (NVIDIA), B200 about 180 GB (1,440 GB per 8-GPU DGX B200). Weight columns are params x bytes and leave out KV cache.

Prerequisites that break most first attempts (NIM prerequisites):

• GPU driver 580 or later, CUDA 12.9 or later, Docker 24.0 or later.
• NVIDIA Container Toolkit 1.14.0 or later, configured with sudo nvidia-ctk runtime configure --runtime=docker (Container Toolkit install guide). On Kubernetes the GPU Operator installs it for you.
• --shm-size raised well above Docker's 64 MB default. NVIDIA calls that default "too small for multi-GPU vLLM" and asks for 16 GB on most MiniMax-M3 profiles.
• An HF_TOKEN. An NGC API key is only needed for Production Branch images, NIMs older than 2.0.10, or private artifacts.

Size the cache disk above the checkpoint. Kimi-K3 is about 1.6 TB per checkpoint, and NVIDIA warns the first download and load "can take several hours".

How do you run a model-free NIM container with Docker, step by step?

Five steps take you from a bare GPU host to an OpenAI-compatible endpoint: create a persistent cache directory, export HF_TOKEN, optionally log in to nvcr.io, run the model-free image with NIM_MODEL_PATH set, then poll /v1/health/ready and call /v1/chat/completions. Every command below is copied from the NIM 2.1.5 docs.

Step 1: create the weight cache. This is the most common mistake: without a mounted cache, every restart downloads the whole model again.

export LOCAL_NIM_CACHE=~/.cache/nim
mkdir -p "$LOCAL_NIM_CACHE"
# The NIM container runs as a non-root user with GID 0
sudo chgrp -R 0 "$LOCAL_NIM_CACHE"
sudo chmod -R g+rwX "$LOCAL_NIM_CACHE"

Use one cache directory per model and per image. NVIDIA recommends this for air-gapped redeploys too.

Step 2: export your Hugging Face token from a secret store, not a pasted literal in shell history:

export HF_TOKEN="$(cat ~/.secrets/hf_token)"

Step 3 (only if required): log in to nvcr.io. Public model-free images pull without a login. You only need it for Production Branch, gated or older images. The username is the literal string $oauthtoken:

echo "$NGC_API_KEY" | docker login nvcr.io --username '$oauthtoken' --password-stdin

Step 4: run the container. This serves gpt-oss-20b, pinned to a commit:

export NIM_LLM_MODEL_FREE_IMAGE=nvcr.io/nim/nvidia/vllm-model-free-nim:2.1.5
export NIM_MODEL_PATH=hf://openai/gpt-oss-20b:6cee5e81ee83917806bbde320786a8fb61efebee

docker run --rm --gpus=all \
  --shm-size=16GB \
  -u $(id -u):0 \
  -e NIM_MODEL_PATH \
  -e NIM_SERVED_MODEL_NAME="openai/gpt-oss-20b" \
  -e HF_TOKEN \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  -p 8000:8000 \
  ${NIM_LLM_MODEL_FREE_IMAGE}

Two settings people get wrong first:

• Tensor parallel size: set -e NIM_TENSOR_PARALLEL_SIZE=8, or pass --tensor-parallel-size 8 after the model path. Better still, run list-model-profiles first and set NIM_MODEL_PROFILE to an ID from its output.
• Served model name: without NIM_SERVED_MODEL_NAME, clients must send the derived name (the URI minus hf:// and the :revision). Set it explicitly so your clients keep the same model ID when you change precision or revision.

For MiniMax-M3 or Kimi-K3, use NVIDIA's tags (vllm-model-free-nim:2.1.4-minimax-m3, vllm-model-free-nim:2.1.4-kimi-k3) and the checkpoints listed in the support matrix.

Step 5: wait for ready, then call it.

until curl -sf http://localhost:8000/v1/health/ready > /dev/null; do sleep 10; done
curl -s http://localhost:8000/v1/models

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-oss-20b",
    "messages": [{"role": "user", "content": "Hello! How are you?"}],
    "max_tokens": 100
  }'

A trimmed response looks like {"object":"chat.completion","model":"openai/gpt-oss-20b","choices":[{"message":{"role":"assistant","content":"..."},"finish_reason":"stop"}],"usage":{...}}.

Cold start: the first boot downloads the weights, generates the manifest and selects a profile. Later starts read from the cache. For a 14 GB model that takes minutes. For a 1.6 TB checkpoint NVIDIA says hours.

Serving local weights instead: mount the directory and point NIM_MODEL_PATH at it, for example -v /mnt/models:/mnt/models -e NIM_MODEL_PATH=/mnt/models/my-120b-model. NIM then generates the manifest with "no network access or credentials", which is the air-gapped, regulated and low-bandwidth pattern (NIM air-gap deployment).

How do you run NIM on Kubernetes so it survives restarts, scales, and does not burn GPU hours idle?

To run NIM on Kubernetes, wrap the same container in a Deployment that requests nvidia.com/gpu in its limits and mounts a persistent volume at /opt/nim/.cache. Give it a startup probe on /v1/health/ready long enough to cover the first model download. Schedule it onto a tainted GPU node pool that scales to zero when nobody calls the model.

GPU scheduling: the cluster needs the vendor device plugin. After that it "exposes a custom schedulable resource such as nvidia.com/gpu", and GPUs "are only supposed to be specified in the limits section" (Kubernetes docs). The NVIDIA GPU Operator installs the driver, the container toolkit, the device plugin and DCGM monitoring in one go. Taint the GPU nodes so CPU pods never land on a $55/hour machine.

The weight cache is again the biggest practical problem. If every new pod downloads 250 GB, scale-up takes forever and you pay the egress each time. You have three options:

• A ReadWriteOnce PVC per replica.
• Node-local NVMe that you pre-warm.
• A ReadWriteMany volume shared across replicas. NVIDIA's minimal Helm values use persistence.accessMode: ReadWriteMany (NIM Helm deployment).

A minimal manifest for MiniMax-M3 NVFP4 on an 8x H100 node:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: minimax-m3
spec:
  replicas: 1
  selector:
    matchLabels: { app: minimax-m3 }
  template:
    metadata:
      labels: { app: minimax-m3 }
    spec:
      nodeSelector: { gpu-pool: h100 }
      tolerations:
        - { key: nvidia.com/gpu, operator: Exists, effect: NoSchedule }
      securityContext: { fsGroup: 0 }   # NIM writes its cache as GID 0
      containers:
        - name: nim
          image: nvcr.io/nim/nvidia/vllm-model-free-nim:2.1.4-minimax-m3
          env:
            - { name: NIM_MODEL_PATH, value: "hf://nvidia/MiniMax-M3-NVFP4" }
            - { name: NIM_SERVED_MODEL_NAME, value: "minimaxai/minimax-m3" }
            - name: HF_TOKEN
              valueFrom: { secretKeyRef: { name: hf-token, key: token } }
          ports: [{ containerPort: 8000 }]
          resources:
            limits: { nvidia.com/gpu: 8 }
          startupProbe:           # first boot downloads ~250 GB: allow up to 2 hours
            httpGet: { path: /v1/health/ready, port: 8000 }
            periodSeconds: 30
            failureThreshold: 240
          readinessProbe:
            httpGet: { path: /v1/health/ready, port: 8000 }
            periodSeconds: 10
          livenessProbe:
            httpGet: { path: /v1/health/live, port: 8000 }
            periodSeconds: 20
          volumeMounts:
            - { name: nim-cache, mountPath: /opt/nim/.cache }
            - { name: dshm, mountPath: /dev/shm }
      volumes:
        - name: nim-cache
          persistentVolumeClaim: { claimName: minimax-m3-cache }   # 400Gi, fast SSD
        - name: dshm
          emptyDir: { medium: Memory, sizeLimit: 16Gi }

Why startupProbe and not just a long initialDelaySeconds: with a short liveness check, the kubelet kills the pod mid-download, and you end up in a crash loop that never finishes. The startup probe holds liveness off until the model is ready.

Scaling and scale-to-zero:

• Run replicas behind a Service for throughput.
• Karpenter deletes "entirely empty nodes", and the Cluster Autoscaler can "scale a node group to 0".
• KEDA scales workloads "all the way to zero", though not on CPU or memory triggers. The KEDA HTTP add-on does request-driven scaling and is still beta.
• On dev and evaluation environments, scale-to-zero makes the biggest difference to your bill.

Observability: /v1/metrics passes through the engine's native Prometheus metrics with a vllm: or sglang: prefix (NIM observability). Set SLOs on time-to-first-token, inter-token latency and tokens per second per GPU. CPU utilization tells you nothing here. Add DCGM exporter for DCGM_FI_DEV_GPU_UTIL and DCGM_FI_DEV_FB_USED.

Is self-hosting an open-weight model actually cheaper than a hosted inference API?

Self-hosting is cheaper than a hosted API only when your GPUs stay busy. Effective $/M tokens = node hourly price / (sustained tokens per second x 0.0036). Compare that with the provider's published price per million tokens. An idle GPU node costs exactly as much as a saturated one, so self-hosting wins above a utilization threshold and loses badly below it.

The formula:

• Self-hosted $/M tokens = hourly price / (tokens per second x 0.0036). 3,600 seconds per hour, divided by 1,000,000 tokens.
• At utilization U, divide that result by U.
• Break-even utilization = (self-hosted $/M at 100%) / (API $/M).

A worked example, sourced, as of November 5, 2026:

• Throughput: SGLang's team measured 22,282 output tokens/s per decode node for DeepSeek-V3 on 8x H100 at 2K-token inputs. The setup used prefill-decode disaggregation and expert parallelism across 12 nodes (96 GPUs) (LMSYS). Treat it as an upper bound for a heavily tuned deployment.
• Node price: AWS p5.48xlarge (8x H100), us-east-1 on-demand, $55.04/hour (AWS Price List API).
• Result: $55.04 / (22,282 x 0.0036) = $0.69 per million output tokens at 100% utilization. Prefill nodes come on top.
• Same node elsewhere: Google Cloud a3-highgpu-8g in Iowa costs $88.49/hour (Google Cloud), or $1.10/M. Azure ND96isr_H100_v5 in East US costs $98.32/hour (Azure retail prices), or $1.23/M.
• Hosted APIs for the same architecture: DeepSeek V3.1 lists at $0.58 input / $1.68 output per M on Amazon Bedrock (Bedrock pricing) and $0.25 / $0.95 on OpenRouter (OpenRouter).
• Break-even on AWS on-demand: 0.69 / 1.68 = 41% sustained utilization against Bedrock, and 0.69 / 0.95 = 72% against OpenRouter.

Utilization is the number that matters most. At 10% utilization the same p5 node costs $6.86/M output tokens, four times Bedrock's price. Continuous batching, scale-to-zero and auto-stop on non-production move your effective price more than switching from H100 to H200.

Reasons to self-host that have nothing to do with price:

• Prompts, outputs and weights never leave your VPC.
• Data residency for regulated workloads, which is usually what decides it.
• Fine-tuned or proprietary checkpoints that no API will host.
• No third-party rate limits, more predictable tail latency, and no forced model deprecations.

Reasons not to self-host:

• You take on the operations: driver and CUDA drift, GPU quota approvals and regional capacity shortages.
• Bursty, low-volume traffic will never pay for a reserved node. For that workload, a hosted API is the right answer.

What model-free NIM changes: trying the newest open-weight model in production becomes a config change (NIM_MODEL_PATH) instead of a packaging project. That makes owning the serving layer more valuable.

Discounts belong to the account that buys them:

• AWS EC2 Instance Savings Plans go "up to 72%" (AWS). EC2 Capacity Blocks for ML reserve 1 to 64 GPU instances up to eight weeks ahead, for up to 182 days.
• Google Cloud's published A3 resource-based commitments price a3-highgpu-8g at $61.38/hour (1 year) and $38.86/hour (3 years), about 31% and 56% off on-demand.
• Each of these is tied to your account, organization or project.

How do you keep the GPU bill, the weights, and the prompt data inside your own cloud account?

Run the GPU nodes in your own cloud account. Your committed-use discounts, Savings Plans, GPU quotas and capacity reservations belong to that account, and a third-party platform cannot use them for you. Prompts and fine-tuned checkpoints are also the most sensitive things an AI team handles, so they should stay in your VPC.

Bring your own cloud (BYOC), applied to inference, means all of this stays in your name:

• The cloud bill.
• The capacity blocks and reservations.
• The GPU quotas.
• The VPC.
• The prompt logs.
• The model weights.

GPU commitments run for months or years and cost six or seven figures. A leaked prompt log or fine-tuned checkpoint is a far worse incident than a leaked static asset. I wrote more about the BYOC trade-offs here.

This is where Qovery fits. Qovery deploys and operates workloads inside your own AWS, GCP, Azure or Scaleway account, or on your existing Kubernetes cluster. You get Kubernetes without managing Kubernetes directly, and you ship without a dedicated DevOps engineer. What it gives you:

• Git-push deployments for the NIM container and the apps that call it.
• Preview environments per pull request, useful for A/B testing a prompt or model swap before it reaches production (how preview environments help teams).
• Environment auto-stop on a schedule, so the evaluation GPU node is off at 7 PM and back at 8 AM (deployment rules).
• Per-environment RBAC, so the person who can redeploy staging cannot touch production.
• Managed cluster upgrades, and databases backed by your cloud's managed services.

What the GPU pattern looks like:

• On Qovery-managed AWS EKS clusters, you enable a dedicated GPU node pool (Karpenter) and pick the instance types (Qovery EKS docs, GPU workloads on EKS).
• On GCP, Azure or Scaleway, you connect an existing cluster that already has GPU nodes (bring your own Kubernetes).
• The NIM container then runs as a Qovery container service with a GPU allocation and persistent storage mounted at /opt/nim/.cache. Qovery persistent storage cannot be shared between replicas, so give each replica its own cache.
• Auto-stop keeps the eval environment from running overnight.

Be fair to the alternatives:

• If you want zero infrastructure, use a hosted API.
• If you are running a one-off experiment, vLLM on a single rented GPU is enough.
• Qovery does not make GPUs cheaper. It runs them in your account, on your discounts, and switches them off when nobody uses them.
• For tuning tips on the rest of the cluster, see our Kubernetes cost optimization guide, and for the engine choice itself, vLLM vs TGI vs NIM.

How do I deploy this with Qovery from Claude Code or Cursor?

Install the Qovery Agent Skills with one curl command, then ask your coding agent in plain language to deploy the NIM container into your own cloud account. The agent proposes the service, the GPU allocation and the persistent volume, and you approve each step before anything runs.

curl -fsSL https://skill.qovery.com/install.sh | bash

# Or scope the install to the current project only
curl -fsSL https://skill.qovery.com/install.sh | bash -s -- --project

# Add the Qovery MCP Server to Claude Code over OAuth (read-only by default)
claude mcp add --transport http qovery https://mcp.qovery.com/mcp --callback-port 4242

# Write access, only when you explicitly want it (also enable it in the Qovery Console)
claude mcp add --transport http qovery "https://mcp.qovery.com/mcp?read_write=true" --callback-port 4242

The skills deploy new services, and the MCP Server operates what already exists:

Then a prompt like: "deploy the vllm-model-free-nim container to my AWS cluster's GPU node pool with NIM_MODEL_PATH=hf://openai/gpt-oss-120b, mount a 200 GB persistent volume at /opt/nim/.cache, and auto-stop the staging environment at night."

Full setup is in the AI agent quickstart and on the Claude Code integration page.