How to Self-Host NVIDIA Nemotron 3.5 Super VL 120B on Your Own Kubernetes Cluster

The model is the easy part. I can pull NVIDIA Nemotron 3.5 Super VL 120B off Hugging Face in one command. What breaks is everything underneath it: the GPU node pool that has to exist before the pod schedules, the driver stack that has to match the CUDA build in the container, and the bill that keeps running whether the GPUs serve a request or sit idle at 3am. I have run GPU workloads on Kubernetes long enough to know the weights are never what costs me a weekend.

So this is the hands-on version. How much VRAM a 120B vision-language model actually needs, which node type fits it, what the four-layer serving stack looks like, a Deployment manifest you can adapt, and the real break-even against a hosted API. One thing up front: Nemotron releases move fast, and where this article and the Hugging Face model card disagree, the model card wins. Read it before you touch a manifest.

Key Points:

• A 120B model at BF16 needs about 240 GB of VRAM for weights alone (2 bytes per parameter), before KV cache and the vision encoder. NVIDIA recommends 4x H100 or 2x GB200 for the BF16 build, and the NVFP4 build quarters the weight footprint to roughly 60 GB.
• This is a Mixture-of-Experts model: 120B total parameters, 12B active. Every expert still has to be resident in GPU memory, so VRAM is sized by the 120B, while throughput benefits from only 12B being active per token.
• The serving stack on Kubernetes is four layers: NVIDIA GPU Operator for drivers and the device plugin, a vLLM container as the engine, a Deployment requesting nvidia.com/gpu with --tensor-parallel-size equal to the GPU count, and an authenticated OpenAI-compatible endpoint on port 8000.
• Keep tensor parallelism inside one node. NVLink moves data between GPUs at 900 GB/s, while the best cloud node-to-node fabric is 3.2 Tbps, about 400 GB/s. Split a model across two nodes and tokens per second collapse.
• Self-hosting only beats per-token APIs above a sustained utilisation threshold. An 8x H100 node is $55.04/hour on-demand on AWS, roughly $40k/month at 24/7, so idle GPUs are the real cost risk, not the model.

What is NVIDIA Nemotron 3.5 Super VL 120B, and can you legally self-host it?

Nemotron 3.5 Super VL 120B is an open-weights multimodal model from NVIDIA that handles text, images and video, released on Hugging Face under the permissive OpenMDW-1.1 license, which allows commercial use and self-hosting with no field-of-use restriction. The "Super" tier sits between Nemotron Nano and Ultra in NVIDIA's size ladder, and the "VL" means there is a vision encoder in front of the language model that turns images and video frames into tokens, and those tokens eat context and VRAM like any others.

Architecturally it is a Nemotron Hybrid Mixture-of-Experts: 120B total parameters with 12B active, 512 routed experts per MoE layer, a C-RADIO vision encoder, and a context window of up to 1M tokens shared across the prompt, images, conversation history, tool results and the response. The MoE design matters operationally: all 120B parameters sit in GPU memory because any token can route to any expert, but only 12B are active per forward pass, so you get the throughput profile of a much smaller model with the memory bill of a large one.

Before you write a single manifest, read six fields off the model card: total and active parameter count, context window, supported precisions (the official builds are BF16 and NVFP4), the chat template, the max images and video per prompt (the card's example sets --limit-mm-per-prompt '{"image":1,"video":1}'), and NVIDIA's recommended GPU configuration. Those six numbers determine everything downstream.

On licensing, OpenMDW-1.1 is about as permissive as model licenses get. The Linux Foundation's OpenMDW grants you the right to use, copy, modify, distill and distribute the model commercially with no copyleft and no separate commercial license. The one real condition is that if you redistribute the weights, you include a copy of the license and the original notices. Outputs carry no restriction. I am not your lawyer, so read the license text, but there is nothing here that stops a company from running this in production.

Why self-host at all? Prompts and images that cannot leave your VPC (healthcare, defence, EU data residency), sustained high throughput, a fine-tuned or distilled variant you own, or fixed latency without a provider's rate limits. And the blunt counterpoint: if your traffic is bursty and low volume, a hosted endpoint is almost always cheaper, and you should just use one.

The reason this model earns the GPU bill is the eval sheet. The model card reports strong numbers for an open model, including GPQA-Diamond at 89.52, MMLU-ProX at 76.91, MathVision at 78.16 and SWE-bench Verified at 71.10. Treat the card's full table as the source of truth and re-check it against your own task before committing hardware.

Where Nemotron 3.5 Super VL sits among open VL models

Parameter counts and licenses come from each model's card. Verify the exact figures and license conditions on the card before you build a plan around any row here, because several of these change between point releases.

How much GPU memory does a 120B vision-language model need, and which node type fits?

A 120B vision-language model needs roughly 240 GB of VRAM for weights at BF16, which is why NVIDIA recommends 4x H100 (320 GB total) or 2x GB200 for the BF16 build, with the NVFP4 build dropping the weight footprint to roughly 60 GB. The arithmetic is simple enough to recompute for any model, so here it is on its own line.

VRAM = (params x bytes-per-param) + KV cache + vision encoder and image-token overhead + ~20% headroom

where KV cache = 2 x layers x kv-heads x head-dim x bytes x context-length x concurrency.

Worked BF16 example: 120B params x 2 bytes = 240 GB for weights. On 4x H100 (320 GB), that leaves about 80 GB for KV cache, the C-RADIO encoder and activations, which is why the card lands on four GPUs rather than two. Worked NVFP4 example: 120B x ~0.5 bytes = ~60 GB for weights, which fits comfortably on far less hardware and leaves room for a healthy KV cache. The one-line NVFP4 takeaway: expect roughly a quarter of the BF16 weight memory.

The image-token point is where VL models surprise people. A single high-resolution image does not cost one token; depending on the tiling scheme it expands into hundreds or thousands of tokens, so the KV cache for a multimodal prompt at a given context length is materially larger than for the same length of plain text. Size your KV cache for the worst-case image count you will actually send, and cap it with --limit-mm-per-prompt as the model card shows.

GPU node options for a 120B VL model

Prices move and vary by region, so I only quote the AWS p5 figure I verified and link each provider's live pricing for the rest. The point of the table is the shape, not a frozen number.

Here is the rule that decides node count: the model must fit on one node. NVLink and NVSwitch move data between GPUs inside a box at 900 GB/s per GPU, while the best cloud node-to-node fabric, 3.2 Tbps EFA on a p5, works out to about 400 GB/s aggregate per node. Tensor parallelism chatters between GPUs on every layer, so run it inside NVLink and keep pipeline parallelism as a last resort only if a model genuinely will not fit in one node. For this model at BF16, 4 to 8 GPUs in a single node is the whole story.

Two more practical notes. Quantization trades memory for a little accuracy: NVFP4 roughly quarters the footprint, but a text-benchmark accuracy delta does not automatically hold on image tasks, so re-evaluate NVFP4 on your own visual prompts before you trust it in production. And forget consumer cards: an RTX 4090 or 5090 has 24-32 GB and no NVLink, so you would be splitting a 240 GB model across PCIe at a fraction of NVLink bandwidth, which does not work.

On cold starts, the BF16 weights are around 240 GB of files. Pulling that from Hugging Face on every pod restart is minutes of dead time, so stage the weights once: a Hugging Face token as a Secret with HF_HOME on a persistent volume, or better, pre-pull the files into S3, GCS or Azure Blob and mount them read-only so a restart is seconds, not a download.

What does the Kubernetes serving stack for Nemotron actually look like?

The serving stack for Nemotron 3.5 Super VL on Kubernetes is four layers in order: the NVIDIA GPU Operator for drivers and the device plugin, a vLLM container as the inference engine, a Deployment that requests nvidia.com/gpu with tensor parallelism set to the GPU count, and a Service plus authenticated ingress exposing an OpenAI-compatible API on port 8000.

Layer 1, NVIDIA GPU Operator. It installs the driver, container toolkit, device plugin, DCGM exporter and node feature discovery as a set of DaemonSets. The one trap: EKS, GKE and AKS GPU-optimised node images often already ship the driver and device plugin, and running both the managed components and the Operator causes driver conflicts. Check the GPU Operator platform support matrix and disable whichever layer you are not using.

Layer 2, the inference engine. vLLM is my default here: it is open source, OpenAI-compatible, has the fastest support for new architectures, and the Nemotron model card itself documents vllm serve plus an SGLang path. NVIDIA NIM is the vendor-supported container with pre-built TensorRT-LLM engines, which is the right pick when you already pay for NVIDIA AI Enterprise and want a support contract for production. TensorRT-LLM gives the best raw throughput if you will invest in engine builds; SGLang is excellent and card-documented; TGI is fine inside the Hugging Face stack; Ollama is for a laptop, not a 120B production endpoint.

Layer 3, the workload. Use a Deployment, not a DaemonSet: one pod per node, resources.limits['nvidia.com/gpu'] set to your GPU count, --tensor-parallel-size matching it, a readiness probe on /health, and a long startupProbe because weights take minutes to load. Raise /dev/shm: the Kubernetes default emptyDir is small and NCCL will hang on multi-GPU without a larger shared-memory volume.

Layer 4, exposure. A ClusterIP Service plus ingress, with raised proxy read and send timeouts so streaming responses are not cut off, and an auth proxy or API gateway in front. Never expose an inference endpoint unauthenticated: an open endpoint is free compute for whoever finds it and a path to exfiltrate whatever prompts flow through it.

Two operational habits that save money and sleep. Taint the GPU nodes and add tolerations so no CPU workload ever lands on a $55/hour box, use a nodeSelector on the accelerator label, and set a PodDisruptionBudget so a cluster upgrade does not evict your only warm replica. And autoscale on the right signals: vLLM exposes Prometheus metrics like vllm:num_requests_waiting, vllm:time_to_first_token_seconds and vllm:gpu_cache_usage_perc, plus DCGM for GPU utilisation and framebuffer memory. Scale on queue depth and KV cache usage, never on CPU, which is meaningless for GPU inference.

Inference engine comparison for Kubernetes

How do you deploy Nemotron 3.5 Super VL on Kubernetes step by step?

Deploying Nemotron on Kubernetes is five ordered steps: create a GPU node pool that scales to zero, install the NVIDIA GPU Operator, stage the weights on persistent storage, deploy the vLLM pod with tensor parallelism matching the GPU count, then smoke-test the endpoint with a text prompt and an image prompt.

Step 1. Create a dedicated GPU node pool with the right instance type, taints applied, and min 0 / max N so it scales to zero when nothing is scheduled. Label it with the accelerator type you will target with nodeSelector.

Step 2. Install the GPU Operator with Helm, then verify: kubectl describe node should show nvidia.com/gpu capacity, and the device-plugin pod logs should be clean. Follow the official install docs rather than inventing chart values, and remember the EKS/GKE/AKS double-install warning from above.

Step 3. Stage the ~240 GB of weights. Put a Hugging Face token in a Secret and point HF_HOME at a persistent volume, or pre-pull the files to object storage and mount them read-only so restarts take seconds.

Step 4. The Deployment. Pin the image tag, because a brand-new architecture usually needs a very recent vLLM release, and check vLLM's supported-models list for the minimum version that lists Nemotron VL.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nemotron-vl
spec:
  replicas: 1
  selector: { matchLabels: { app: nemotron-vl } }
  template:
    metadata: { labels: { app: nemotron-vl } }
    spec:
      nodeSelector: { nvidia.com/gpu.product: NVIDIA-H100-80GB-HBM3 }
      tolerations:
        - { key: nvidia.com/gpu, operator: Exists, effect: NoSchedule }
      containers:
        - name: vllm
          image: vllm/vllm-openai:vX.Y.Z   # pin a real, recent tag
          args:
            - "--model=nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16"
            - "--tensor-parallel-size=4"
            - "--max-model-len=32768"
            - "--gpu-memory-utilization=0.92"
            - "--limit-mm-per-prompt={\"image\":1,\"video\":1}"
          ports: [{ containerPort: 8000 }]
          resources: { limits: { nvidia.com/gpu: 4 } }
          env:
            - name: HF_TOKEN
              valueFrom: { secretKeyRef: { name: hf-token, key: token } }
          volumeMounts:
            - { name: dshm, mountPath: /dev/shm }
          readinessProbe: { httpGet: { path: /health, port: 8000 }, periodSeconds: 10 }
          startupProbe:
            httpGet: { path: /health, port: 8000 }
            failureThreshold: 60
            periodSeconds: 30   # allow ~30 min for weight load
      volumes:
        - name: dshm
          emptyDir: { medium: Memory, sizeLimit: 16Gi }

Step 5. Validate both paths.

curl -s http://nemotron-vl:8000/v1/models | jq '.data[].id'

# Text + image in one request (prove the vision path works)
curl -s http://nemotron-vl:8000/v1/chat/completions \
  -H "Content-Type: application/json" -H "Authorization: Bearer $KEY" \
  -d '{"model":"nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16",
       "messages":[{"role":"user","content":[
         {"type":"text","text":"What is in this image?"},
         {"type":"image_url","image_url":{"url":"data:image/png;base64,'"$B64"'"}}]}]}'

The failures you will actually hit, and the fix for each: CUDA OOM on KV cache means lower --max-model-len or --gpu-memory-utilization; a driver/CUDA mismatch means your container is newer than the node driver, so align them; "architecture not supported" means the vLLM build is too old, so bump to the version that lists Nemotron VL; a pod killed mid-load means the startupProbe was too short; an NCCL hang means /dev/shm is still the tiny default; and image-pull timeouts on a 20 GB+ container mean you need to pre-pull or warm the node image.

Is it cheaper to self-host Nemotron than to call a hosted API?

Self-hosting Nemotron is cheaper than a hosted API only above a sustained utilisation threshold, because an 8x H100 node at $55.04/hour on-demand costs about $40,000/month running 24/7 whether it serves one request or a million. The break-even depends almost entirely on how busy the GPUs are, not on the model. Here is the cost model on its own line.

effective $/1M tokens = (node $/hour x hours) / tokens served x 1,000,000

The node cost is fixed, so the only lever on the left is the denominator: tokens served, which is utilisation. At 100% utilisation that $40k/month spreads across a large token volume and the effective rate can beat a hosted per-token price. At 50% it doubles. At 10%, which is close to the 8% average CPU utilisation CAST AI measures across real clusters, your effective cost is roughly ten times the busy case and you would have been far better off on an API. I am deliberately not inventing a hosted per-token figure here; check the live price on build.nvidia.com or a provider like OpenRouter, Together AI or Fireworks AI and plug it into the formula with your own token volume.

Because this model is a 12B-active MoE, its throughput per GPU is better than a 120B dense model, which pushes the break-even point lower than you might expect. That is an argument for benchmarking your own tokens/sec on the node before deciding, not for trusting a generic number.

The levers that move the bill, with real discounts: AWS Savings Plans cut up to 72% versus on-demand for a commitment, GCP committed-use discounts reach up to 55% (more for memory-optimised), and spot or preemptible GPUs can be 60-91% off for batch and eval work that tolerates interruption. Scale-to-zero node pools and NVFP4 (fewer GPUs) stack on top.

Self-hosted vs hosted vs GPU-as-a-service

Here is why bring-your-own-cloud matters specifically for GPUs: when the instances run in your own account, your negotiated committed-use rate applies to these exact machines. A vendor reselling GPU hours charges a markup on the same silicon. And the hidden costs people forget are real: cross-AZ and egress traffic, object storage for 240 GB per model variant, idle dev and staging GPUs, and engineer hours on driver and CUDA upgrades.

To be fair about it, Modal, RunPod, Baseten, Together AI, Fireworks AI, Amazon SageMaker and Google Vertex AI are good answers when you do not want to own a cluster. Owning the cluster wins when you need data residency, your own committed-use rate, fine-tuned weights, no per-token markup, and no surprise model deprecation. If none of those apply, use a hosted endpoint and move on.

How do you keep GPU costs and cluster operations under control once it is running?

Treat the GPU node pool as the expensive object, not the model: scale it to zero when idle, auto-stop non-production environments, autoscale replicas on queue depth rather than CPU, and plan the coupled driver, CUDA, vLLM and Kubernetes upgrade path before it breaks. A staging GPU environment left running nights and weekends burns roughly two thirds of its monthly hours serving nobody, and CAST AI's measurement of 8% average utilisation and 69% overprovisioning across real clusters is the whole argument for scale-to-zero in one statistic.

Keep one warm replica in production for latency and autoscale additional replicas on vllm:num_requests_waiting or vllm:time_to_first_token_seconds. GPU utilisation alone lies when requests are queueing behind a busy pod. Upgrades are coupled: a Kubernetes bump that moves the node kernel can break the GPU driver, and a vLLM bump can require newer CUDA, so validate on a second node pool and cut traffic over rather than upgrading in place. And gate the expensive node pool with RBAC so only the right people can schedule onto it, with an approval step on production.

This is the part where I mention Qovery, and only here. Qovery provisions and manages the cluster and the GPU node pools, handles managed cluster upgrades, and gives your team git-push deploys, preview environments, per-environment RBAC and environment auto-stop for non-production GPU environments, so engineers self-serve inference environments without a platform engineer hand-writing every manifest. It runs on your own AWS, GCP, Azure or Scaleway account, or on your existing Kubernetes cluster, so nothing here is AWS-only. The GPU instances run under your cloud account, which means your committed-use discounts and your data-residency guarantees apply unchanged. That is the BYOC point made concrete: the bill and the discount stay in your name.

How do I deploy Nemotron on my own cluster from Claude Code or Cursor?

Install the Qovery agent skills once and your coding agent can create the environment, deploy the vLLM container and read back logs on your own cloud account, with you approving each action before it runs. It works with Claude Code, Cursor, Codex, OpenCode and Gemini CLI.

curl -fsSL https://skill.qovery.com/install.sh | bash

# Or project-scoped
curl -fsSL https://skill.qovery.com/install.sh | bash -s -- --project

# Add the MCP Server to Claude Code (OAuth, read-only by default)
claude mcp add --transport http qovery https://mcp.qovery.com/mcp --callback-port 4242

# Enable write access only when you want the agent to change things
# (also requires write access enabled in the Qovery Console)
claude mcp add --transport http qovery "https://mcp.qovery.com/mcp?read_write=true" --callback-port 4242

The skills (qovery-onboard, qovery-deploy, qovery-troubleshoot, qovery-optimize, qovery-speedup, qovery-preview, qovery-terraform) handle deployment, and the MCP Server operates what already exists. Then ask in plain language: "deploy this vLLM container serving Nemotron to my GPU node pool on GCP with Qovery." The agent proposes the steps and asks you to approve each one. The full walkthrough is in the AI agent quickstart and the Claude Code integration page.