How to Self-Host Nvidia Nemotron 3.5 Bolt on a Single Cloud GPU (and What It Actually Costs)

Key Points:

• Nemotron 3.5 Bolt is NVIDIA's local-agent tier of Nemotron 3.5, and the 74B-A9B "24GB" checkpoint runs on one GPU. It is a 74B-total, 9B-active hybrid Mamba-Transformer MoE compressed from Nemotron 3.5 Super VL, quantized to 2 bits, with up to 256K context, under the OpenMDW-1.1 license. A smaller Bolt E4B (8B, 4B effective) targets voice and edge assistants.
• Pick the GPU by runtime. NVIDIA's llama.cpp recipes need a 32GB+ or 48GB+ Ada or Blackwell card, so an L40S 48GB works. NVIDIA's vLLM recipe lists Blackwell SM120 and SM121, which in the cloud means an RTX PRO 6000 Blackwell 96GB instance. 24GB cards (L4, A10G) only fit Bolt E4B.
• vLLM is the production default. It has an official Bolt recipe, continuous batching and an OpenAI-compatible API. NVIDIA NIM ships a container for Bolt E4B only. SGLang and TensorRT-LLM do not list Bolt 74B yet.
• On unit price alone, a hosted API usually wins. The parent model, Nemotron 3.5 Super VL, costs $0.15 per million input tokens and $0.50 per million output tokens on Together AI and DeepInfra. One RTX PRO 6000 instance running 24/7 needs about 10 billion blended tokens a month to break even. Self-host for data control, no rate limits and predictable spend, not for a cheaper token.
• Idle hours are the biggest line item. A GPU used 8 hours on weekdays is busy for about 174 of 730 monthly hours, so 76% of a 24/7 bill buys nothing. Auto-stopping non-production GPU environments in your own AWS, GCP, Azure or Scaleway account, or your existing Kubernetes cluster, saves more than any quantization trick.

An agent that makes 30 model calls per task turns a per-token bill into a number you cannot forecast. One user story ends in 3 calls, the next loops through 60 tool results and re-reads a 100K-token context each time. Finance asks for next quarter's AI budget and engineering answers with a shrug. That is usually the moment a team starts eyeing a single GPU they control.

NVIDIA built Nemotron 3.5 Bolt for exactly that scenario: an agentic model sized to fit one GPU. I went through the model cards, the technical report and four clouds' price lists so you can decide whether to run it yourself, on which GPU, and what it will really cost.

What is Nvidia Nemotron 3.5 Bolt, and why does it fit on a single GPU?

Nemotron 3.5 Bolt is NVIDIA's open-weights model family for running everyday agentic tasks on a single device. The flagship, Bolt 74B-A9B 24GB, is compressed from Nemotron 3.5 Super VL and quantized to 2 bits so that the whole deployment fits in about 24GB of memory. That makes self-hosting it a one-instance decision rather than a cluster project.

NVIDIA publishes the weights on Hugging Face under the OpenMDW-1.1 license, which allows commercial use. The family currently has two models:

• Bolt 74B-A9B 24GB: 74B total parameters with 9B active per token. It is a Mamba-2/Transformer hybrid LatentMoE with a C-RADIO v4-H vision encoder, takes text, images and video as input, and supports up to 256K tokens of context. It ships as a vLLM checkpoint, a GGUF build for llama.cpp, and a BF16 checkpoint that NVIDIA says is for customization and post-training, not production inference.
• Bolt E4B 4GB: an 8B-parameter (4B effective) text model post-trained from Gemma 4 E4B, with 64K input context. It targets voice assistants, personas and edge devices. If you need a small, fast tool-caller on a cheap GPU, this is the one.

The rest of this guide focuses on the 74B, because that is the one teams consider for real agent workloads.

Why Bolt targets agentic work

An agent's quality ceiling comes from the model, but its wall-clock time comes from tokens per second multiplied by dozens of calls. NVIDIA designed Bolt 74B to fit 24GB and decode at least 80 tokens per second, and reports 84 TPS on an RTX Spark laptop running llama.cpp (NVIDIA Research). NVIDIA also trained it with end-to-end RL on MCP and software-engineering environments to cut task completion time on tool-heavy multi-turn work.

The trade-off is visible in NVIDIA's own numbers. On the same laptop, Bolt is more accurate than Qwen3.6 35B-A3B and Gemma 4 26B-A4B on MCP-Atlas (52.2 vs 51.2 and 48.4) and BrowseComp (66.75 vs 48.74 and 26.30). It is also slower: 600.8 seconds on NVIDIA's agent replay benchmark versus 454.0 for Qwen and 480.7 for Gemma (NVIDIA Research). You buy accuracy per GB, not raw speed.

What "fits on a single GPU" means

GPU memory has to hold the weights plus a KV cache for every concurrent session. Rough weight sizes are 2 bytes per parameter at BF16, 1 at FP8 and about 0.5 at 4-bit. Bolt 74B's BF16 checkpoint is 148GB, which no 80GB GPU can hold. The 2-bit build brings the weights down to 22.01GB, with 0.87GB of cache and activations at batch size 1 and 128K context (technical report). Every additional concurrent session needs more KV cache on top of that.

How Bolt compares to other self-hostable agent models

Every cell comes from the linked model card as of October 25, 2026. Where a card does not state a figure, I wrote "see model card" instead of guessing. Pick Bolt when agentic accuracy per GB matters most. Pick gpt-oss or Qwen when you need the more mature serving ecosystem today.

Which cloud GPU instance should I pick to run Nemotron 3.5 Bolt cost-efficiently?

For Bolt 74B, an L40S 48GB running the llama.cpp GGUF build is the cheapest documented single-GPU option. An RTX PRO 6000 Blackwell 96GB is the best production pick, because it matches the Blackwell SM120 target of NVIDIA's vLLM recipe and leaves room to batch. 24GB cards like the L4 and A10G only make sense for Bolt E4B.

Map the GPU to the runtime:

• L4 24GB, A10G/A10 24GB: too small for NVIDIA's Bolt 74B recipes. The smallest one (28GB, 64K context) asks for a 32GB+ discrete GPU. Fine for Bolt E4B, and NVIDIA's own hosted preview of E4B runs on an L40S.
• L40S 48GB (Ada): fits NVIDIA's 32GB llama.cpp recipe at the full 262,144-token context with one concurrent sequence. The GGUF card lists Ada Lovelace and Blackwell as supported.
• RTX PRO 6000 Blackwell 96GB (SM120): the cleanest match for the vLLM checkpoint, whose card lists "NVIDIA Blackwell SM120 & SM121" as supported microarchitectures. With 96GB you can raise concurrency well past one sequence.
• A100 80GB, H100 80GB: the vLLM card's summary lists Ampere and Hopper, but its software-integration section lists only Blackwell SM120/SM121. Test on your exact GPU before you reserve capacity for it.

Use the same sizing rule for any checkpoint: weight bytes, plus KV cache that grows with context length times concurrent sequences, plus 10 to 15% headroom. Bolt's Mamba layers keep a fixed-size state instead of a growing KV cache. That is why NVIDIA's whole runtime cache at 128K context is under 1GB.

Single-GPU instances across AWS, GCP, Azure and Scaleway

Prices are on-demand Linux, checked on October 25, 2026. Monthly = hourly x 730 hours, the convention used by the AWS Pricing Calculator and Azure.

AWS has no single-GPU A100 instance (p4d and p4de come with 8 GPUs), and Scaleway does not offer the RTX PRO 6000. GCP bundles the GPU into the machine price for G2, G4, A2 and A3. Check regional capacity and GPU quota before you fall in love with a price, because a cheap SKU you cannot get is worth nothing.

Spot and preemptible GPUs cut the hourly rate by up to 90% on AWS and up to 91% on GCP. The catch is the warning time. AWS gives two minutes, while GCP and Azure give 30 seconds or less on a best-effort basis. That is enough for batch evals and stateless workers behind a retry queue. It is not enough for a user-facing agent endpoint that takes minutes to load 22GB of weights.

What is the fastest way to serve Nemotron 3.5 Bolt - vLLM, Nvidia NIM, TensorRT-LLM, SGLang, or Ollama?

Use vLLM for production Bolt 74B serving. NVIDIA publishes the official vLLM recipe, and vLLM gives you continuous batching and an OpenAI-compatible endpoint on one GPU. Use llama.cpp for the cheapest single-stream setup on an L40S. Use NVIDIA NIM only for Bolt E4B, because that is the only Bolt NIM container today.

The batching argument is old but still holds. The original vLLM release measured up to 24x higher throughput than Hugging Face Transformers, and found existing systems wasted 60 to 80% of KV cache memory, against under 4% with PagedAttention (vLLM blog). The peer-reviewed paper reports 2 to 4x over FasterTransformer and Orca at the same latency (Kwon et al.).

Bolt's 74B checkpoint is already quantized by NVIDIA with quantization-aware distillation, so do not stack your own quantization on it. If you quantize Bolt E4B BF16 or the 74B BF16 yourself to FP8 or 4-bit, re-run a tool-calling eval before shipping. Quantization damage shows up first in malformed JSON arguments, not in chat quality.

Whatever you pick, keep the endpoint OpenAI-compatible. The OpenAI SDK, LangGraph, CrewAI, LlamaIndex and most coding agents then work with a base_url change and nothing else.

How do I deploy Nemotron 3.5 Bolt on a single GPU instance, step by step?

Five steps get you a production endpoint: provision a GPU node with a pinned driver and the NVIDIA Container Toolkit, cache the weights on a persistent volume, start vLLM with explicit memory and concurrency flags, put auth and TLS in front, then validate a real tool call.

Step 1 - Provision the GPU and the driver stack. Install the NVIDIA Container Toolkit on a VM, or the GPU Operator on Kubernetes. Pin the driver and CUDA versions. A driver older than the CUDA build inside the container is the classic cause of a container that starts fine and then sees zero GPUs.

Step 2 - Cache the weights. Mount a persistent disk or shared filesystem at the Hugging Face cache path. Every restart that re-downloads 22GB costs minutes of GPU time and egress.

Step 3 - Run the vLLM server. This follows NVIDIA's recipe, pins the same nightly image, and changes context and concurrency for a 96GB card:

docker run -d --name bolt --gpus all --ipc=host \
  -p 127.0.0.1:8000:8000 \
  -e VLLM_API_KEY -e VLLM_USE_V2_MODEL_RUNNER=1 \
  -v /mnt/models:/root/.cache/huggingface \
  --entrypoint /bin/bash \
  vllm/vllm-openai:nightly-77154ca6116ae34cbfe314beef1016bde9b63989 -c '
  pip install b12x && vllm download-kernels &&
  wget -q https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB/resolve/main/ultra_v3_reasoning_parser.py &&
  vllm serve nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB \
    --trust-remote-code \
    --max-model-len 65536 \
    --max-num-seqs 8 \
    --gpu-memory-utilization 0.90 \
    --kv-cache-dtype fp8 \
    --linear-backend b12x --moe-backend b12x \
    --enable-auto-tool-choice --tool-call-parser qwen3_coder \
    --reasoning-parser-plugin ultra_v3_reasoning_parser.py \
    --reasoning-parser ultra_v3'

• --max-model-len 65536 caps context at 64K instead of the card's 262,144, which frees memory for more sessions.
• --max-num-seqs 8 allows 8 concurrent sequences. NVIDIA's recipe uses 1, so measure memory and latency as you raise it.
• --gpu-memory-utilization 0.90 is the fraction of VRAM vLLM may claim. The default is 0.92 (vLLM docs).
• --kv-cache-dtype fp8 halves KV cache size, as in NVIDIA's recipe.
• The b12x backends load the 2-bit expert weights. The tool-call and reasoning parsers make function calling and thinking mode work.

Copy the attention and Mamba backend flags from the model card too, and drop the vision flags if you only send text.

Step 4 - Secure it. vLLM checks the VLLM_API_KEY bearer token. Put TLS termination in front with a reverse proxy or load balancer, and use /health as the readiness probe. Port 8000 binds to localhost above on purpose, so never open it to the internet.

Step 5 - Validate a tool call. Agents fail silently when tool calling is miswired, so test it explicitly:

curl -s https://bolt.internal.example.com/v1/chat/completions \
  -H "Authorization: Bearer $VLLM_API_KEY" -H "Content-Type: application/json" \
  -d '{
    "model": "nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB",
    "messages": [{"role": "user", "content": "How many keyboards are in stock?"}],
    "tools": [{"type": "function", "function": {
      "name": "get_inventory",
      "parameters": {"type": "object",
        "properties": {"product": {"type": "string"}}, "required": ["product"]}}}],
    "chat_template_kwargs": {"enable_thinking": false}
  }' | jq '.choices[0].message.tool_calls'

You want a get_inventory call with {"product": "keyboard"}. If you get prose instead, check the parser flags.

Day two: scrape vLLM's Prometheus metrics for KV cache usage, waiting requests, time to first token and generation throughput, alongside GPU utilization from DCGM. Set a restart policy, and add a warmup request after model load so the first real user does not pay for compilation.

When is self-hosting Nemotron actually cheaper than a per-token API?

Self-hosting is cheaper only when your sustained volume pushes the GPU's cost per million tokens below the API price. With today's hosted prices for Nemotron models, that takes billions of tokens a month on a busy GPU. For spiky, low-volume or prototype traffic, a hosted API from Together AI, DeepInfra, Baseten or NVIDIA is cheaper and far less work.

The break-even formula is simple:

Compare that number to the API's blended price per million tokens.

No provider I checked hosts Bolt 74B, so I compared against its parent, Nemotron 3.5 Super VL. It costs $0.15 per million input tokens and $0.50 per million output tokens on Together AI and DeepInfra (checked October 25, 2026). Agents read far more than they write, so I assume 3 input tokens per output token, which gives a blended price of $0.2375 per million.

Now the GPU. An AWS g7e.2xlarge in us-east-1 costs $3.363 an hour, or $2,455 for a 24/7 month. Divide $2,455 by $0.2375 and you get a break-even of about 10.3 billion blended tokens a month. Spread over 730 hours, that is about 3,900 tokens per second, every second. Even the cheaper L40S needs about 5.7 billion tokens a month.

Single-stream speed cannot get you there. If I assume NVIDIA's 84 tokens per second from the RTX Spark benchmark (a laptop, not a cloud GPU, so treat it as an order of magnitude), one stream produces about 221 million output tokens a month. That puts output at roughly $11 per million on the g7e, 22x the hosted price. Batching is the only lever that closes the gap, which is why --max-num-seqs matters more than the GPU you pick. Measure your aggregate tokens per second under real agent traffic before you commit.

So why does anyone self-host? Price is rarely the reason. Teams self-host because prompts and tool results never leave their VPC, there are no rate limits on an agent burst, latency stays predictable, and a looping agent cannot produce a surprise bill. A fixed $2,455 is easier to forecast than a meter.

The second argument is commitment. AWS Savings Plans, GCP committed use discounts and Azure reservations only apply to GPUs running in your own cloud account. If your company already committed spend to a hyperscaler, inference inside that account burns down a commitment you are paying for anyway.

The idle trap is the third number to watch. 8 hours a day on weekdays is about 174 of 730 hours, so 76% of a 24/7 bill buys an idle GPU. Cast AI's 2026 analysis of production Kubernetes clusters found GPU utilization averaged just 5% (Cast AI).

How do I keep the GPU bill down once Nemotron is running?

Three controls cut a self-hosted GPU bill more than anything else: stop non-production GPU environments when nobody uses them, right-size context and concurrency, and keep one well-fed GPU instead of several half-idle ones.

• Auto-stop non-production environments on a schedule or after inactivity. A staging GPU stopped outside working hours drops from $2,455 to about $585 a month on the g7e.
• Right-size --max-model-len. vLLM's docs list capping context length and max_num_seqs as the main levers to reduce memory (vLLM). Reserving 256K context you never use shrinks your achievable batch.
• Drop a GPU class when you can. If Bolt E4B passes your tool-calling eval, it runs on a $0.80-an-hour L4 instead of a $3.36 Blackwell card.
• Queue before you scale out. Watch vLLM's waiting-requests metric. A second GPU is only justified when the queue grows while the first one is already batching at full memory.
• Decide scale-to-zero per environment. Loading 22GB of weights from disk into VRAM, plus kernel warmup, is the real cost of scaling to zero. Measure it on your checkpoint. Then scale preview environments to zero and keep production warm.
• Tag GPU spend by environment and team, so every idle instance has an owner who gets the bill.

Should I run Nemotron on a raw VM or on Kubernetes?

A raw VM is the fastest route to a working endpoint and the worst way to run inference for a team. Kubernetes with the NVIDIA GPU Operator gives you GPU scheduling, rolling model updates, per-environment isolation and auto-stop. The price is someone operating the cluster, the drivers and the autoscaler.

A VM with Docker and systemd is legitimate for one always-on endpoint with nobody else touching it. It breaks down the day you need staging, a second model version, secrets rotation or access control for five engineers.

Kubernetes solves those problems with GPU node pools, taints and tolerations, autoscaling on queue-depth metrics, and preview environments per pull request for prompt and agent changes. It also creates a recurring workload: cluster upgrades, driver and CUDA compatibility, and GPU autoscaler tuning. That is a real DevOps job.

This is where Qovery fits. Qovery deploys and operates the cluster and the GPU workload inside your own AWS, GCP, Azure or Scaleway account, or on your existing Kubernetes cluster. You get git-push deploys, preview environments per pull request, environment auto-stop for expensive non-production GPUs, managed cluster upgrades and per-environment RBAC. A small team can self-host Nemotron without a dedicated DevOps engineer, and the GPU reservations and committed spend stay on their own cloud bill. Our Super VL guide covers the multi-GPU version of this setup.

Be fair to the alternatives. If you do not want the GPU in your own account at all, Modal, RunPod, Baseten, Amazon SageMaker endpoints and Google Vertex AI are good answers.

How do I do this with Qovery from Claude Code or Cursor?

Install the Qovery Agent Skills once, then ask your coding agent in plain language to deploy the inference service into your own cloud account. The agent runs the deployment steps and asks you to approve each action before it touches infrastructure.

Install the skills globally for Claude Code, Cursor, Codex, OpenCode or Gemini CLI. This installs qovery-onboard, qovery-deploy, qovery-troubleshoot, qovery-optimize, qovery-speedup, qovery-preview and qovery-terraform:

curl -fsSL https://skill.qovery.com/install.sh | bash

Or scope the skills to a single repository:

curl -fsSL https://skill.qovery.com/install.sh | bash -s -- --project

Add the MCP Server to Claude Code for read-only operations on what already exists. It logs you in with OAuth in the browser:

claude mcp add --transport http qovery https://mcp.qovery.com/mcp --callback-port 4242

Enable write access only when you explicitly want the agent to change infrastructure. This also requires enabling write access in the Qovery Console:

claude mcp add --transport http qovery "https://mcp.qovery.com/mcp?read_write=true" --callback-port 4242

Then ask: "deploy this vLLM service to my GCP account on a G4 node pool and auto-stop the staging environment at night." Setup details are in the AI agent quickstart and on the Claude Code integration page.