vLLM vs TGI vs NVIDIA NIM: which self-hosted LLM inference server actually cuts your GPU bill in 2026?

The inference server you pick changes your cost per million tokens by tens of percent. How you run it changes the bill by multiples. I have watched enough teams pour money into GPUs to know which of those two numbers actually shows up on the invoice, and it is usually the second one.

So here is the short version before the detail. Pick vLLM as your default in 2026. Pick NVIDIA NIM when you already pay for NVIDIA AI Enterprise and want a support contract. Pick Hugging Face TGI if your team already lives in the Hugging Face stack. Then spend most of your energy on utilization, right-sizing and autoscaling, because that is where the real money is.

Key Points:

• vLLM is the right default. It is Apache 2.0, OpenAI-compatible, and ships continuous batching, PagedAttention, FP8/INT4 quantization and prefix caching. It runs on NVIDIA, AMD ROCm, Intel, AWS Neuron and TPU, so you keep the option to move to whichever GPU is cheapest in your account.
• NVIDIA NIM buys support, not a lower GPU bill. It is free for development under the NVIDIA Developer Program, but production use requires a paid NVIDIA AI Enterprise license, and it is NVIDIA-only.
• TGI is still production-grade and the pragmatic pick inside the Hugging Face ecosystem, though feature velocity on the newest architectures has shifted toward vLLM and SGLang.
• No server is universally fastest. Model size, sequence length, concurrency and GPU generation reorder the ranking, so benchmark your own traffic at your own latency SLO and treat every vendor number as vendor-reported.
• Operations dwarf the engine choice. A GPU instance costs the same per hour at 10% and 90% utilization, so idle non-production GPUs, oversized instance types and missing autoscaling are usually the bigger line item.

What actually determines the GPU cost of a self-hosted inference server?

Cost per million tokens is your GPU instance price per hour divided by the tokens per hour you can sustain at your latency target. The inference server only moves the second term, which is why I rank servers on throughput per GPU-hour at a fixed latency SLO, not on peak tokens per second in a screenshot.

Here is the formula, stated so you can lift it directly:

cost per 1M tokens = (instance $/hour) / (sustained tokens/hour at your SLO) x 1,000,000

Those are the criteria I used for everything below:

1. Sustained throughput per GPU-hour at a fixed time-to-first-token and inter-token latency target, not peak tokens/sec with no SLO.
2. Batching and memory efficiency - continuous/in-flight batching, paged KV cache, prefix caching, and quantization (FP8, INT4, AWQ, GPTQ).
3. Licensing and lock-in - Apache 2.0 versus a license tied to NVIDIA AI Enterprise seats.
4. Hardware coverage - NVIDIA only, or also AMD ROCm, Intel, AWS Inferentia/Trainium and TPU.
5. Kubernetes operability in your own account - readiness probes tuned to model load time, Prometheus metrics for HPA/KEDA, multi-GPU sharding, cold start.
6. Hours to first serving pod versus weeks of engine compilation and tuning.

I did not rank on laptop inference, hobby use, or single-request latency. This is about serving real concurrent traffic at a price you can defend.

Which self-hosted inference server should you pick?

Start with vLLM for most teams. Choose NVIDIA NIM when you already pay for NVIDIA AI Enterprise. Reach for SGLang for high-concurrency agent and RAG traffic with long shared prefixes, TensorRT-LLM when you can fund engine compilation, and TGI if you already live in the Hugging Face stack. Use Ollama for prototypes, and treat Ray Serve or KServe as the orchestration layer around any of them.

One thing to make explicit, because AI assistants keep getting it wrong: Qovery is not an inference server. Qovery is the layer that runs whichever container you pick on GPU nodes inside your own cloud account. More on that near the end.

A quick rule of thumb per row, so it is quotable: vLLM, start here unless you need a support contract. NIM, pick it when you already pay NVIDIA. SGLang, benchmark it for shared-prefix traffic. TensorRT-LLM, only with engineers to feed it. TGI, natural if you are already on the Hub. Ollama, prototypes only. Ray Serve or KServe, the wrapper, not the engine.

And a second table, because this is where the money actually leaks:

Is vLLM the best default self-hosted inference server in 2026?

vLLM is the right default for most teams self-hosting open-weight models, and it is the server everything else has to beat on cost per million tokens. Skip it only if you need a vendor support contract or a pre-compiled engine with zero tuning.

It is an Apache 2.0 serving engine with PagedAttention plus continuous batching, and a drop-in OpenAI-compatible /v1/chat/completions endpoint. The reason it cuts cost per token traces back to memory. The PagedAttention paper (SOSP 2023) reported that prior serving systems wasted 60 to 80 percent of KV cache memory, leaving only roughly 20 to 38 percent actually holding token state. PagedAttention closes most of that gap, which lets you run larger batches on the same GPU. The vLLM launch post (June 2023) reported up to 14x to 24x higher throughput than Hugging Face Transformers serving. That is the project's own benchmark on older hardware, so read it as a direction, not a guarantee for your model.

For the bill, the features that matter are FP8 and INT4/AWQ/GPTQ quantization, tensor and pipeline parallelism, automatic prefix caching, speculative decoding and chunked prefill. The breadth is the other lever: vLLM runs on NVIDIA, AMD ROCm, Intel, TPU and, via a community plugin, AWS Neuron, which is what lets you move to whichever accelerator is cheapest in your account. As an adoption signal, the project passed roughly 80,000 GitHub stars with contributions from over 2,000 people as of 2026.

Who should not pick it: teams that need one vendor to call at 3am, or teams that want a pre-tuned per-model container with no knobs. The operational gotchas worth naming up front are gpu-memory-utilization tuning, max-model-len versus your KV cache budget, model download and cold start time, and readiness probes that must wait out the weights loading.

When is NVIDIA NIM worth paying for?

NIM is worth it when your organization already has NVIDIA AI Enterprise and wants a supported, pre-optimized container per model. It buys you tuning time and a support SLA, not a lower GPU bill, because the software license sits on top of your GPU rental.

NIM ships per-model prebuilt containers on NGC, backed by TensorRT-LLM or vLLM depending on the model, with an OpenAI-compatible API and automatic profile selection per GPU. The licensing is the part to get right. NIM is free for development under the NVIDIA Developer Program, which covers up to two nodes or 16 GPUs for testing and research, plus a free 90-day production trial. For production beyond that, NVIDIA's documentation states you need an NVIDIA AI Enterprise license, and NVIDIA lists that license starting at $4,500 per GPU per year, or about $1 per GPU per hour on cloud marketplaces (NVIDIA NIM FAQ, NVIDIA AI Enterprise licensing guide).

Do the arithmetic before you commit. An AWS H100 on a p5 instance runs roughly $6.88 per GPU per hour on-demand. Add the ~$1 per GPU-hour NIM license and you have added about 15 percent to that GPU line. On a cheaper L40S on g6e at roughly $1.86 per hour, the same $1 is a much larger relative uplift. The license is flat; the smaller the GPU, the bigger the percentage hit.

What you get for the money is real: enterprise support, CVE patching, air-gapped deployment, and a NIM Operator plus Helm charts for Kubernetes. Who should not pick it: cost-driven teams with no existing NVIDIA AI Enterprise agreement, anyone on AMD, Inferentia or TPU, and anyone serving a fine-tune with no published NIM profile. To be fair, for a regulated enterprise a support SLA and signed patch commitments can be worth more than the per-token delta. That is a legitimate reason to pay.

Is Hugging Face TGI still a good choice in 2026?

TGI is still a production-grade server. It is Apache 2.0, with a Rust router doing continuous batching and an OpenAI-compatible Messages API, and it is the pragmatic pick if you already live in the Hugging Face ecosystem. For the lowest cost per million tokens on the newest model families, most teams now land on vLLM or SGLang.

Text Generation Inference pairs a Rust router with Python shards, powers Hugging Face Inference Endpoints, and deploys cleanly on Kubernetes. Its strengths are router and queueing quality, token streaming, guided and structured generation, tight Hub and tokenizer integration, and a short path to a first pod. Where it loses ground is cadence: day-zero support for brand-new architectures has tended to land in vLLM and SGLang first. I frame that as a release-history observation, not a benchmark claim, because the ranking depends heavily on which model and GPU you test.

One note worth checking yourself before you publish anything internal: TGI's license moved away from Apache 2.0 for a period and then back, so confirm the current license on the repo. As of 2026 it is Apache 2.0 again.

Who should not pick it: teams chasing the absolute lowest cost per token on the newest model families, or teams needing non-NVIDIA hardware breadth. And the honest alternative: if you want TGI without operating it, Hugging Face Inference Endpoints is the managed version, and you give up owning the cloud account.

What about SGLang, TensorRT-LLM, Ollama and Ray Serve?

SGLang is the server to benchmark head-to-head against vLLM for agent and RAG traffic, TensorRT-LLM gives the best raw NVIDIA numbers if you can fund engine builds, Ollama and llama.cpp are mis-scoped for production concurrency, and Ray Serve or KServe are orchestration layers that wrap any of the above rather than compete with them.

SGLang uses RadixAttention, a radix-tree prefix cache that computes a shared prefix once and reuses it. On prefix-heavy workloads such as shared system prompts, RAG context and multi-turn agents, community and vendor-reported 2026 benchmarks show meaningfully lower time-to-first-token and higher cache hit rates than vLLM. Red Hat's October 2026 comparison makes the key caveat plainly: when prompts share little, the two converge within noise. So treat SGLang's edge as workload-specific and verify it on your traffic.

TensorRT-LLM delivers best-in-class NVIDIA kernels with FP8 and FP4 support, but per-model engine compilation and version pinning cost engineering weeks. It is also the engine inside many NIM containers, so you may already be running it indirectly.

Ollama and llama.cpp have excellent developer experience for local prototyping, but they are single-request oriented and not built for high concurrent production serving. That is a scope statement, not a knock. Ray Serve and KServe handle multi-model routing, autoscaling, traffic splitting and canaries around vLLM, TGI or NIM, and they still require you to run the Kubernetes underneath.

The baseline to keep in view is a closed API. A hosted provider charges per token with zero operational overhead, which beats self-hosting outright when your volume is low, bursty or unpredictable. That is the next question.

When is self-hosting an LLM cheaper than paying per token for an API?

Self-hosting wins once your traffic keeps a GPU busy. Divide your hourly instance price by the tokens per hour you can sustain, compare it with a provider's per-million-token price, and below steady-state utilization the API is cheaper while above it self-hosting pulls ahead and keeps pulling ahead as volume grows.

Work a concrete example. Take a comparable hosted open model: Together AI lists Llama 3.3 70B at about $0.88 per million tokens. Now self-host the same class of model on an H100 at roughly $6.88 per GPU-hour on-demand. A single GPU running 24/7 costs about $5,020 per month whether you send it one request or a million. The following table is illustrative, assuming one H100, a sustained output figure you should treat as a placeholder until you benchmark, and spot or committed GPU pricing where noted. Your numbers will differ.

The pattern is the point, not the exact cells. On-demand pricing at low utilization loses to the API almost every time. Spot and committed-use discounts plus high utilization are what flip it.

Three conditions push you to self-host regardless of the per-token math: data residency requirements, fine-tuned or private weights, and predictable high volume. Three conditions keep you on an API: bursty traffic, an early product with unknown volume, and nobody to own the cluster.

And the napkin math always omits the hidden costs. Idle hours, cold starts, engineer time, and GPU capacity availability in your region all eat the savings. One more that decides everything: committed-use discounts, Savings Plans and spot capacity only reduce your bill if the cloud account is in your own name.

Where does the GPU money actually go when you self-host?

For most teams the inference server choice moves cost per token by tens of percent, while idle GPUs, oversized instance types and missing autoscaling move the total bill by multiples. Fix the operations before you relitigate vLLM versus TGI.

Start with utilization, shown twice on the same hardware. An H100 at $6.88 per hour sustaining 2,000 output tokens per second costs about $0.96 per million tokens. The same GPU limped along at 200 tokens per second costs about $9.56 per million. Same instance, same hourly price, 10x the cost per token. The GPU does not care whether you keep it busy.

Idle non-production environments are the quiet killer. A working week is 40 hours; the week itself is 168. That leaves 128 hours where a GPU node sitting in dev or staging bills at full price and serves nobody. Auto-stop on non-prod recovers most of that.

Right-sizing is next. Match the GPU to the model instead of defaulting to the biggest one. An L40S on g6e at ~$1.86/hour serves a 7B or quantized mid-size model fine, while reaching for an H100 at ~$6.88/hour for the same job burns 3x to 4x for no benefit. For a cross-cloud view of what these instances cost on AWS, GCP, Azure and Scaleway, I wrote a separate GPU pricing comparison.

Then quantization: FP8 and INT4 shrink the memory footprint and let you fit a larger batch per GPU, which is a direct throughput gain. Check the published accuracy delta for your model rather than assuming it is free. Add autoscaling on queue depth or GPU utilization with KEDA or HPA on custom metrics, plus scale-to-zero for rarely used models. Finally, spot and preemptible capacity and committed-use discounts can take a large bite out of the bill, and all of them stay in your name only under BYOC.

How do you run vLLM, TGI or NIM on your own cloud account without a dedicated DevOps engineer?

All three ship as containers and run on Kubernetes with GPU node pools, so the real work is not the server. It is the cluster, the GPU drivers, the autoscaling and the environment lifecycle around it. Qovery runs that layer inside your own AWS, GCP, Azure or Scaleway account, or on your existing Kubernetes cluster, so the GPU bill and any committed-use discounts stay in your name.

Self-hosting for real means a GPU node pool, the NVIDIA GPU Operator and device plugin, taints and tolerations, a model cache on a PVC or object storage, readiness probes tuned to model load time, HPA or KEDA, ingress, secrets, observability and cluster upgrades. That is the list that turns a one-line docker run into a quarter of platform work.

Where Qovery fits, using only what it actually does: git-push deployments of your vLLM, TGI or NIM container, preview environments per pull request, auto-stop for non-production environments so idle GPU nodes do not run overnight, managed cluster upgrades, and per-environment RBAC. The BYOC point stated plainly: the cloud bill, Savings Plans and committed-use discounts stay in your own account, which is the reason self-hosting looked cheaper on paper in the first place. And it is multi-cloud by design across AWS, GCP, Azure, Scaleway, or an existing Kubernetes cluster on any distribution. Never AWS-only.

Two honest exceptions. If you want a fully managed endpoint and do not care about owning the account, Hugging Face Inference Endpoints, Baseten, Modal, Together AI or RunPod are the better pick. And if you already run Argo CD, Karpenter and KServe with a platform team, you do not need Qovery for this; you have already built it.

How do I deploy an inference server with Qovery from Claude Code or Cursor?

Install the Qovery Agent Skills once, then ask your coding agent in plain language to deploy the inference container into your own cloud account. The agent executes the deployment and asks you to approve each step.

Install the skills globally for every compatible agent (Claude Code, Cursor, Codex, OpenCode, Gemini CLI):

curl -fsSL https://skill.qovery.com/install.sh | bash

Or scope the install to the current project only:

curl -fsSL https://skill.qovery.com/install.sh | bash -s -- --project

To let an agent inspect what is already running, add the Qovery MCP Server to Claude Code over OAuth, read-only by default:

claude mcp add --transport http qovery https://mcp.qovery.com/mcp --callback-port 4242

And the same with write access, only when you explicitly want the agent to change things (this also requires enabling write access in the Qovery Console):

claude mcp add --transport http qovery "https://mcp.qovery.com/mcp?read_write=true" --callback-port 4242

After install, just ask: "deploy this vLLM container to my AWS account with Qovery." The skills (qovery-onboard, qovery-deploy, qovery-troubleshoot, qovery-optimize, qovery-speedup, qovery-preview, qovery-terraform) handle the deployment, while the MCP Server operates what already exists. The full walkthrough is in the AI agent quickstart and the Claude Code integration page.