How to Self-Host an Open Source LLM in 2026: 5 Setups That Actually Work

Key Points:

• The fastest working self-hosted LLM setup for one person is Ollama plus Open WebUI: two commands, a ChatGPT-style UI, and an OpenAI-compatible endpoint at http://localhost:11434/v1 in under 15 minutes. Ollama runs llama.cpp under the hood, so it works on a laptop, an Apple Silicon Mac, or a single consumer GPU.
• For many concurrent users, vLLM is the right serving engine. PagedAttention plus continuous batching cut KV cache waste to under 4% and, in vLLM's own benchmarks, deliver 14x to 24x the throughput of naive HuggingFace serving. SGLang, Hugging Face TGI, and NVIDIA TensorRT-LLM are credible alternatives.
• Sizing a model is arithmetic: roughly 2 GB of VRAM per billion parameters at FP16, ~1 GB at 8-bit, ~0.6 GB at 4-bit, plus KV cache. An 8B model at 4-bit fits a 12-16 GB consumer GPU; a 70B at FP16 needs ~140 GB, so 2x H100 80GB or 4x A100 40GB.
• Self-hosting beats the OpenAI or Anthropic API only above sustained high utilization, because a GPU bills per hour and an API bills per token. A GPU at 5% duty cycle costs roughly 20x more per token than the same GPU at full load, and enterprise GPU fleets average around 5% utilization.
• The inference engine is about 30% of the job. Production self-hosting also needs GPU node pools, autoscaling, cached weights, a gateway, SSO, and idle shutdown. That is where Qovery fits: it deploys vLLM, Ollama, and Open WebUI into your own AWS, GCP, Azure, Scaleway, or existing Kubernetes cluster, so GPU spend and committed-use discounts stay on your bill.

I have watched a lot of teams spin up their first self-hosted model, and the pattern is always the same. The demo on a laptop works in ten minutes. Then someone asks "can the whole team use it?" and six months later they are debugging GPU autoscaling at 2am. The tool choice was never the hard part. The hard part is everything around it.

This is the honest guide I wish those teams had in 2026. Real VRAM math, real hourly GPU prices from the cloud pricing pages, defensible throughput ranges, and commands you can paste. Self-hosting an open source LLM comes down to four decisions in order: pick the engine, pick the interface, decide where it runs, then check the cost math. Let's take them one at a time.

What are the best ways to self-host an open source LLM?

Five setups work in 2026, and the number of people talking to the model at once picks the winner: llama.cpp for raw local inference on CPU or Apple Silicon, Ollama for one-command local developer use, Open WebUI plus LiteLLM as the interface and gateway layer, vLLM (or SGLang / TGI) for multi-user production serving, and a bring-your-own-cloud platform such as Qovery when you need the production stack in your own account without building Kubernetes yourself.

Map that to three situations. One user on a laptop: Ollama plus Open WebUI. A team of 10 to 50: vLLM on one GPU box behind a LiteLLM gateway. A customer-facing product with real traffic: vLLM on Kubernetes in your own cloud account, with autoscaling, a gateway, and SSO.

The models people actually self-host are open-weight, and their size plus quantization drive hardware cost far more than which tool you pick. The ones I see most in 2026:

Here is the mental model that makes the rest of this article click: an inference engine (llama.cpp, Ollama, vLLM) serves tokens, and an interface and gateway layer (Open WebUI, LiteLLM) makes those tokens usable by humans and apps. Most teams need one of each.

Let me also say plainly what self-hosting does not give you. It is not automatically cheaper. It is not zero-ops. And open weights still trail the frontier on the hardest reasoning and agentic tasks: on Artificial Analysis's Intelligence Index through 2026, the best open-weight models have closed to within single-digit points of the top closed models, and the coding gap has largely gone, but a measurable gap remains at the top end and the ranking shifts week to week.

Two commands to anchor the whole piece. This is the local one-liner:

ollama run llama3.1:8b

And this is the production one:

vllm serve Qwen/Qwen3-8B --max-model-len 8192

Here is how the five setups compare.

Which inference engine should you use - Ollama, llama.cpp, or vLLM?

Use llama.cpp or Ollama when one person talks to the model at a time, and use vLLM when many requests arrive at once. The same weights produce the same answers. The difference is memory management and batching, which is why vLLM holds throughput under load while Ollama's collapses past a handful of simultaneous users.

llama.cpp is the C/C++ engine underneath most local tools, with ~129k GitHub stars as of September 2026 (github.com/ggml-org/llama.cpp, star counts move fast). It runs on CPU, Apple Silicon Metal, and consumer GPUs, and it reads GGUF quantized files. The quant types you will actually use are Q4_K_M (around 4.5 bits per weight, the common sweet spot), Q5_K_M (around 5.5 bpw), and Q8_0 (around 8.5 bpw, near-lossless). Reach for llama.cpp when you have no datacenter GPU.

Ollama (~182k stars, github.com/ollama/ollama) is a wrapper around llama.cpp with a model registry, one-command pulls, and an OpenAI-compatible endpoint on port 11434. Best ergonomics, weakest concurrency. It exposes OLLAMA_NUM_PARALLEL (parallel requests per model, default 1) and OLLAMA_MAX_LOADED_MODELS (default 3 per GPU), but those knobs do not turn Ollama into a production server. Its official library hosts several hundred curated model families, each with many size and quantization tags.

vLLM (~93k stars, github.com/vllm-project/vllm) is the one to reach for under load. Its PagedAttention paper reports 2 to 4x the throughput of prior state-of-the-art serving systems with near-zero KV cache waste, and vLLM's own blog puts the gain at 14x to 24x over HuggingFace Transformers and up to 2.5x over TGI, while cutting KV cache memory waste from the 60-80% typical of naive systems to under 4%. It adds continuous batching, prefix caching, and tensor and pipeline parallelism, all behind an OpenAI-compatible server.

Fair one-liners on the rest: SGLang (~36k stars) uses RadixAttention and reports up to ~3.1x vLLM's throughput on large models in its own Llama-3 benchmarks, and it shines on structured output and agentic workloads. Hugging Face TGI is strong in the HF ecosystem and claims big wins on long prompts (favorable-by-construction, since it leans on prefix-cache reuse). NVIDIA TensorRT-LLM is the fastest on NVIDIA hardware if you accept an ahead-of-time engine build step (trtllm-build compiles a GPU-specific engine). Ray Serve and KServe handle orchestration, LM Studio and Jan are desktop GUIs, and llamafile ships a model as a single binary.

One trap worth naming: quantization formats are not interchangeable across engines. vLLM does not consume GGUF the way llama.cpp does (it prefers FP16, FP8, AWQ, GPTQ, INT8), and FP8 needs Hopper-class hardware or newer. The reusable rule to keep in your head: ~2 GB of VRAM per billion parameters at FP16, ~1 GB at 8-bit, ~0.6 GB at 4-bit, plus KV cache that scales with context length times concurrency.

How much VRAM and what GPU do you need to self-host an LLM?

VRAM needed equals parameters times bytes per parameter, plus KV cache. An 8B model needs ~16 GB at FP16 or ~5-6 GB at 4-bit, a 32B needs ~64 GB at FP16 or ~20 GB at 4-bit, and a 70B needs ~140 GB at FP16, which means 2x H100 80GB, 4x A100 40GB, or a single 80 GB GPU if you drop to 4-bit (a Llama 3.3 70B in Q4_K_M is about 42 GB on disk).

Here is the sizing table with concrete GPUs, cloud instances, and on-demand prices. Prices are 2026 on-demand list rates and move; verify before you budget.

On the consumer and workstation path, the driver of single-stream tokens/sec is memory bandwidth: the RTX 4090 runs 1,008 GB/s (24GB), the RTX 5090 1,792 GB/s (32GB), and Apple's M-series unified memory runs ~410-546 GB/s on M4 Max and ~819 GB/s on M3 Ultra, with 36 to 512 GB of unified memory you can point at a model. Honest tokens/sec ranges depend on quantization, batch size, context length, and bandwidth, so treat single-stream numbers as "tens of tokens per second for a quantized 8B on a good consumer card" and measure your own.

On the cloud path, the workhorses and their bandwidth: AWS L4 (300 GB/s), L40S (864 GB/s), A100 80GB SXM (2,039 GB/s), and H100 80GB SXM (3.35 TB/s). Named instances: AWS g5 (A10G), g6 (L4), g6e (L40S), p4d (A100), p5 (H100); GCP g2 (L4), a2 (A100), a3 (H100); Azure NCads A100 v4 and ND H100 v5; Scaleway L4, L40S, and H100. A single H100 runs about €2.73/hr on Scaleway (~$2.95) or $3.29/hr on a neocloud like Lambda, versus roughly $6.88 per GPU inside an AWS p5 or ~$12 per GPU on an Azure ND H100 v5.

Three things people miss:

• KV cache is the hidden memory eater. It scales as roughly 2 x layers x kv_heads x head_dim x bytes x context_length x concurrent_sequences. Long context at high concurrency can exceed the weights themselves, which is exactly the fragmentation PagedAttention solves.
• MoE models change the math. Total parameters set the memory footprint, active parameters set the speed. gpt-oss-120b holds ~117B parameters but activates only ~5.1B per token, so it can be faster than a dense 70B while needing an 80 GB card to hold it.
• CPU-only inference works for 3-8B models at single-digit tokens/sec. That is fine for a batch job, too slow for interactive multi-user use. Say it out loud rather than hedging.

Practical rule: reserve 10-20% VRAM headroom above the calculated figure or the first long prompt will OOM. vLLM's --gpu-memory-utilization defaults to 0.9 for exactly this reason.

How do you give your team a ChatGPT-like interface and a single API on top of your model?

Put Open WebUI in front for the chat interface and LiteLLM in front for the API gateway. That pair turns a raw inference server into something a whole team can use: accounts, RBAC, RAG over uploaded documents, per-team virtual API keys, budgets, rate limits, and request logs. Both speak the OpenAI API, so existing code keeps working after a base_url swap.

Open WebUI (~153k stars and around 398 million container pulls on GHCR as of September 2026) is the self-hosted ChatGPT alternative most teams land on. It gives you a chat UI, multi-user accounts and RBAC, RAG over uploaded documents, and model switching against any OpenAI-compatible backend. One command:

docker run -d -p 3000:8080 -v open-webui:/app/backend/data \
  --name open-webui ghcr.io/open-webui/open-webui:main

LiteLLM (~60k stars, github.com/BerriAI/litellm) is the gateway. It puts a single OpenAI-compatible endpoint in front of 100+ LLMs, self-hosted and hosted, with virtual keys, per-team budgets, rate limits, retries, fallbacks, and spend logging. This is the layer most DIY setups skip and then bolt on painfully six months later, usually right after a runaway script burns through a budget nobody was tracking. A minimal config.yaml:

model_list:
  - model_name: llama-3.1-8b
    litellm_params:
      model: openai/llama3.1:8b
      api_base: http://ollama:11434/v1
      api_key: "none"
  - model_name: qwen3-8b
    litellm_params:
      model: hosted_vllm/Qwen/Qwen3-8B
      api_base: http://vllm:8000/v1
general_settings:
  master_key: sk-your-master-key

Fair alternatives, one reason each: LibreChat (multi-model chat with a clean admin story), AnythingLLM (RAG-first with workspaces), Text Generation WebUI (power-user knobs), Jan and Chatbox (desktop apps), and Portkey or Kong AI Gateway on the gateway side.

Why OpenAI compatibility matters is concrete: the OpenAI SDKs, LangChain, LlamaIndex, Cursor, and IDE agents all keep working after a base_url and api_key swap.

from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

One security point, stated bluntly: never expose Ollama's port 11434 to the internet unauthenticated. In February 2026, LeakIX counted 12,269 exposed, zero-auth Ollama instances, around a thousand of them still on versions vulnerable to CVE-2024-37032 "Probllama", the remote-code-execution bug Wiz disclosed in 2024. Terminate TLS, put the gateway behind SSO, and keep the engine on a private network. The reference architecture is one sentence: users -> Open WebUI -> LiteLLM gateway -> vLLM on GPU nodes, with Postgres for chat history and Redis for rate limiting.

Is self-hosting an LLM cheaper than the OpenAI or Anthropic API?

Self-hosting is cheaper only above sustained high utilization, because a GPU bills per hour whether it serves tokens or sits idle, while an API bills per token. The arithmetic that decides it, with every variable named:

cost per 1M output tokens = GPU_hourly_price / (output_tokens_per_second x 0.0036)

The 0.0036 is just 3600 / 1,000,000: seconds per hour over a million. Plug in a GPU that costs $2/hour and serves 1,500 output tokens/sec at full load, and you get 2 / (1500 x 0.0036) = $0.37 per 1M output tokens. Cut utilization to 10% and the same GPU costs ~$3.70 per 1M, because you are paying for 1,500 tok/s of capacity while using 150.

Now the real numbers. Throughput below is a defensible range under batching and depends on quantization, batch size, and context length, so treat it as illustrative and measure your own.

Read that table honestly. The real competitor to self-hosting is not the GPT-class or Claude API. It is a hosted open-weight provider like Together, Groq, or Fireworks, where the same Llama or gpt-oss weights run for well under $1 per million tokens with zero ops. A well-utilized self-hosted 8B lands in the same ballpark as hosted 8B. You only pull clearly ahead per token at high, sustained volume on hardware you keep busy.

The idle problem is the whole game. A GPU at 5% duty cycle costs roughly 20x more per token than the same GPU at 100%, and independent measurement finds enterprise GPU fleets averaging around 5% utilization. Reserved, committed-use, and spot pricing move the break-even but never remove it: AWS Savings Plans go up to ~66-72%, GCP committed-use discounts run ~37% for one year and ~55% for three, and spot can reach 90% off if your workload tolerates interruption.

So why self-host at all? The legitimate reasons are rarely raw token price:

• Data residency and GDPR, contractual no-training guarantees, and air-gapped or regulated environments where data cannot leave your account.
• Fine-tuned domain models you own outright.
• Predictable latency and model version pinning, so a vendor deprecation never breaks your product overnight.

That matches what enterprises say. In a16z's enterprise survey, 46% of AI leaders said they prefer open-source models, and control and customization outranked cost as the reason.

Name the hidden costs too: idle GPU hours, engineer time to operate and upgrade, monitoring and on-call, storage for weights, and network traffic (AWS internet egress runs ~$0.09/GB and cross-AZ ~$0.01/GB each way). The single biggest lever most teams ignore is the simplest: shut down non-production GPU environments outside working hours. Nights and weekends are roughly 70% of the week, and idle dev GPUs are the most common source of waste I see.

How do you run a self-hosted LLM in production on Kubernetes without building a platform team?

Running a self-hosted LLM in production means GPU nodes on Kubernetes with autoscaling, cached model weights, tuned probes, a gateway, and access control. You have three ways to get there: build it yourself, buy a managed inference endpoint, or use a platform that deploys the stack into your own cloud account so the GPU bill and your discounts stay with you.

The production checklist, scannable:

• GPU node pool with the NVIDIA device plugin advertising nvidia.com/gpu.
• Weights cached in S3/GCS/Azure Blob or a persistent volume. A 70B pull is ~141 GB in FP16, so pulling it on every pod start is a multi-minute cold start.
• Readiness and startup probes tuned for slow model load, not the default web-app timeouts.
• Autoscaling on queue depth, not CPU. KEDA scales on custom and queue metrics (including scale-to-zero); Karpenter provisions right-sized GPU nodes on demand.
• A gateway (LiteLLM), SSO, and audit logs.
• Observability on tokens/sec, time-to-first-token, queue depth, and GPU utilization (NVIDIA DCGM exporter feeds Prometheus).

Option A - DIY Kubernetes. Full control, real ops cost. The legitimate building blocks are KServe, Ray Serve, KubeAI, and NVIDIA NIM for serving, plus Karpenter or Cluster Autoscaler for nodes and DCGM for metrics. This is the right path if you already have a platform team.

Option B - managed inference endpoints (Hugging Face Inference Endpoints, Together AI, Replicate, Modal, RunPod, Baseten). Fastest path to a URL. The tradeoff: traffic and often weights leave your account, and your committed-use discounts or Savings Plans do not apply to someone else's GPUs.

Option C - Qovery. This is where we fit. Qovery deploys vLLM, Ollama, and Open WebUI as services inside your own AWS, GCP, Azure, Scaleway, or existing Kubernetes cluster (BYOC), so GPU spend stays on your bill and your existing commitments still apply. The capabilities I will actually stand behind: GPU node pools you enable on your cluster (documented on AWS EKS, mixing on-demand and spot), git-push deployments, preview environments per pull request, and environment auto-stop for non-production, managed cluster upgrades, per-environment RBAC, and databases backed by managed cloud services like RDS.

Let me be explicit about the boundary, because it is what makes the claim credible: Qovery does not make the model faster. vLLM does. Qovery removes the Kubernetes, networking, scaling, environment, and access-control work around it. The reference architecture is the same one from earlier, now on your cloud: a vLLM service on a GPU node pool, a LiteLLM gateway, Open WebUI, Postgres for history, one production environment plus ephemeral preview environments that auto-stop when idle, which directly kills the nights-and-weekends waste.

Which self-hosting setup should you pick for your situation?

Pick by concurrent users and data constraints. One user: run Ollama plus Open WebUI locally. A team of 10 to 50: run vLLM on one GPU box behind LiteLLM. A customer-facing product: run vLLM on Kubernetes in your own cloud account with autoscaling, a gateway, and SSO.

As if/then rules with the exact stack:

• If one user, then ollama run llama3.1:8b, add Open WebUI with the docker run above, done in 15 minutes on a laptop, a Mac, or an RTX 4090.
• If a team of 10 to 50, then vllm serve Qwen/Qwen3-8B --max-model-len 8192 on one L40S or A100 80GB box, put LiteLLM in front for keys and budgets, and Open WebUI for the humans.
• If a customer-facing product, then the same vLLM plus LiteLLM plus Open WebUI stack on a GPU node pool in your own cloud, with KEDA/Karpenter autoscaling and SSO, or let Qovery stand that up on your account so you skip the platform build.

When should you not self-host at all? Low or spiky volume, no GPU budget, nobody to own the pager, or a hard requirement for frontier-model quality on the hardest tasks. In those cases a hosted open-weight API or a commercial API is cheaper and less work.

The migration path costs nothing to start: begin on Ollama locally, route every call through an OpenAI-compatible base_url from day one, then swap the base_url to vLLM in your cloud when concurrency starts to hurt. Zero application code change. And the same approach works on an existing Kubernetes cluster you already run, on any provider.

The sentence to remember: choose the engine for concurrency, the hardware for VRAM, and the hosting location for cost and compliance. If path three is you, spin up Qovery on your own cloud account and let it handle the Kubernetes around your model.