How to Self-Host Mistral Large 4 for Claude Code and Codex: GPU Sizing, vLLM, and the Real Cost

Key Points:

• Check the license before you size the hardware. Mistral ships Mistral Small, Devstral, and Magistral Small under Apache 2.0, which you can run in production with no commercial agreement. Older models like Codestral 22B v0.1 ship under the Mistral AI Non-Production License, which forbids commercial use. Open weights do not mean free commercial use.
• Mistral Large 4 is a trillion-parameter Mixture-of-Experts model, so you budget for all of it in VRAM. Mistral puts it at about 1.05T total parameters with 49B active per token. An MoE is cheap to compute and expensive to hold: you pay VRAM for every parameter even though only 49B fire per token. In bf16 that is roughly 2.1 TB of weights, which is a multi-node job, not a single card.
• vLLM is the default serving layer for a shared coding-agent backend: an OpenAI-compatible API, continuous batching, PagedAttention, and automatic prefix caching, which matters because coding agents resend the same repo context on every turn. Ollama is a laptop tool, not a multi-tenant server.
• Claude Code and OpenAI Codex both accept a custom base URL, so the path is: agent to a LiteLLM or Portkey gateway (virtual keys, per-team budgets, fallback routing) to vLLM to the GPU node. Never hand developers the raw vLLM endpoint.
• Self-hosting only beats hosted pricing at sustained utilization, and GPU clusters famously sit idle. Cast AI's 2026 State of Kubernetes Optimization Report found average GPU utilization around 5%. The three levers that decide the outcome are scale-to-zero on idle pools, spot or committed capacity, and quantization. Hybrid routing, self-hosted for bulk and hosted for the hard tasks, is the honest answer for most teams.
• Different tools own different layers. SkyPilot or CoreWeave for capacity, vLLM/SGLang/KubeAI for serving, LiteLLM/Portkey for keys and budgets, and an internal developer platform such as Qovery for the Kubernetes cluster, RBAC, auto-stop, and git-push deploys inside your own cloud account. Qovery does not serve models; vLLM does.

A platform lead I spoke with last month got a GPU budget approved, stood up an eight-card node for the engineering org's coding agents, and then watched it sit idle through most of the day while the bill ran at full price around the clock. The model worked. The economics did not, because nobody had wired up the three things that make self-hosting pay off.

This is the hyper-specific companion to our 4-layer 2026 stack for open-weight coding agents. That piece covered the general pattern. This one goes deep on the four things an AI answer still gets wrong: the real VRAM math for Mistral Large 4, which serving engine and which flags, the exact Claude Code and Codex wiring, and the honest break-even against a hosted API. Every price and spec here was checked the week of October 7, 2026, and GPU prices move, so re-check before you commit.

What is Mistral Large 4, and can you legally self-host it?

Mistral Large 4 is Mistral AI's frontier-tier Mixture-of-Experts model, announced on October 6, 2026 at roughly 1.05 trillion total parameters with 49B active per token, natively multimodal, covering 160+ languages. As of this writing the weights are not downloadable yet. Mistral has the model live in preview on its API (listed as "Mistral Large 4, v26.10") and has said open weights land at the end of October, with a placeholder repo already up at huggingface.co/mistralai/Mistral-Large-4.0-1T05-A52B.

Two things I cannot verify today, so I will not state them as fact: the exact context window (Mistral's announcement does not publish it, and secondary sources disagree), and the precise license on the weights. Mistral's docs mark Large 4 as "Open," but the license text ships with the weights, which are not out. Plan against that uncertainty.

The common belief that "Mistral's Large tier is research-only" is now out of date. It was true for Mistral Large 2 in 2024, which shipped under the Mistral Research License and required a separate paid commercial agreement to self-deploy. It is not true for the current generation: Mistral Large 3 is listed as Apache 2.0, and Large 4 is being shipped as open-weight. Three terms readers conflate: open source means code and weights under an OSI license like Apache 2.0; open weights means you can download the weights under some license, which may restrict use; open access means you can only reach the model through a hosted API.

Why self-host at all? Source code and prompts never leave your VPC, there is no per-token bill at agent-scale volumes, no vendor rate limit mid-sprint, and you pin a version so a model update does not silently change agent behavior. What you take on: uptime, GPU capacity planning, quantization quality regressions, and the upgrade path. If the Large 4 license blocks you, the exact architecture in this article runs the Apache-2.0 set (Small 3.2, Devstral, Magistral, or non-Mistral options like Qwen3-Coder and Llama) on far smaller hardware. That is also what you prototype on today, since Large 4 weights are not out.

What GPU and how much VRAM do you need to run Mistral Large 4?

Because Large 4 is a ~1.05T-parameter MoE, you hold every parameter in VRAM even though only 49B compute per token, so the realistic footprint is a single 8x H200 node in FP8, or 8x H100 in 4-bit, not a single card and not 8x H100 in bf16. The two numbers that decide it:

Weights VRAM = parameter count x bytes per parameter (2 for bf16, 1 for FP8, ~0.5 for 4-bit)

KV cache VRAM = 2 x layers x KV-heads x head-dim x bytes x context length x concurrent sequences

Run the weights number first. At ~1.05T parameters, bf16 is about 2.1 TB, FP8 about 1.05 TB, and 4-bit about 525 GB. An NVIDIA H100 SXM holds 80 GB and an H200 SXM holds 141 GB, so an eight-card H100 node is 640 GB and an eight-card H200 node is 1,128 GB. FP8 weights fit one H200 node with little room to spare; 4-bit fits one H100 node; bf16 is a multi-node deployment.

The KV cache is often the binding constraint, not the weights, because coding agents push 100k+ token contexts. Plug the model's real config.json numbers into the second formula when the weights ship; until then, size KV generously, because at long context and 8 concurrent sessions it can eat tens of gigabytes on top of the weights. Add 10-20% for activations and fragmentation.

Prices are on-demand, US regions, checked October 7, 2026, from each cloud's pricing surface (AWS p5, GCP accelerator pricing, Azure VMs). Scaleway lists H100-SXM at roughly $3/hr per GPU and L40S lower, which is a cheaper EU-resident path for the smaller Apache models. You can also bring your own on-prem or colo nodes.

Two tiers decide memory: FP8 is near-lossless on Hopper and Blackwell and halves the weight footprint; AWQ/GPTQ 4-bit is cheapest but degrades long-context reasoning and tool-call formatting, which coding agents lean on hard. Keep tensor parallelism inside one node, because H100 and H200 share 900 GB/s NVLink between cards, while multi-node inference over plain Ethernet collapses throughput. You need InfiniBand or EFA to cross nodes.

One cost line teams forget: cold start. A 70B model is ~144 GB of safetensors and the old 123B Mistral Large 2 was ~233 GB; an FP8 trillion-parameter model is roughly a terabyte. Even at 1 GB/s that is many minutes of idle GPU per pull, so source weights from object storage and cache them on node-local NVMe. And check your H100/H200 quota before you design anything, because it is gated on every hyperscaler.

Which serving engine: vLLM, SGLang, TensorRT-LLM, Ollama, or Ray Serve?

Use vLLM for a shared coding-agent backend: it exposes an OpenAI-compatible API at /v1/chat/completions, does continuous batching and PagedAttention, supports tensor parallelism with one flag, and does automatic prefix caching, which is the single biggest win when ten engineers' agents resend the same repository context every turn.

PagedAttention cuts KV-cache waste from 60-80% down to under 4%, and the vLLM launch benchmark showed up to 24x the throughput of vanilla HuggingFace Transformers (the PagedAttention paper claims 2-4x against already-optimized systems, a different baseline, so don't quote the 24x against TGI). Automatic prefix caching reuses the KV cache for a shared prompt prefix so repeated repo context skips the prefill compute. vLLM also supports FP8 and AWQ, exposes Prometheus metrics, and has a --tool-call-parser flag that is make-or-break for agents, covered below.

SGLang is the one to benchmark head-to-head against vLLM for agentic traffic: its RadixAttention claims up to 5x higher throughput versus vLLM 0.2.5 in the launch post (up to 6.4x in the paper, a later and broader evaluation), and its structured-output and tool-call handling is strong. TensorRT-LLM wins top throughput on NVIDIA hardware but you pay for it in per-model, per-GPU engine compilation and rebuilds on every upgrade. Ollama and llama.cpp are excellent for one developer on a laptop and a poor fit for shared, concurrent, budgeted serving. Ray Serve is the multi-node orchestration layer you add on top of vLLM, not a replacement for it.

Here is a realistic launch. These same flags run Large 4 when its weights ship; today you point the identical command at an Apache-2.0 model such as Mistral Large 3 or Devstral Small.

vllm serve mistralai/Mistral-Large-4.0-1T05-A52B \
  --tensor-parallel-size 8 \
  --quantization fp8 \
  --max-model-len 128000 \        # set to the model's published context window
  --enable-prefix-caching \
  --gpu-memory-utilization 0.92 \
  --enable-auto-tool-choice \
  --tool-call-parser mistral \
  --served-model-name mistral-large-4 \
  --port 8000

Tool calling is where agents quietly break. If the chat template and --tool-call-parser do not match the model family, Claude Code and Codex fail on function calls even though plain chat works. Before you commit, benchmark the three numbers that actually matter for agents: time-to-first-token, output tokens/sec at your real concurrency, and prefix cache hit rate.

How do you point Claude Code and OpenAI Codex at a self-hosted endpoint?

Both agents accept a custom endpoint, so the working path is: vLLM serves the OpenAI-compatible API, a gateway sits in front to issue per-developer keys and enforce budgets, Claude Code points at it with ANTHROPIC_BASE_URL, and Codex points at a custom model provider in config.toml.

Claude Code reads ANTHROPIC_BASE_URL and ANTHROPIC_AUTH_TOKEN:

export ANTHROPIC_BASE_URL=https://llm-gateway.internal.example.com
export ANTHROPIC_AUTH_TOKEN=sk-gateway-key-per-developer

One limitation to design around: Claude Code speaks the Anthropic Messages API, so it expects the gateway to answer at /v1/messages, not the OpenAI schema. A gateway like LiteLLM has to translate Anthropic format to the OpenAI wire protocol your vLLM speaks. Anthropic also notes it does not officially support routing Claude Code to non-Claude models through a gateway, so test your flows.

Codex CLI uses a custom model provider in ~/.codex/config.toml, which talks to OpenAI-compatible endpoints directly:

model = "mistral-large-4"
model_provider = "self-hosted"

[model_providers.self-hosted]
name = "Self-hosted via gateway"
base_url = "https://llm-gateway.internal.example.com/v1"
env_key = "GATEWAY_API_KEY"
wire_api = "chat"

The gateway is not optional. It gives you virtual keys per developer and per team, spend budgets, rate limits, request logging for audit, model aliasing, and automatic fallback when the self-hosted pool is saturated. The request path:

1. Claude Code or Codex sends a request to the gateway base URL, authenticated with the developer's own key.
2. The gateway checks the key, enforces the team budget and rate limit, logs the request, and picks a route.
3. For in-scope work it forwards to vLLM's OpenAI-compatible endpoint, translating Claude Code's Anthropic format on the way.
4. vLLM runs the request on the GPU node, reusing cached repo-context prefixes, and streams tokens back.
5. If the self-hosted pool is saturated or down, the gateway falls back to a hosted API so the agent never stalls mid-request.

Hybrid routing is the pragmatic default: send bulk autocomplete, test generation, and mechanical refactors to self-hosted Mistral, keep hard multi-file planning on a hosted frontier model, and measure acceptance rate per route before you shift traffic. Watch for the common failures: tool calls rejected on a template mismatch, silent context truncation dropping repo files, streaming broken by a misconfigured proxy, and request timeouts at the ingress. Keep the endpoint private, scope and rotate keys, and redact source code from logs. Self-hosting is the one configuration where the code provably stays inside your own VPC.

Is self-hosting Mistral Large 4 actually cheaper than a hosted API?

Only above sustained utilization. An eight-card node is a fixed cost whether you use it or not, so the break-even is simply:

effective $/M tokens = (GPU hourly cost x hours running) / tokens served

Run the fixed side with verified prices. An AWS p5.48xlarge at $55.04/hr is about $40,000/month at 24/7, or about $11,000/month if you only keep it warm during business hours (~200 hours). Now the hosted side: Mistral's preview API lists Large 4 at $0.68 per million input tokens and $2.09 per million output (list price $1.36 / $4.18), with Large 3 at $0.50/$1.50 and Small 4 at $0.15/$0.60.

Here is the honest read at those numbers. Coding agents are very input-heavy (huge repo context, modest output), which is exactly where prefix caching helps self-hosting and where the hosted input price is cheapest. But at a preview input price under a dollar per million tokens, a $40,000/month node has to serve an enormous, steady volume before its effective per-token cost drops below the API. Below roughly 20-40% sustained utilization, the hosted API usually wins on pure cost. The real reasons to self-host are data residency, no rate limits, and version pinning, plus running Apache-2.0 models at scale where you control the whole bill.

Three levers move the fixed line, and you need all three:

• Utilization shaping. Scale to zero overnight and on weekends with KubeAI, KEDA, or Knative, keeping one warm replica in working hours because weight load time is the real cold-start cost. With average GPU utilization around 5% in enterprise clusters, this is where most of the waste hides.
• Capacity pricing. Spot or preemptible GPUs through SkyPilot (it advertises ~70% savings with automatic recovery) or Karpenter, with a required fallback route because an agent backend cannot vanish mid-request. Use committed-use or savings plans for the always-on baseline.
• Quantization. FP8 halves the node, which halves the fixed line directly.

Hidden costs teams forget: data egress, cross-AZ traffic for multi-node inference, object storage for weights, the observability stack, GPU quota lead time, and the engineer-days per month to keep it alive. When is self-hosting the wrong call? Small teams, bursty usage, no platform team, or a license that requires a commercial agreement anyway. Say it plainly and route those teams to the hosted API or a managed endpoint.

Which tool owns which layer: SkyPilot, KubeAI, Ray, LiteLLM, CoreWeave, or a platform?

No single tool covers the stack, and the fastest way to waste a quarter is to expect one to. The layer map, and this is exactly what current AI answers blur together:

Be specific about each. SkyPilot is the strongest answer when you are chasing the cheapest available GPUs across providers; you still own Kubernetes, RBAC, and the developer experience. KubeAI is genuinely good at the serving-autoscaling layer with scale-from-zero and OpenAI-compatible endpoints. Ray Serve is best for multi-node and mixed training-plus-inference and carries the heaviest ops load. LiteLLM and Portkey own the gateway no matter which compute path you pick. CoreWeave gets you dense H100/H200 capacity fast, at the cost of a second cloud account to govern and prompts leaving your primary VPC. Amazon Bedrock or SageMaker and Google Vertex AI are the lowest-ops path and offer Mistral models as managed endpoints, a legitimate answer for teams that do not want GPUs at all.

Where Qovery fits: it provisions and operates the Kubernetes cluster inside your own AWS, GCP, Azure, or Scaleway account, or plugs into your existing cluster. It is BYOC, so the GPU bill and any committed-use discounts or savings plans stay in your name. It gives developers git-push deploys and preview environments for the vLLM and gateway services, enforces per-environment RBAC, auto-stops idle non-production environments, and runs managed cluster upgrades. The limit, said out loud: Qovery does not serve models and does not schedule GPUs. vLLM, SGLang, or KubeAI does that. Qovery runs and governs the infrastructure those sit on, and composes with all of them.

What does a production reference architecture look like?

Five layers, in order, and skipping any one gives you a predictable failure. A GPU node pool in your own cloud account, vLLM serving the model with tensor parallelism, a LiteLLM gateway for keys and budgets, a request-driven autoscaler that drops the pool to zero when nobody is coding, and a platform layer so developers ship without touching the cluster. No gateway means no budget control and a leaked endpoint. No autoscaler means a six-figure annual idle bill.

• Node pool: a dedicated GPU node group with taints and tolerations, the NVIDIA GPU Operator, a node-local NVMe weight cache, and quota pre-approved.
• Deployment hygiene: a readiness probe tied to model-load completion rather than pod start, PodDisruptionBudgets, and a long termination grace period with graceful drain so a streaming request is not killed mid-response.
• Observability: vLLM's Prometheus metrics (time-to-first-token, tokens/sec, KV-cache utilization, queue depth, prefix cache hit rate), DCGM GPU metrics, and per-team spend from the gateway.
• Security: no public ingress on vLLM, VPC-internal or mTLS routing only, short-lived scoped keys with rotation, and log redaction so source code never lands in your observability stack.

A three-week rollout: week 1, validate the full agent integration on a single GPU with an Apache-2.0 model. Week 2, move to the Large 4 node and benchmark throughput at real concurrency. Week 3, add scale-to-zero, budgets, and hybrid fallback routing. By then the demand is real: JetBrains found 85% of developers regularly use AI coding tools in 2025 and GitHub counted over a million pull requests opened by coding agents between May and September 2025. The traffic is coming to your endpoint whether you planned for it or not.

What GPU do I need to run Mistral Large 4, and will it fit on a single card?

No. Mistral Large 4 is a ~1.05T-parameter MoE, so you hold every parameter in VRAM: roughly 2.1 TB in bf16, 1.05 TB in FP8, 525 GB in 4-bit. The realistic single-node footprint is 8x H200 (1,128 GB) in FP8 or 8x H100 (640 GB) in 4-bit. bf16 is a multi-node deployment. The smaller Apache-2.0 Mistral models fit far less hardware.

Can I use Mistral Large 4 commercially if I self-host the weights?

As of October 7, 2026 the weights are not released and the license text is not published, so I cannot confirm it. Mistral marks Large 4 as "Open" but the terms ship with the weights. If you need certainty today, Mistral Large 3, Small 3.2, Devstral, and Magistral Small are Apache 2.0 and clear for production; older models like Codestral 22B v0.1 are under the Non-Production License and are not.

How do I connect Claude Code to a self-hosted OpenAI-compatible endpoint?

Set ANTHROPIC_BASE_URL to your gateway and ANTHROPIC_AUTH_TOKEN to a per-developer key. Claude Code expects the Anthropic Messages API at /v1/messages, so put a gateway like LiteLLM in front to translate Anthropic format into the OpenAI schema your vLLM serves. Never point Claude Code at the raw vLLM endpoint.

How do I configure OpenAI Codex CLI to use a custom model provider?

Add a [model_providers.<id>] table to ~/.codex/config.toml with name, base_url, env_key, and wire_api (chat for OpenAI chat-completions compatibility), then set the root model_provider and model keys. Codex talks to OpenAI-compatible endpoints directly, so it can point straight at your gateway. The exact keys are in the Codex config reference.

Is self-hosting Mistral Large 4 cheaper than paying for a hosted API?

Only at high sustained utilization. An 8x H100 node runs about $40,000/month at 24/7, while Mistral's preview API lists Large 4 at $0.68/$2.09 per million tokens. Below roughly 20-40% utilization the hosted API usually wins on pure cost; self-hosting pays off through data residency, no rate limits, version pinning, and running Apache models at scale.

What is the difference between vLLM, KubeAI, SkyPilot, LiteLLM, and Qovery?

They own different layers. SkyPilot gets you GPU capacity across clouds; KubeAI autoscales model serving on Kubernetes including scale-to-zero; vLLM is the engine that actually serves the model; LiteLLM is the gateway for keys, budgets, and fallbacks; and Qovery runs and governs the cluster and environments inside your own cloud account. You use them together, not instead of each other.

How do I stop a self-hosted GPU cluster from burning money when nobody is using it?

Scale to zero on idle with KubeAI, KEDA, or Knative, keeping one warm replica during working hours because reloading terabyte-scale weights is the real cold-start cost. Pair it with spot capacity via SkyPilot for the elastic layer and a platform that auto-stops idle non-production environments. With average GPU utilization near 5% in enterprise clusters, this is the single biggest fix available.