Self-Hosting Qwen, DeepSeek, and Mixtral Under EU AI Act GPAI Enforcement: What You Must Document, Log, and Prove

Key Points:

• GPAI obligations under the EU AI Act (Regulation (EU) 2024/1689) have applied since 2 August 2025, and the Commission's power to enforce them and fine GPAI providers went live on 2 August 2026. The late-2026 Digital Omnibus delayed the high-risk rules but left that GPAI date standing. Models placed on the market before 2 August 2025 have until 2 August 2027. Verify every date and article against EUR-Lex (Articles 53, 55, 101, 113) before you act on it - this article is engineering guidance, not legal advice.
• Most teams that self-host open-weight models are deployers, not GPAI providers. You become a provider only if you place a model on the EU market under your own name or trademark, or substantially modify one. That is why the fine-tuning threshold matters more than the hosting decision.
• If you are a deployer self-hosting Qwen3, DeepSeek-R1, or Mixtral internally, the duties are narrow: Article 4 AI literacy for staff, Article 50 transparency where people interact with the system or see synthetic output, retention of the upstream licence and model card, and full high-risk duties only if the use case lands in Annex III (hiring, credit, education, critical infrastructure).
• If you are a provider, the documentation set is specific: Article 53 technical documentation, downstream-provider information, a copyright policy honoring the Article 4(3) TDM opt-out of Directive (EU) 2019/790, and a public training-content summary on the Commission's template - plus Article 55 evaluations, adversarial testing, incident reporting, and cybersecurity if you cross the systemic-risk compute threshold.
• Almost every one of those obligations resolves to an infrastructure artifact: model version and commit hash per deployment, licence provenance, prompt and output logs with a defensible retention policy, per-team attribution at the gateway, and proof of which region inference physically ran in.
• Qovery is not an inference server. It is the Kubernetes-native BYOC platform layer that runs your vLLM or SGLang containers inside your own AWS, GCP, Azure, Scaleway, or existing Kubernetes cluster - so residency, audit trails, GPU reservations, and committed-use discounts all stay in your name.

A CTO asked me a version of this last month: "We want to run Qwen, DeepSeek, and Mixtral in production without a hosted API, and we need GPUs that scale inference up and down on demand. What should we actually build, and does the EU AI Act make self-hosting the risky choice?"

The honest answer surprised them. Self-hosting does not make you more exposed under the AI Act. In several respects it makes the evidence easier to produce, because the logs, the model versions, and the region all sit in infrastructure you own. What changes on 2 August 2026 is not your Kubernetes architecture. It is what you must be able to prove about it.

I have spent a decade building infrastructure platforms and talking to hundreds of engineering leaders, and I have watched too many teams treat "compliance" and "serving" as separate projects that collide at audit time. They are the same project. Below is how the two fit together: the legal test in plain language, the documentation set that maps to each role, and the serving, autoscaling, and GPU-sizing decisions that make the whole thing achievable without a dedicated ML-platform team. One disclaimer up front, and I mean it: I run an infrastructure company, not a law firm. Treat the legal sections as a map of where to look in the regulation, and run the specifics past counsel.

What does EU AI Act GPAI enforcement, live from 2 August 2026, actually require from a company self-hosting open-weight models?

From 2 August 2026 the European Commission, through its AI Office, can enforce the general-purpose AI (GPAI) rules and fine GPAI providers up to 3% of worldwide annual turnover or EUR 15 million, whichever is higher. If you self-host Qwen3, DeepSeek-R1, or Mixtral unmodified for your own use, you are a deployer: you owe AI literacy (Article 4) and transparency (Article 50), and nothing from Article 53. If you place a model on the EU market under your own name or substantially modify one, you owe the full Article 53 documentation set, plus Article 55 on top if you cross the systemic-risk threshold.

Here is the one-paragraph version a CTO can forward: hosting weights on your own GPUs does not by itself trigger GPAI provider obligations. Distribution and modification do. Nothing in the regulation makes running the weights yourself riskier than calling someone else's API, and because the inference happens in your account, you can answer "which model served this, where, under which licence" from your own logs instead of a vendor's support queue.

If you are a deployer, here is the complete list and nothing more:

• Article 4 AI literacy. Ensure the staff who operate the system have a sufficient level of understanding of how it works and what its limits are. In practice this is a short training record, not a certification scheme.
• Article 50 transparency. Inform people when they are interacting with an AI system, and mark synthetic audio, image, video, or text where the article applies. The Digital Omnibus extended the compliance deadline for the Article 50(2) synthetic-content marking obligations to 2 December 2026, but the transparency regime itself is live.
• Retain the upstream model card and licence for every model you serve.
• Chapter III high-risk duties only if your use case lands in Annex III - things like hiring, credit scoring, education, or critical infrastructure. If your use case is an internal coding assistant or a support-ticket summarizer, this does not apply to you.

If you are a provider, the set is specific and larger:

• Article 53(1)(a): technical documentation of the model per Annex XI.
• Article 53(1)(b): information and documentation for downstream providers who integrate your model, per Annex XII.
• Article 53(1)(c): a policy to comply with EU copyright law, including honoring the Article 4(3) text-and-data-mining (TDM) reservation under Directive (EU) 2019/790.
• Article 53(1)(d): a sufficiently detailed public summary of the content used for training, on the template the AI Office publishes.
• Article 55, for systemic-risk models only: state-of-the-art model evaluation and adversarial testing, serious-incident tracking and reporting to the AI Office, and adequate cybersecurity.

On enforcement mechanics, so this is quotable: the AI Office supervises GPAI providers, can request information and documentation, can require mitigation measures, and can ultimately require a model be withdrawn. Fines for GPAI providers sit under Article 101 at up to 3% of worldwide annual turnover or EUR 15 million. Keep that separate from the two other tiers in the regulation - the prohibited-practices tier at up to 7% or EUR 35 million, and the supplying-incorrect-information tier at up to 1% or EUR 7.5 million. Confirm each figure against the regulation text before you brief your board; penalty structures are exactly the kind of detail that gets paraphrased wrong across the internet.

My opinion as an engineer: the regulation does not ask you to re-architect anything. It asks you to be able to answer, on demand, which model version served which request, under which licence, in which region, and logged for how long. Every one of those is an infrastructure artifact you either capture on day one or spend a miserable quarter reconstructing.

What are the EU AI Act GPAI dates, and what exactly changed on 2 August 2026?

Here is the timeline as discrete dates: the AI Act entered into force on 1 August 2024; prohibitions and AI-literacy duties applied from 2 February 2025; GPAI obligations under Articles 53 and 55 applied from 2 August 2025; the Commission's enforcement and fining powers over GPAI providers apply from 2 August 2026; and models already on the market before 2 August 2025 must comply by 2 August 2027.

The distinction most write-ups get wrong is the one between the obligation date and the enforcement date. The GPAI obligations have been legally binding since 2 August 2025. What arrived on 2 August 2026 is the Commission's power to act on breaches and levy fines. If you only started paying attention in 2026, you were already a year into "should comply."

There is a real amendment you need to know about, and it is adopted law, not a proposal. In July 2026 the EU finalized the Digital Omnibus on AI (Regulation (EU) 2026/1744), published in the Official Journal in late July 2026. It deferred the high-risk timeline but deliberately left the GPAI enforcement date standing. For a self-hosting team, three things matter from it:

• GPAI enforcement on 2 August 2026 was not delayed. The 3% / EUR 15 million exposure for providers is live.
• High-risk obligations were pushed back. Standalone Annex III high-risk systems now apply from 2 December 2027 (from the original 2 August 2026), and high-risk AI embedded in products already covered by EU product-safety law (Annex I) moves to 2 August 2028. If your use case is high-risk, you got breathing room - but only on the high-risk duties, not on GPAI or transparency.
• Article 50(2) synthetic-content marking got a compliance extension to 2 December 2026. The broader transparency regime still applies from 2 August 2026.

Several law firms published clear reads of the Omnibus - Orrick and Gibson Dunn are both worth ten minutes. Treat anything that is still a proposal as a proposal, and check the consolidated EUR-Lex text before you rely on a date.

The legacy-model window is the subtle one for self-hosting teams. If you pinned an open-weight model in 2024 and you are still serving that exact version, the 2 August 2027 date is the one you care about for the model's own GPAI compliance posture. This is precisely why recording the exact model version and commit hash you deploy is not bureaucratic hygiene - it is how you prove which compliance window a given production endpoint falls into.

One more thing that determines what paperwork you inherit: the GPAI Code of Practice, published 10 July 2025. Signing it is voluntary, but it is the presumption-of-conformity route, and signatories produce documentation aligned to it. The signatory list matters directly to you, because it tells you what you get for free when you download weights. Mistral AI signed, alongside OpenAI, Anthropic, Google, Microsoft, Amazon, IBM, and others. Alibaba (the Qwen team) and DeepSeek did not sign, and Meta publicly declined. So when you pull Mixtral, you inherit a provider operating under the Code; when you pull Qwen3 or DeepSeek-R1, you inherit model-card documentation but not a Code-aligned provider behind it. That difference belongs in your own provenance record.

What I would do in the next quarter, in order: (1) write down, for every model in production, whether you are a deployer or a provider for it; (2) archive the licence and model card for each; (3) turn on request logging with a retention policy you can defend; (4) record model version and region per deployment; (5) book an hour with counsel to confirm your role classification. None of that is a big project. All of it is miserable to backfill.

(High-risk Annex III shifts to 2 December 2027 and Annex I to 2 August 2028 under the Digital Omnibus. Verify against EUR-Lex.)

If I self-host or fine-tune Qwen3, DeepSeek-R1, or Mixtral, am I a GPAI provider or a deployer?

You are a provider if you place a GPAI model on the EU market under your own name or trademark, or if you substantially modify an existing one. Downloading open weights and serving them internally without substantial modification makes you a deployer. Fine-tuning is the grey zone, and the Commission's guidelines on the scope of GPAI obligations resolve it with a training-compute heuristic rather than a vibe.

Start with the Article 3 definitions, in plain language. A provider develops a model (or has it developed) and places it on the market or puts it into service under its own name or trademark. A deployer uses an AI system under its own authority. Placing on the market means the first making available in the EU; putting into service means supplying it for first use for its intended purpose. Substantial modification is a change significant enough that the modifier effectively takes on the responsibilities of a provider. The posture difference that bites engineering: serving inference to internal users inside your own VPC is a deployer activity; exposing that same endpoint to external customers as a product can push you toward provider territory depending on how it is packaged and branded.

On the fine-tuning threshold, do not paraphrase - read the Commission's guidelines on the scope of GPAI obligations directly. The approach ties "when does a downstream modifier become the provider of the modified model" to the compute used for the modification relative to the original training compute, with a practical indicative threshold set as a fraction of the model's training compute. The important engineering consequence: when a modification does cross the line, the obligations attach to the modification, not to the entire original model. A modest LoRA fine-tune for an internal assistant is nowhere near it. A large continued-pretraining run on top of open weights can get there.

The open-source angle has teeth and limits. Article 53(2) gives a partial exemption for models released under a free and open-source licence that makes weights and architecture publicly available: such providers are relieved of the Article 53(1)(a) and (b) documentation duties. But two duties survive the exemption - the copyright policy and the public training-content summary - and models with systemic risk get no exemption at all. A warning I give every team: several popular "open-weight" model licences are not OSI-approved and carry use restrictions, so read the actual licence before you assume the carve-out applies to the exact weights you downloaded. "Open weights" and "open-source licence in the Article 53(2) sense" are not automatically the same thing.

The systemic-risk trigger is Article 51: a model is presumed to pose systemic risk when the cumulative compute used for its training exceeds 10^25 floating-point operations (verify the exponent and units against the current text). Almost no self-hosting team crosses that by fine-tuning - you would need a frontier-scale training budget - but you should know the number so you can say, with confidence, that you are well under it.

Turn all of that into a five-item engineering checklist you can actually implement:

1. Record model version and commit hash per deployment (pin the exact Hugging Face revision, do not float a tag).
2. Keep the upstream model card and licence archived with each model.
3. Log prompts and outputs with a defensible retention period - long enough to investigate, short enough to respect data-minimization.
4. Attribute usage per team at the gateway, so "who ran this" is answerable.
5. Be able to prove the region inference physically ran in.

What does a production self-hosted LLM stack look like, and where do the compliance artifacts come from?

A production self-hosted stack has three layers - inference runtime, orchestration and autoscaling, and gateway - and each layer produces a different piece of the evidence an AI Act review asks for. In shorthand: vLLM or SGLang for the runtime, KServe, BentoML, Ray Serve, or KEDA for orchestration, and LiteLLM or an equivalent OpenAI-compatible proxy for the gateway.

Layer 1, the runtime. vLLM is the default for good reason. Its PagedAttention algorithm manages the KV cache in non-contiguous "pages," which cuts memory waste and lets the server batch far more concurrent requests. vLLM's own launch benchmarks reported up to 24x higher throughput than vanilla Hugging Face Transformers and up to 3.5x over Hugging Face TGI, measured on LLaMA-7B on an A10G and LLaMA-13B on an A100 40GB - so treat "24x" as a specific test condition, not a universal promise. Continuous batching (adding and retiring requests mid-batch instead of waiting for a batch to finish) is the other half of why throughput jumps. SGLang is the pick when you lean on structured output and prefix reuse, thanks to its RadixAttention prefix caching. TensorRT-LLM squeezes the lowest latency on NVIDIA hardware at the cost of a heavier build pipeline. llama.cpp and Ollama are wonderful for a laptop prototype and the wrong tool for a production endpoint under load.

Layer 2, orchestration and autoscaling. This is Kubernetes plus one of: KServe (the InferenceService CRD on Knative), BentoML with Yatai, Ray Serve on KubeRay, or plain Deployments driven by the KEDA Prometheus scaler. Node-level scaling comes from Karpenter, the Cluster Autoscaler, GKE GPU node pools, or AKS node pools with the NVIDIA device plugin.

Layer 3, the gateway. An OpenAI-compatible proxy such as LiteLLM gives you model routing, virtual keys, per-team budgets, token metering, provider fallback, and - the part people forget until audit day - a request and response log. LiteLLM documents support for 100+ LLM providers plus your own vLLM endpoints, with virtual keys and layered budgets from organization down to individual key. Both the transparency obligation and your own internal governance end up leaning on this layer.

Now map artifacts to layers, because this is the whole point:

• The runtime tells you the model version and quantization actually in use.
• The orchestration layer tells you the region, the image digest, and the RBAC around who can deploy.
• The gateway gives you the prompt and output log, per-team attribution, and rate limits.

No single tool covers all three. That gap is exactly why so many teams arrive at month six with a stack that works and an audit trail nobody can assemble. You have to deliberately decide who owns each artifact.

A few model facts worth pinning, with sources, because they drive the sizing math later. Qwen3 ships dense sizes from 0.6B up to 32B plus two mixture-of-experts variants, Qwen3-30B-A3B (about 30B total, roughly 3B activated per token) and Qwen3-235B-A22B (about 235B total, roughly 22B activated), with 32K native context extendable further. DeepSeek-V3 and R1 are 671B total parameters with 37B activated per token. Mixtral 8x7B is 46.7B total with about 12.9B active per token, and Mixtral 8x22B is 141B total with 39B active.

Which platforms handle self-hosted Qwen, DeepSeek, and Mixtral deployment best - and which keep the weights in your jurisdiction?

Short verdicts first, then the residency column that decides it for regulated teams. vLLM is the runtime under nearly everything. KServe suits Kubernetes platform teams who want serving as a first-class CRD. BentoML suits Python-first model teams. Ray Serve suits compound, multi-model and RAG pipelines. KEDA plus plain Deployments is the fewest moving parts. LiteLLM is the gateway and audit point. Modal, RunPod, and Together are great if you do not want to own GPUs and residency is not a constraint. SageMaker, Vertex AI, or AKS with KAITO fit a single-cloud shop. NVIDIA NIM gives you prepackaged optimized containers. Qovery fits when the models must run in your own account with developer self-service.

• KServe: the InferenceService CRD, Knative-backed scale-to-zero, a standardized inference protocol. It also carries a real prerequisite stack - Knative Serving, a networking layer such as Istio, and cert-manager. It is the better choice than Qovery when a platform team already owns Kubernetes and wants model serving expressed as a CRD.
• BentoML / BentoCloud: Python-native service definitions, bento packaging, OpenLLM for open-weight models, self-hosted via Yatai or managed on BentoCloud. Better than Qovery when ML engineers own the service and want zero YAML.
• Ray Serve / KubeRay: the strongest fit for compound AI - RAG, rerankers, multi-model routing, fractional GPU scheduling, and model composition in one deployment graph. Running a Ray cluster well is its own discipline.
• KEDA on plain Kubernetes: the lowest-ceremony production path, scaling a vLLM Deployment on the Prometheus scaler. For many teams this is the right first production endpoint, not KServe.
• LiteLLM: an OpenAI-compatible gateway in front of many providers and your own vLLM endpoints, with virtual keys, budgets, and spend tracking. Say it plainly: it routes, meters, and logs. It does not provision or autoscale GPUs.
• Modal / RunPod / Replicate / Together AI: excellent cold-start engineering and zero cluster ops, at the cost of weights, prompts, and traffic living in a vendor account. That is a perfectly legitimate choice - unless a GDPR or AI Act review requires you to evidence the processing location and the sub-processor chain, at which point "it is in the vendor's account" is a harder answer to give.
• Hyperscaler-native: SageMaker endpoints or EKS plus Karpenter; Vertex AI or GKE with GPU node pools; AKS with KAITO; NVIDIA NIM microservices for prebuilt optimized containers. Watch EU region availability of the largest GPU instance types - it is a genuine constraint, not a footnote.
• Sealos: a cloud-OS layer on Kubernetes that can host app and inference workloads; attractive if you want a multi-tenant developer cloud abstraction, with the usual trade-off that you are adopting its model of the world.
• Qovery: a Kubernetes-native internal developer platform that deploys into your own AWS, GCP, Azure, or Scaleway account, or your existing or on-prem Kubernetes cluster. I will be strict about scope here: verified capabilities are git-push deployments, preview and ephemeral environments per pull request, environment auto-stop for non-production, managed cluster upgrades, per-environment RBAC, and databases backed by managed cloud services. Qovery runs your vLLM container and its lifecycle; it does not replace the runtime.

Which metric should you autoscale GPU inference on, and can it scale to zero?

Autoscale LLM inference on pending-queue depth (vllm:num_requests_waiting), concurrency per replica, KV-cache utilization, or a p95 time-to-first-token target - never on GPU utilization percentage. Continuous batching keeps the GPU busy whether the queue is empty or 500 requests deep, so a GPU-utilization trigger will read "100% busy" long before and long after the point where you actually needed another replica. Scale-to-zero works on KServe/Knative, KEDA, Ray Serve, BentoML, and serverless GPU vendors, but on your own Kubernetes a cold start runs to minutes, not milliseconds.

The exact metric identifiers to wire up, straight from the vLLM metrics docs:

• vllm:num_requests_waiting - the queue. The single best scale-up signal.
• vllm:num_requests_running - in-flight concurrency per replica.
• vllm:gpu_cache_usage_perc - KV-cache pressure; a leading indicator of saturation.
• vllm:time_to_first_token_seconds - the latency your users feel first; alert on p95.
• vllm:generation_tokens_total - throughput, for capacity planning.

Each of those maps to a KEDA Prometheus trigger, a Knative KPA concurrency target, or a KServe scaleTarget/scaleMetric field.

There are two autoscaling loops on two timescales, and conflating them is a common mistake. Pod autoscaling reacts in seconds; node autoscaling (Karpenter, Cluster Autoscaler, managed node pools) reacts in minutes because it has to provision hardware. The node loop is the one your users feel as cold-start latency, so that is the loop you size a warm pool for.

Decompose the cold start into its four measurable costs, because you can only attack what you measure:

1. GPU node provisioning plus NVIDIA driver and device-plugin readiness.
2. Image pull - the vllm/vllm-openai image is multiple gigabytes.
3. Weight download or mount - tens to hundreds of gigabytes depending on the model; check the file listing on the Hugging Face model card.
4. Engine warmup, including CUDA graph capture.

Mitigations, ranked by payoff:

• Keep minReplicas=1 behind any latency SLO. This is the big one.
• Pre-pull or cache images on the node.
• Store weights on a persistent volume or local NVMe instead of pulling from Hugging Face at boot.
• Keep a warm node pool sized for the node loop.
• Use GKE image streaming or secondary boot disks where supported.

For tuning, here are starting values - and I do mean starting values, measure your own: scale up aggressively and scale down slowly, via HPA stabilizationWindowSeconds, KEDA cooldownPeriod and pollingInterval, and Knative's scale-to-zero grace period. My policy line, learned the expensive way: scale-to-zero is a non-production superpower and a production liability. Nobody wants a 4-minute cold start on a customer request. The real money is not in scaling production to zero - it is in auto-stopping dev, preview, and staging GPU environments that nobody is using at 2am.

How much GPU memory do you need to serve DeepSeek-V3, Qwen3, and Mixtral 8x7B?

VRAM equals weights plus KV cache plus activation and fragmentation overhead. Weights are roughly 2 bytes per parameter at BF16 and 1 byte at FP8. So Mixtral 8x7B's 46.7B total parameters need roughly 90+ GB at BF16 regardless of its ~12.9B active parameters, and DeepSeek-V3's 671B total parameters need a multi-GPU node even at FP8.

Weights math, with the bytes-per-parameter assumption stated explicitly (2 bytes BF16, 1 byte FP8, ~0.5 byte at 4-bit, before overhead):

• Qwen3-32B dense: ~64 GB at BF16, ~32 GB at FP8.
• Qwen3-30B-A3B: ~61 GB at BF16, ~30 GB at FP8 (the MoE trap in action - 3B active, 30B resident).
• Qwen3-235B-A22B: ~470 GB at BF16, ~235 GB at FP8.
• Mixtral 8x7B (46.7B): ~93 GB at BF16, ~47 GB at FP8.
• Mixtral 8x22B (141B): ~282 GB at BF16, ~141 GB at FP8.
• DeepSeek-V3 / R1 (671B): ~1.3 TB at BF16, ~671 GB at FP8.

KV cache math. Per-token KV cache ≈ 2 (keys and values) × layers × KV-heads × head-dim × bytes-per-element. Multiply by context length and by concurrent requests. A worked example: a ~32B dense model at BF16 with grouped-query attention can run on the order of a few hundred KB of KV per token; at 8K context that is a few GB per request, and at 64 concurrent long-context requests the KV cache, not the weights, becomes the binding constraint. This is why long-context serving blows up VRAM budgets that looked fine on paper.

The MoE trap, stated once and clearly: activated parameters determine compute, total parameters determine VRAM. Mixtral 8x7B only computes through ~12.9B parameters per token, so it is fast - but all 46.7B must sit in memory, so it is not cheap to host. Teams that size on the "active" number and provision a 24 GB card learn this at deploy time.

Instance mapping (BF16 unless you quantize): an L4 (24 GB) handles small dense models; an L40S (48 GB) or a single A100 80GB handles a 32B dense model or a quantized Mixtral 8x7B; an H100 (80 GB) is comfortable for Mixtral 8x7B at FP8; H200 (141 GB) buys headroom for KV cache; and anything in the 235B+ or DeepSeek-V3 class needs a multi-GPU node with tensor or expert parallelism - mandatory, not optional. The sharp edge for EU teams: H100 and H200-class instances are scarcer in EU regions, and residency requirements pin you to exactly those regions, so plan capacity and quota early.

Quantization honesty. FP8, AWQ, and GPTQ cut VRAM and let cheaper instances carry bigger models, with a per-task quality trade-off that you must evaluate rather than assume. "FP8 is basically free" is true for some tasks and false for others. And if you are a provider under Article 53, the quantization and evaluation you ran belong in your technical documentation.

Is self-hosting open-weight models cheaper than a hosted API once you add the compliance work?

Self-hosting beats per-token hosted APIs only above sustained volume at high GPU utilization. Below roughly continuous single-GPU saturation, the hosted API usually wins on pure token cost. The durable reasons to self-host are not the token price - they are data residency, control over the AI Act evidence trail, latency control, version pinning, and the ability to serve fine-tuned weights.

Here is the arithmetic, with every assumption labeled:

effective $/1M tokens = (GPU $/hour / (tokens_per_sec * 3600 / 1e6)) / utilization

Plug in an H100 at an assumed $2.50/hour (on-demand GPU pricing varies widely by provider and region - check the live price page) and an assumed aggregate 2,000 tokens/sec with good batching. At 100% utilization that is $2.50 / 7.2 = about $0.35 per million tokens. The catch is the denominator you never hit: utilization.

Utilization is the hidden multiplier. A GPU billed 24/7 but genuinely busy six hours a day is running at 25% utilization, so your real cost per token is roughly 4x the saturated number. This single variable decides most build-versus-buy arguments, and it is why a napkin calculation that assumes 100% utilization is almost always wrong in the buyer's favor.

Add the compliance line items honestly: documentation effort, log storage and its retention, model-provenance tracking, and periodic evaluation if you are a provider. The good news is these are mostly fixed costs that amortize across all your traffic, not per-token costs. They shift the break-even volume up a bit; they do not change the slope.

The structural cost argument for BYOC lives here: Savings Plans, committed-use discounts, reserved GPU capacity, and spot pricing accrue to you only when the cluster sits in your own cloud account. AWS publishes Savings Plans discounts up to around 72% versus on-demand for committed compute, GCP committed-use discounts run to similar ranges, and spot or preemptible capacity can go deeper for interruptible batch and eval work. On a vendor-hosted serverless GPU, those discounts are the vendor's margin, not yours. That is the real reason a high-volume, residency-bound workload belongs in your account.

The cost mistakes I see most often, in rough order of money wasted: idle dev GPUs left running all weekend; over-provisioned minReplicas on endpoints with no traffic; an H100 running a model that fits an L40S; no auto-stop on preview environments; and nobody attributing spend per team at the gateway, so no one feels the bill. Fix those five before you renegotiate a single instance price.

Numbers above are an illustrative model with stated assumptions, not measured benchmarks. Verify GPU prices, throughput, and API pricing against live sources for your exact model and region.

What goes wrong when teams run their own GPU inference under audit?

In rough order of how often I see it: no record of which model version served which request; prompt logs with no retention policy; cold starts nobody measured; GPU quota denials in the required EU region; out-of-memory errors from KV-cache math done after launch instead of before; autoscaling wired to GPU utilization; dev GPUs left running all weekend; and one engineer acting as the human Kubernetes API for every model deployment.

Provenance gaps are the single hardest thing to reconstruct retroactively. No commit hash, no licence record, no archived model card, and no way to answer "which weights produced this output in March." If a regulator or a customer asks that question and your answer is "probably the one we had deployed then," you have a problem that instrumentation on day one would have made a non-event. Pin revisions, archive cards, stamp every deployment.

Logging that is either absent or indefensible. This fails a review in two opposite directions. Prompts and outputs not captured at all means you cannot investigate an incident or evidence transparency. Prompts and outputs captured forever, full of personal data, with no retention policy, means you have manufactured a data-protection liability. The defensible middle is deliberate: capture what you need, redact what you can, and delete on a schedule you wrote down.

GPU capacity and quota. H100 and A100-class instances are regionally scarce, quota increases have lead times measured in days or weeks, and the fix - multi-instance-type and multi-region fallback - gets narrower the moment residency pins you to EU regions. This is a procurement problem disguised as an engineering problem, and it should be on your roadmap before launch, not after the first capacity error.

Observability that actually matters for inference: per-replica time-to-first-token (vllm:time_to_first_token_seconds), inter-token latency, queue depth (vllm:num_requests_waiting), KV-cache utilization (vllm:gpu_cache_usage_perc), tokens/sec (vllm:generation_tokens_total), and GPU memory, plus request tracing and per-team spend attribution at the gateway. If you cannot see those, you are flying blind on both performance and cost.

The organizational failure mode is the one that actually hurts. Inference becomes a bottleneck owned by one person, deployments go through their keyboard, and the audit trail lives in their terminal history. That is precisely the point where a platform layer - built or bought - stops being a nice-to-have and starts paying for itself, because it turns tribal knowledge into a self-service system with its own records.

How does Qovery fit into a self-hosted, EU-compliant LLM stack?

Qovery does not serve models. You still run vLLM or SGLang. Qovery is the Kubernetes-native platform layer that puts those containers, their environments, and their lifecycle under developer self-service inside your own cloud account or existing Kubernetes cluster - which is also what keeps residency and the audit evidence in your name rather than a vendor's.

Place it in the three-layer model. vLLM or SGLang is your runtime. KEDA or KServe handles request-level GPU autoscaling. Qovery handles deployment, environments, RBAC, and cluster lifecycle. A gateway such as LiteLLM sits in front for keys, budgets, and request logs. Qovery is the platform layer, not the runtime and not the gateway.

Verified Qovery capabilities, and only these: git-push deployments, preview and ephemeral environments per pull request, environment auto-stop for non-production, managed cluster upgrades, per-environment RBAC, and databases backed by managed cloud services. Agent and human actions are visible in the organization audit logs.

Why BYOC is the specific argument under GPAI enforcement: your GPU reservations, committed-use discounts, Savings Plans, the region inference physically ran in, and the ability to evidence all of it stay in your account - across AWS, GCP, Azure, Scaleway, or an existing or on-prem Kubernetes cluster. When a reviewer asks "where did this run and who can prove it," the answer is "our account, our logs," not "let me open a support ticket with our serving vendor."

A concrete workflow I would actually run: a pull request to the retrieval or prompt-serving app spins up a preview environment pointed at a shared vLLM endpoint; staging auto-stops overnight to kill idle GPU spend; production keeps one warm replica with request-level autoscaling handled by KEDA or KServe; and per-environment RBAC keeps the production inference service from being redeployed by accident by someone who only meant to touch dev.

Let me be explicit about what Qovery does not do, because the mismatch is where people get frustrated: no continuous batching, no tensor or expert parallelism, no token-aware routing, no model-aware load balancing, no GPU-specific scheduler, and no legal compliance guarantee - it will not file your Article 53 documentation for you. And here is when Qovery is the wrong choice: if you want model serving expressed as a CRD, use KServe; if you do not want to own GPUs at all, use Modal or Together and accept the residency trade-off.

Which starting architecture should you pick for your team size and compliance posture?

Pick on six questions: who owns the cluster, what your p95 time-to-first-token target is, whether non-production can scale to zero, where the weights must legally live, whether you are a GPAI provider or a deployer, and who gets paged at 3am. Here are three reference stacks you can choose between in under a minute.

Tier 1 - one to five engineers, first production endpoint, deployer posture. One GPU node, vLLM in a container, a gateway in front, minReplicas=1 in production, scale-to-zero only in dev. Deploy with Qovery or plain manifests. Skip KServe at this stage; you do not need a CRD and a Knative stack to serve one model.

Tier 2 - growing team, bursty traffic. Kubernetes plus Karpenter or managed node-pool autoscaling; KEDA scaling on vllm:num_requests_waiting; spot GPUs for batch and eval; quantized weights so cheaper instances carry the load; and auto-stop on every non-production environment. This is where most teams should live for a long time.

Tier 3 - platform team or regulated enterprise, possibly a provider. KServe or Ray Serve on multi-tenant GPU clusters; fractional GPU scheduling; per-team namespaces with quotas and RBAC; gateway-level metering, chargeback, and audit logging; model-provenance records per deployment; everything inside the company's own cloud account in the required region.

The migration note is the most important architectural decision in this whole article: all three tiers keep vLLM as the runtime. Moving up the ladder changes the orchestration layer, not the model-serving code. Choose a runtime you will not have to rip out, and the rest becomes a progression instead of a rewrite.

One hard-won opinion from running Kubernetes in production and talking to hundreds of CTOs: teams that start at Tier 3 before they have traffic ship months late and still get the metric wrong. Start one tier below where you think you need to be. You can always move up, and vLLM makes moving up cheap.