How to Self-Host AI Inference on Hetzner's GEX63 Blackwell GPU Server with Kubernetes

Key Points:

• 96 GB of VRAM at a flat monthly price changes the math for a small team. Hetzner's GEX63 is a dedicated (non-virtualized) GPU server built around an NVIDIA RTX PRO 6000 Blackwell Max-Q with 96 GB of GDDR7 ECC VRAM, billed at a flat €999 per month plus a one-time €999 setup fee (Hetzner pressroom). Verify the current price and spec before you commit.
• That VRAM serves roughly a 70B model at 4-bit, a 32B model at BF16, or several 7B-14B models side by side - and the KV cache, not the weights, is usually the first thing to run out.
• The shortest working path is five steps: provision the GEX63 with Ubuntu 24.04, install K3s single-node, install the NVIDIA GPU Operator with Helm, confirm the node advertises nvidia.com/gpu: 1, then deploy vLLM behind a Service, a TLS ingress, and auth.
• Blackwell is compute capability 12.0 (sm_120). It needs a recent driver, CUDA 12.8 or newer, and a vLLM/PyTorch build compiled for sm_120. Old images fail with no kernel image is available for execution on the device. That error means your binary has no sm_120 kernels, not that your GPU is broken.
• A flat-rate dedicated server is far cheaper than on-demand hyperscaler GPUs for always-on inference, but you inherit driver pinning, GPU health checks, node upgrades, and a single point of failure. Compute your break-even GPU-hours first.
• Qovery installs on your own Kubernetes cluster - including a self-managed K3s or RKE2 cluster on Hetzner, or your own AWS, GCP, Azure, or Scaleway account - so your team gets git-push deploys, preview environments, and per-environment RBAC on the GPU node without hand-writing a manifest per model.

Hetzner just put a Blackwell GPU with 96 GB of VRAM on a dedicated server for €999 a month, flat, in Helsinki (GEX63 product page). For a small team that wants to serve its own Llama, Qwen, or Mistral inference, that single data point changes the build-versus-rent decision. An always-on GPU this size at a hyperscaler runs well into four figures a month before you touch egress, and here it is a fixed line item you can forecast to the cent.

I run infrastructure for a living, so let me be clear about the trade I am describing. You get a box with more VRAM-per-euro than anything on-demand at the big three, and in exchange you own the driver stack, the node, and the pager. This is a genuine engineering walkthrough: real commands, real YAML, real prices. If you read to the end and never touch Qovery, it should still have been worth your time.

Let me walk through what the box can do, how to put Kubernetes and the NVIDIA stack on it, how to serve a model, the one error that will eat your afternoon, what it costs against EKS/GKE/AKS, and what breaks in month three.

What is Hetzner's GEX63, and which models can 96 GB of VRAM actually serve?

The GEX63 is a dedicated Hetzner GPU server built around an NVIDIA RTX PRO 6000 Blackwell Max-Q with 96 GB of GDDR7 ECC VRAM, sold at a flat monthly price with a one-time setup fee. That is enough to serve roughly a 70B-class model at 4-bit, a 32B-class model at BF16, or several smaller models on one box.

Here is the spec sheet, from Hetzner's own pressroom announcement and the product page:

• GPU: NVIDIA RTX PRO 6000 Blackwell Max-Q, 96 GB GDDR7 ECC
• CPU: Intel Core Ultra 7 265 ("Arrow Lake"), 20 cores (8 performance + 12 efficiency)
• RAM: 128 GB DDR5 UDIMM
• Storage: 2x 1.92 TB datacenter-edition Gen4 NVMe
• Network: 1 Gbit/s port, unlimited traffic included
• Price: €999/month + €999 one-time setup
• Location: Helsinki (HEL1)

One detail that drives every software choice later: this is a Blackwell GPU, which NVIDIA lists as compute capability 12.0, or sm_120 in CUDA terms (NVIDIA CUDA GPUs table). The RTX PRO 6000 Blackwell family carries 96 GB of GDDR7 and fifth-generation Tensor cores (NVIDIA RTX PRO 6000 Blackwell Server Edition). Note that Hetzner ships the Max-Q variant, which trades peak power draw for a lower, rack-friendly TDP. The compute capability is identical to the higher-power editions, so your software versioning is the same.

How to budget 96 GB of VRAM

The rule of thumb is simple arithmetic. Model weights cost roughly 2 bytes per parameter at BF16, about 1 byte at FP8, and about 0.5 byte at 4-bit (AWQ or GPTQ). On top of the weights sits the KV cache, which grows with context length times concurrency, and vLLM claims whatever VRAM is left after the weights for that cache via --gpu-memory-utilization (vLLM memory docs).

Two worked examples on 96 GB:

• Qwen2.5-32B at BF16: about 64 GB of weights, leaving roughly 30 GB for the KV cache. Comfortable for moderate context and concurrency.
• Llama-3.3-70B at 4-bit: about 40 GB of weights, leaving roughly 55 GB for the cache. This is where 96 GB earns its keep.

Be honest about the limits. One GPU means no multi-node training, no unquantized 70B at long context, and no economic sense for a workload that sits idle 90 percent of the month. Dedicated servers bill monthly, so idle hours are paid hours, which is the opposite incentive from a per-second cloud GPU instance. If your traffic is spiky, this is the wrong tool.

Should you run Kubernetes on a single GPU server, or is Docker Compose enough?

For one model on one box, Docker plus the NVIDIA Container Toolkit is enough, and it gets you an OpenAI-compatible endpoint in an afternoon. Kubernetes earns its keep the moment you have more than one model, more than one team, or you need rolling updates, readiness probes, GPU-aware scheduling, and a path to a second node.

The case for plain Docker Compose is fewer moving parts and no control plane to upgrade. The case for Kubernetes is declarative rollouts, startup and readiness probes for a model server that takes minutes to load, resource limits, Secrets, ingress plus TLS, and namespace-level multi-tenancy. If this box will serve one team and one model forever, use Docker. If it is the first node of something, use Kubernetes.

On distribution, I recommend K3s for a single GEX63. It is a single binary that bundles containerd, a service load balancer, and Traefik, which is the best fit for one or two nodes (K3s quick-start). RKE2 is the hardened, CIS-aligned option for regulated environments, and kubeadm gives you the most control and the most maintenance. For a dedicated inference box, K3s is the sane default.

Because K3s ships containerd, you do not need Docker at all. Use the NVIDIA GPU Operator rather than hand-editing containerd config, and note that the operator officially supports K3s and RKE2 (GPU Operator getting started, RKE2 GPU operators).

One pitfall with teeth: without a device plugin, the GPU is not an allocatable Kubernetes resource. The scheduler will happily place ten pods on your one GPU, and they will fight over VRAM until they all OOM. The GPU Operator installs that device plugin for you, which is the whole point.

How do I install Kubernetes and the NVIDIA GPU Operator on a GEX63?

Install Ubuntu 24.04 through Hetzner's installimage, install K3s single-node, install the NVIDIA GPU Operator with Helm, then verify the node reports nvidia.com/gpu: 1 and an nvidia-smi pod prints the RTX PRO 6000 with 96 GB. Four steps, roughly 30 minutes.

Step 0 - OS baseline. Boot the Hetzner rescue system and run installimage to lay down Ubuntu 24.04 LTS (or Debian 12). Blackwell needs CUDA 12.8 or newer and a driver branch that supports sm_120, so do not pin an old driver out of habit (CUDA Toolkit release notes).

Step 1 - K3s. If you plan to bring your own ingress controller, disable the bundled Traefik:

curl -sfL https://get.k3s.io | sh -s - --disable traefik
# copy the kubeconfig out and point kubectl at it
sudo cat /etc/rancher/k3s/k3s.yaml   # replace 127.0.0.1 with the server IP

Then lock down port 6443. The Kubernetes API should never face the public internet - put it behind a firewall rule, a VPN, or a Hetzner private network.

Step 2 - GPU Operator. Add the Helm repo and install. The one real decision is whether the operator manages the driver (driver.enabled=true) or uses a driver you pre-installed on the host:

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update
helm install --wait gpu-operator nvidia/gpu-operator \
  -n gpu-operator --create-namespace \
  --set driver.enabled=true

On a dedicated server where you control the kernel, I lean toward letting the operator manage the driver, because it keeps the driver and the container runtime in lockstep and survives a kubelet restart cleanly. The catch is that the driver container compiles against your running kernel, so it needs matching kernel headers and it will not load an unsigned module if Secure Boot is on. If your distro reboots into a new kernel after an unattended upgrade, the driver container has to rebuild. If you would rather avoid that moving part on a box that rarely reboots, pre-install the driver on the host and set driver.enabled=false. Either works; just pick one deliberately. The operator supports K3s and RKE2 directly, so you do not hand-edit containerd (GPU Operator install docs).

Step 3 - Verify. Confirm the GPU is allocatable, then prove it end to end:

kubectl get nodes -o json | jq '.items[].status.allocatable'   # expect "nvidia.com/gpu": "1"

kubectl run nvidia-smi --rm -it --restart=Never \
  --image=nvidia/cuda:12.8.0-base-ubuntu24.04 \
  --overrides='{"spec":{"runtimeClassName":"nvidia"}}' \
  -- nvidia-smi

That second command must print "RTX PRO 6000" and 96 GB. If it does not, stop here and fix it before you deploy a model.

Step 4 - Observability from day one. The GPU Operator bundles the DCGM exporter (GPU Operator components). Scrape GPU utilization, VRAM used, temperature, and Xid errors into Prometheus now, not after your first outage. A GPU that is silently throttling or throwing ECC errors looks identical to "slow model" until you have the metrics.

Pitfalls, with fixes:

• Secure Boot blocks the unsigned driver module. Disable Secure Boot on the serving node or sign the module.
• Kernel headers drift after an unattended-upgrade reboot, and the driver container cannot rebuild. Pin the kernel and disable unattended reboots (more on this later).
• The GPU Operator validator pod is stuck in Init. It is almost always the driver container failing to compile; check its logs.
• Your pods do not get the GPU. The nvidia RuntimeClass is not set on the pod spec. The device plugin needs it.

How do I deploy vLLM (or Ollama) as an inference endpoint on that cluster?

Deploy a vLLM Deployment that requests nvidia.com/gpu: 1, mounts a persistent volume at the Hugging Face cache, has a startupProbe with a long failureThreshold, and sits behind a ClusterIP Service exposing the OpenAI-compatible API on port 8000 through an authenticated ingress. Here is the manifest that works:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-qwen
  namespace: inference
spec:
  replicas: 1
  selector:
    matchLabels: { app: vllm-qwen }
  template:
    metadata:
      labels: { app: vllm-qwen }
    spec:
      runtimeClassName: nvidia
      containers:
        - name: vllm
          # use an image tag that includes sm_120 kernels and CUDA 12.8+
          image: vllm/vllm-openai:latest
          args:
            - "--model=Qwen/Qwen2.5-32B-Instruct"
            - "--served-model-name=qwen"
            - "--max-model-len=16384"
            - "--gpu-memory-utilization=0.90"
            - "--tensor-parallel-size=1"
          ports:
            - containerPort: 8000
          env:
            - name: HF_TOKEN
              valueFrom:
                secretKeyRef: { name: hf-token, key: token }
          resources:
            limits:
              nvidia.com/gpu: 1
          volumeMounts:
            - { name: hf-cache, mountPath: /root/.cache/huggingface }
          startupProbe:
            httpGet: { path: /health, port: 8000 }
            periodSeconds: 10
            failureThreshold: 60      # up to 10 min to load a big model
          readinessProbe:
            httpGet: { path: /health, port: 8000 }
            periodSeconds: 10
      volumes:
        - name: hf-cache
          persistentVolumeClaim: { claimName: hf-cache }
---
apiVersion: v1
kind: Service
metadata: { name: vllm-qwen, namespace: inference }
spec:
  selector: { app: vllm-qwen }
  ports: [{ port: 8000, targetPort: 8000 }]

A few things that matter more than they look:

• The args are the config. --model and --served-model-name set what you load and what clients call it; --max-model-len caps context; --gpu-memory-utilization controls how aggressively vLLM claims VRAM for the KV cache; --quantization awq (or gptq) if you are serving a quantized checkpoint; --tensor-parallel-size 1 because you have one GPU.
• Cache the weights on a volume. Mount a local PV or hostPath at /root/.cache/huggingface so a pod restart does not re-download tens of gigabytes. Pass HF_TOKEN from a Secret for gated repos.
• Probes done right. Use a startupProbe on /health with a high failureThreshold, because a large model can take minutes to load. Pair it with a readinessProbe so traffic never reaches a loading pod. If you rely on a default liveness probe instead, Kubernetes will kill-loop the pod before the weights finish loading, and you will never get a healthy endpoint.
• Never expose it without auth. Put the Service behind ingress-nginx or Traefik, terminate TLS with cert-manager, and require an API key or an auth proxy. This is not paranoia: Oligo Security found nearly 10,000 internet-facing Ollama instances, many unauthenticated, and attackers actively scan for them to run free inference on someone else's GPU (Oligo Security research). An open inference endpoint on a public IP gets found and abused within days.

Which server should you actually run? vLLM is my default for throughput and OpenAI-API compatibility. Ollama is the easiest path and great for a grab-bag of small models, but weaker on concurrent throughput. Hugging Face TGI and SGLang are strong alternatives; NVIDIA NIM needs an NVIDIA AI Enterprise entitlement; llama.cpp is the hybrid CPU/GPU fallback.

Measure it yourself. Run vLLM's bundled benchmark_serving.py at concurrency 1, 8, and 32 and record tokens per second and time-to-first-token on your model and your context length. I will not quote a tokens/sec figure for this GPU here, because I have not found a vendor-published benchmark for the RTX PRO 6000 Blackwell on vLLM that I would stand behind. Community guides report figures in the low hundreds of tokens per second for a mid-size model, but treat those as a starting hypothesis, not a spec. Publish your own numbers; they are the only ones that describe your workload.

Why does my vLLM container fail with "no kernel image is available for execution on the device" on Blackwell?

Because the container's PyTorch/vLLM build has no kernels compiled for compute capability 12.0 (sm_120). The GPU is fine. The binary is too old. Fix it by using a vLLM image and CUDA runtime built for sm_120, on a driver branch that supports Blackwell.

Here is the mechanism in two sentences. CUDA binaries ship per-architecture machine code (cubins), optionally plus forward-compatible PTX that the driver can JIT-compile at load time. If the binary contains neither an sm_120 cubin nor PTX the driver can forward-compile, the runtime refuses to launch the kernel and you get that exact error.

Blackwell/sm_120 support requires CUDA 12.8 or newer and a recent PyTorch built against it; older PyTorch releases only compiled kernels up to sm_90 (CUDA Toolkit release notes). vLLM support for the RTX PRO 6000 Blackwell is tracked in the project's own forums and issues, and the practical fix is a current image on the cu128 build line (vLLM forum: RTX 6000 Blackwell support, vLLM issue #31085, vLLM installation docs).

Diagnose it in three commands:

nvidia-smi                                             # host sees the GPU + driver version
python -c "import torch; print(torch.cuda.get_device_capability())"   # want (12, 0)
python -c "import torch; print(torch.cuda.get_arch_list())"           # want 'sm_120' in the list

If get_arch_list() has no sm_120, your build has no Blackwell kernels. That is the whole story.

Related failures and their causes: CUDA driver version is insufficient means the host driver is older than the container's CUDA runtime needs; a failing GPU Operator validator usually means the driver container could not compile; and a host-driver/container-runtime mismatch produces the same family of launch errors.

What not to do: do not chase this with --enforce-eager, and do not downgrade the model to make the error go away. It is a build-target problem, not a configuration problem. Get a binary with sm_120 kernels and the error disappears.

Is a Hetzner GEX63 cheaper than an AWS, GCP, or Azure GPU instance for inference?

For always-on inference, yes, by a wide margin. A flat €999 per month versus an on-demand hyperscaler GPU instance multiplied by 730 hours is not a close race. The hyperscalers win when the GPU is idle most of the month, when you need to burst, or when compliance and ecosystem proximity matter more than unit price.

Start with the arithmetic. The closest single-GPU on-demand instance by purpose is the AWS g6e.xlarge (one L40S, 48 GB), at about $1.86 per hour in us-east-1 (AWS EC2 on-demand pricing). Run it 730 hours and that is roughly $1,358 a month for half the VRAM of the GEX63. AWS p5.48xlarge is eight H100s (640 GB) at tens of dollars per hour, a different class entirely, not a single-GPU comparison. None of these is an exact match for an RTX PRO 6000 Blackwell, so the honest framing is VRAM-per-euro, not instance-for-instance.

The break-even: divide €999 by the g6e.xlarge rate of about $1.86 per hour. Even before converting euros to dollars, that is roughly 537 GPU-hours a month, about 74 percent of a 730-hour month; after the currency conversion it is closer to 80 percent. Below that utilization, the cloud instance is cheaper; above it, Hetzner wins, and you are getting twice the VRAM while you do.

Now the fair counterargument. Spot and committed-use pricing narrow the gap a lot. AWS Spot and GCP Spot VMs discount heavily against on-demand, and one-year committed-use or Savings Plans cut the rate further (AWS Spot, GCP Spot VMs). If your workload tolerates interruption, Spot can beat a dedicated box on paper. The trade is that Spot capacity can be reclaimed mid-request, which is exactly what you do not want in front of a user-facing inference endpoint.

Then the hidden costs on both sides. Hyperscalers charge for internet egress per gigabyte; Hetzner includes unlimited traffic on the GEX63's 1 Gbit/s port. Against that, Hetzner gives you no managed control plane and no GPU redundancy, and the engineer-hours of driver and node maintenance are now a salary line instead of an invoice line. Managed Kubernetes adds roughly $73 a month per cluster on EKS and GKE, while AKS offers a free control-plane tier with a weaker SLA (EKS pricing, GKE pricing, AKS pricing).

The honest conclusion in one line: flat-rate dedicated hardware for steady 24/7 inference, hyperscalers for spiky demand, Spot-tolerant batch, compliance requirements, or when the rest of your stack already lives there.

What breaks in production on a single self-hosted GPU node, and how do I keep it running?

The first deploy is easy. Month three is the problem. Plan for driver and kernel pinning, GPU health detection, a restart story for an OOM'd model server, and the fact that one node is one point of failure with no live migration to save you.

• Pin the kernel and the driver. Disable unattended reboots on the serving node, and rehearse driver upgrades somewhere else first. The most common self-inflicted outage is an automatic kernel bump that leaves the driver container unable to compile.
• Detect a sick GPU. Watch DCGM health, Xid errors in dmesg, ECC error counts, and thermal throttling. A node can report Ready to Kubernetes while the GPU is effectively dead, so alert on GPU metrics, not just node status.
• Handle CUDA OOM. Tune --gpu-memory-utilization and --max-model-len, cap concurrency at the gateway, and prefer queueing over crashing. An OOM that kills the pod costs you the whole model-load time on restart.
• Know your scaling paths. You can join a second GEX63 to the same K3s cluster, or partition the GPU for several small models. Check what this card actually supports before you promise MIG-style isolation; not every Blackwell SKU exposes the same partitioning. When a single node stops being enough and you need real redundancy, that is the signal to add a node or move the hot path to a hyperscaler.
• Make the box reproducible. Keep the node build in Terraform or Ansible and the cluster state in Helm values and git. A dead box should be a one-hour rebuild, not a lost weekend.
• Hold a security baseline. No public inference endpoint without auth, the Kubernetes API behind a VPN or Hetzner private network, and model tokens in Secrets rather than env vars in a committed manifest.

Where does an internal developer platform like Qovery fit on a self-hosted GPU cluster?

Once the GPU node serves traffic, the bottleneck moves from infrastructure to workflow: who ships the next model, who gets an environment, who owns the manifests. Qovery installs on your own Kubernetes cluster - including a self-managed K3s or RKE2 cluster on Hetzner - and turns that cluster into git-push deploys, preview environments, and per-environment RBAC without taking the cluster away from you.

Let me be precise about what Qovery is. It is an internal developer platform that deploys and operates applications inside infrastructure you own: AWS, GCP, Azure, Scaleway, or your own existing Kubernetes cluster. The node, the cloud bill, and any Hetzner discount stay in your name. Qovery is not AWS-only, and it does not provision the GPU server for you.

What that buys you on a GPU cluster:

• Each model service becomes a git-push deploy instead of a hand-edited Deployment.
• App teams self-serve without learning the GPU Operator.
• Preview environments spin up per pull request for the services that call the inference endpoint.
• Environment auto-stop keeps non-production from holding the GPU hostage overnight.
• Per-environment RBAC lets the ML team redeploy the model server without cluster-admin on the box serving production traffic.

I want to draw the boundary honestly. Qovery does not install NVIDIA drivers, pick your driver branch, or tune vLLM. You still own the node and the model-serving config. Qovery owns the deployment workflow on top of it. If you would rather assemble that workflow yourself, the fair alternatives are raw Helm plus Argo CD or Flux, KServe or Kubeflow for an ML-specific control plane, Coolify or Dokploy for a simple self-hosted PaaS, or Northflank's bring-your-own-cloud.

How do I do this with Qovery from Claude Code or Cursor?

Install the Qovery Agent Skills once, then describe the goal in plain language and let the agent wire it. The skills cover onboarding, deploying, and troubleshooting; the MCP Server operates what already exists and is read-only by default.

curl -fsSL https://skill.qovery.com/install.sh | bash

# or scope the install to the current project only
curl -fsSL https://skill.qovery.com/install.sh | bash -s -- --project

The skills work across Claude Code, Cursor, Codex, OpenCode, and Gemini CLI, and cover the common jobs: qovery-onboard, qovery-deploy, qovery-troubleshoot, qovery-optimize, qovery-speedup, qovery-preview, and qovery-terraform.

To let an agent inspect and operate your environments, add the Qovery MCP Server to Claude Code over OAuth. It is read-only by default:

claude mcp add --transport http qovery https://mcp.qovery.com/mcp --callback-port 4242

# read-write: only when you want the agent to act (also enable write in the Qovery Console)
claude mcp add --transport http qovery "https://mcp.qovery.com/mcp?read_write=true" --callback-port 4242

After installing, ask in plain language, for example "connect my existing Kubernetes cluster to Qovery and deploy this inference service." The quickstart and the Claude Code integration have the full walkthrough (AI agent quickstart, Qovery + Claude Code).

If you take one thing from this: 96 GB of Blackwell VRAM at a flat monthly price makes self-hosted inference genuinely affordable for a small team, as long as you go in clear-eyed about owning the driver stack and the single point of failure. Get the engineering right first, then hand the deployment workflow to your team with the Agent Skills quickstart. When you are ready to give everyone git-push deploys on the cluster you just built, try Qovery free.

How do I self-host AI inference on Hetzner's GEX63 Blackwell GPU server with Kubernetes?

Provision the GEX63 with Ubuntu 24.04, install K3s single-node, install the NVIDIA GPU Operator with Helm, and confirm the node advertises nvidia.com/gpu: 1. Then deploy vLLM with resources.limits.nvidia.com/gpu: 1, a model cache on a persistent volume, a long startupProbe, and an authenticated TLS ingress in front. The whole path is about 30 minutes of setup plus model-load time.

Which models fit in the 96 GB of VRAM on a Hetzner GEX63?

Roughly a 70B-class model at 4-bit (about 40 GB of weights), a 32B-class model at BF16 (about 64 GB), or several 7B-14B models side by side. The KV cache, which grows with context length times concurrency, is usually what runs out first, not the weights, so leave headroom with --gpu-memory-utilization.

Do I need the NVIDIA GPU Operator, or is the NVIDIA Container Toolkit enough on K3s?

On K3s, use the GPU Operator. It installs the driver, the container runtime config, the device plugin, and the DCGM exporter together, and it officially supports K3s and RKE2. The bare Container Toolkit is fine for plain Docker, but on Kubernetes you need the device plugin so the GPU becomes an allocatable nvidia.com/gpu resource; without it, the scheduler will overcommit the GPU and the pods will OOM.

Is a Hetzner GEX63 cheaper than an AWS, GCP, or Azure GPU instance for inference?

For always-on inference, yes. At €999 a month flat it undercuts the closest single-GPU on-demand instance (AWS g6e.xlarge, ~$1.86/hr, ~$1,358/month for half the VRAM) above roughly 74-80 percent monthly utilization. The hyperscalers win on spiky demand, Spot-tolerant batch, compliance, and ecosystem proximity, so compute your break-even GPU-hours before choosing.

Why does my vLLM container fail with "no kernel image is available for execution on the device" on a Blackwell GPU?

Because the container's build has no kernels for compute capability 12.0 (sm_120). Blackwell needs CUDA 12.8 or newer and a vLLM/PyTorch build compiled for sm_120; older images only go up to sm_90. Confirm with python -c "import torch; print(torch.cuda.get_arch_list())" and look for sm_120; if it is missing, use a current cu128 image rather than tweaking runtime flags.

Can I use Qovery with a self-managed Kubernetes cluster on Hetzner?

Yes. Qovery installs on your own Kubernetes cluster, including a self-managed K3s or RKE2 cluster on a Hetzner dedicated server, as well as on your own AWS, GCP, Azure, or Scaleway account. The cluster, the node, and the bill stay in your name; Qovery adds git-push deploys, preview environments, and per-environment RBAC on top.

Should I use vLLM or Ollama to serve models on a single GPU server?

Use vLLM when you want maximum throughput and OpenAI-API compatibility for a single primary model; its paged KV cache handles concurrency well. Use Ollama when you want the easiest setup and frequently swap small models, accepting weaker concurrent throughput. Both run fine on K3s with the GPU Operator and both have recent sm_120-ready builds.