Cloud Run GPU vs a dedicated GKE node pool: 7 ways to self-host an open-weight model in 2026

Key points:

• Pick Google Cloud Run with GPU when inference traffic is bursty or intermittent. It scales to zero, attaches one NVIDIA L4 (24 GB) or one RTX PRO 6000 Blackwell (96 GB) per instance, and bills per second, so a service with no traffic costs nothing and you never manage a node pool (Google Cloud docs).
• Pick a dedicated GKE GPU node pool when a GPU stays busy most of the day, or when you need A100 or H100 cards, more than one GPU per replica, MIG partitioning, GPU time-sharing, Spot GPUs or committed-use discounts. Cloud Run offers none of those.
• The deciding number is sustained GPU utilization, not price per hour. From Google's published us-central1 list prices, an always-on GKE L4 node on Spot or a 1-year CUD beats Cloud Run once a Cloud Run instance would be up more than about 30-40% of the month. Against an on-demand node, the crossover moves to about 50-68%.
• Cold start is the hidden tax on serverless GPU. The GPU instance itself starts in about 5 seconds, but loading 10-30 GB of weights dominates the total, so bake weights into the image or stream them from Cloud Storage before you benchmark anything.
• GKE buys hardware control and pays for it in operations. Node pools, GPU drivers, autoscaler tuning, cluster upgrades and RBAC all become your job. Qovery operates that cluster inside your own GCP, AWS, Azure or Scaleway account, so a team without a DevOps engineer can still run dedicated GPU nodes and keep the bill and discounts in its own name.

Most teams self-hosting an open-weight model pick the wrong platform because they compare hourly GPU prices instead of their own utilization curve. An L4 on Cloud Run costs about $1.05 per instance-hour (Cloud Run pricing) and an L4 GKE node costs $0.71 on demand (Compute Engine pricing), so GKE looks cheaper. It only is if the GPU actually works for most of those hours.

My verdict: if your traffic is bursty and one L4 or RTX PRO 6000 holds your model, use Cloud Run GPU. If a GPU would be busy more than roughly a third of the day, or you need A100, H100, multi-GPU or MIG, use a dedicated GKE node pool. The rest of this article is the evidence, with every price linked.

Cloud Run GPU or a dedicated GKE node pool: which should you pick?

Pick Cloud Run GPU if your inference traffic is bursty and a single L4 or RTX PRO 6000 is enough; pick a dedicated GKE node pool if a GPU would be busy more than roughly a third of the day, or if you need A100/H100, multi-GPU or MIG. Both run open weights you control. They differ on who pays for idle time and who operates the hardware.

Three things rule out Cloud Run GPU straight away:

• You need a GPU Cloud Run does not offer. Cloud Run supports exactly two GPUs, the NVIDIA L4 and the RTX PRO 6000 Blackwell. No A100, no H100 (Google Cloud docs).
• You need more than one GPU per replica. Cloud Run allows one GPU per instance, so tensor parallelism across cards is off the table.
• You have a strict latency SLO on the cold path. A scale-from-zero request waits for instance start plus model load.

Three things rule out a dedicated GKE node pool:

• Nobody on the team can operate Kubernetes, and nobody wants to learn on a production GPU bill.
• Traffic is idle most of the day, so an always-on node bills hours that serve no one.
• No one will own cluster upgrades, driver versions and the occasional node stuck in NotReady.

I compare every option on six criteria: scale-to-zero, cold start with large weights, GPU catalogue, who operates it, whose cloud account it runs in, and cost shape at low versus high utilization. Hosted model APIs (OpenAI, Anthropic, Amazon Bedrock's managed models) are out of scope, because the question is about self-hosting open weights.

What are the 7 ways to self-host an open-weight model, compared side by side?

In 2026 there are seven realistic ways to self-host an open-weight model: Google Cloud Run with GPU, a dedicated GKE GPU node pool, GKE Autopilot with GPUs, Qovery-managed Kubernetes in your own cloud account, Amazon SageMaker endpoints or EKS GPU node groups, Modal, and raw GPU clouds such as RunPod or Lambda Labs. The table orders them by how often a small team should consider them.

When is Cloud Run GPU the right fit for self-hosted inference?

Cloud Run with GPU is the right fit when traffic is bursty, intermittent or pre-product-market-fit and one L4 or RTX PRO 6000 can hold your model. It scales to zero, needs no cluster, and bills per second of instance time. Google made Cloud Run GPUs generally available on June 2, 2025, with no quota request required for L4 (Google Cloud blog).

What you get, per Google's documentation as of November 2026 (Google Cloud docs):

• Two GPU types. NVIDIA L4 with 24 GB of VRAM, and NVIDIA RTX PRO 6000 Blackwell with 96 GB.
• One GPU per instance. A sidecar cannot share it.
• Minimum sizing. An L4 instance needs at least 4 vCPU and 16 GiB of memory (8 vCPU and 32 GiB recommended). An RTX PRO 6000 instance needs at least 20 vCPU and 80 GiB.
• Regions. L4 runs in us-central1, us-east4, europe-west1, europe-west4, asia-southeast1 and asia-south1 (invitation only). RTX PRO 6000 runs in us-central1, europe-west4, asia-southeast1 and asia-south2.
• Zonal redundancy is on by default for new GPU services. Turning it off cuts the L4 price from $0.0002909 to $0.0001867 per second (Cloud Run pricing).

The billing detail that decides most comparisons: GPU services must use instance-based billing, so you pay for the whole instance lifetime, including startup and idle time, and minimum instances bill at the full rate even when idle (Google Cloud docs). Cloud Run may also keep a GPU instance idle for up to 10 minutes after its last request (Cloud Run autoscaling docs). A burst of three requests can bill ten minutes.

Concurrency is the cost lever most teams miss. Cloud Run does not scale on GPU utilization. It scales GPU services on request concurrency, so a max-concurrency set too low adds instances while each GPU sits half empty (Cloud Run GPU best practices). Run vLLM with continuous batching and let one instance take many requests at once. The vLLM team measured up to 24x the throughput of Hugging Face Transformers and up to 3.5x that of TGI (vLLM blog). On a 7B model, that is the difference between one instance and several.

Cold start is weight loading. Google quotes about 5 seconds to a GPU-ready instance and about 19 seconds time-to-first-token from zero for the small gemma3:4b model (Google Cloud blog). A 7B model at FP16 is about 16 GB of weights, and a 70B at FP16 about 140 GB (Hugging Face). Mitigations, ranked by impact:

1. Pull weights from Cloud Storage with parallel download, which Google lists as the fastest option for large models, or mount a bucket with Cloud Storage FUSE and an in-memory cache. Both need Direct VPC egress and Private Google Access (Cloud Run GPU best practices).
2. Bake weights into the image for models under 10 GB, which uses Cloud Run's container streaming. Every model change then means a rebuild.
3. Quantize to 4-bit and use a fast-loading format such as GGUF or safetensors instead of pickle checkpoints.
4. Set min-instances to 1 or more. This removes the cold start and also removes the scale-to-zero saving, since one always-on L4 min instance bills at least $764 a month at list price, and about $1,038 with the default zonal redundancy.

Skip Cloud Run if you need A100 or H100 cards, tensor parallelism across GPUs, a 70B model at full precision, or sub-second p99 on a cold path.

When does a dedicated GKE GPU node pool beat Cloud Run?

A dedicated GKE node pool beats Cloud Run on cost once a GPU is busy a large share of the day, and on capability for everything Cloud Run cannot schedule: A100 and H100 cards, multi-GPU replicas, MIG partitioning, GPU time-sharing, and Spot or committed-use pricing.

The catalogue is the first gap. GKE node pools run the g2 family (L4, from 1 to 8 GPUs per VM), a2 (A100 40GB and 80GB), a3 (H100 80GB, 8 per VM on a3-highgpu-8g), plus H200, B200 and RTX PRO 6000 machines (Compute Engine pricing). In us-central1 on demand, an L4 g2-standard-4 costs $0.7068 per hour, an A100 40GB a2-highgpu-1g $3.6734, an A100 80GB a2-ultragpu-1g $5.0688, and an 8x H100 a3-highgpu-8g $88.49.

The two biggest cost levers on GKE don't exist for Cloud Run GPUs:

• Spot GPU nodes. Google advertises up to 91% off on-demand (Google Cloud docs). The L4 node runs $0.4230 per hour on Spot today, about 40% off (Compute Engine pricing). Spot nodes can be reclaimed, so pair them with an on-demand floor for user-facing traffic.
• Committed use discounts. A resource-based CUD brings the L4 node to $0.4453 per hour for 1 year (37% off) or $0.3181 for 3 years (55% off) (Compute Engine pricing). On Cloud Run, the published CUD columns discount CPU and memory but list no discount on the GPU itself (Cloud Run pricing).

You also get serving control: vLLM tensor parallelism across several cards, Triton or TGI, KV cache and batching policy, and GPU sharing. GKE MIG splits an A100 or H100 into up to seven hardware-isolated slices (GKE docs), while time-sharing and NVIDIA MPS pack several small models on one card with software isolation only (GKE docs). I covered that trade-off in GKE fractional GPU vs dedicated node pools.

The operational bill is real. Someone sizes node pools, pins NVIDIA driver versions, tunes the cluster autoscaler so GPU nodes scale down, plans version upgrades, and answers the page when a GPU node goes NotReady at 2am. GKE charges $0.10 per cluster per hour for that control plane, with a $74.40 monthly credit that covers one zonal or Autopilot cluster per billing account (GKE pricing).

GKE Autopilot with GPUs is the middle ground. Google manages the nodes, but Pods that request a GPU use node-based billing, so you pay for the whole node plus an Autopilot management premium, and CUDs still apply to the VM (GKE pricing). It is the better pick when you already run Kubernetes, want GPUs this quarter, and don't need custom node images or driver pinning.

Skip a dedicated pool if you are three people with no Kubernetes experience and a GPU that would sit idle 20 hours a day.

What is the break-even point between serverless GPU and a reserved GPU node?

Break-even is a utilization number: the Cloud Run instance-hours you would bill each month, against 730 hours of a reserved node. From Google's us-central1 list prices, Cloud Run GPU is cheaper below about 30-40% of the month against a Spot or 1-year CUD L4 node, and below about 50-68% against an on-demand node. These are my estimates from published list prices. Here are the inputs so you can redo them.

The arithmetic, in two lines:

1. Cloud Run cost per instance-hour (L4, 4 vCPU, 16 GiB, us-central1): ($0.0001867 GPU + 4 × $0.000018 vCPU + 16 × $0.000002 GiB) × 3,600 = $1.0465 without zonal redundancy, or $1.4216 with it, which is the default (Cloud Run pricing).
2. Monthly instance-hours: (requests per month × GPU-seconds per request ÷ effective concurrency ÷ 3,600) plus the idle tails, up to 10 minutes after each burst.

A GKE L4 node for 730 hours costs $515.99 on demand, $308.80 on Spot and $325.07 on a 1-year CUD (Compute Engine pricing). Dividing by the Cloud Run hourly rate gives the crossover:

In requests, the crossover depends almost entirely on how bursty you are. A request that needs 2 seconds of generation, batched 4 at a time by vLLM, costs 0.5 instance-seconds. A perfectly packed instance would reach 295 hours only at about 2.1 million requests a month. Real traffic arrives in bursts with 10-minute idle tails, so most teams cross over far earlier.

Scenario costs for a 7B model on one L4, estimated from list prices. Assumptions: 0.5 instance-seconds per request after batching, Cloud Run without zonal redundancy, no free tier, and one always-on GKE node.

Hidden costs on each side:

• Cloud Run: min-instances bill 24/7, zonal redundancy adds about 36% to the per-hour rate, and every cold start bills the weight load.
• GKE: a Standard cluster always keeps at least one node for system Pods, plus persistent disk or a bucket for weights, plus egress, plus the second cluster fee once the credit is used.
• Both: image pulls of multi-GB layers and registry egress across regions.

A 70B model breaks this comparison. At FP16 it needs about 140 GB of VRAM, at FP8 about 70 GB and at INT4 about 35 GB (Hugging Face), before KV cache. That rules out the L4 entirely. Quantized, it fits Cloud Run's 96 GB RTX PRO 6000. At full precision it needs two A100 80GB or H100 cards with tensor parallelism, and only GKE offers that.

The second-order effect: batching raises effective utilization. Once vLLM keeps one card full, a single GKE node absorbs traffic that would have spread across several Cloud Run instances, which pushes the break-even toward GKE faster than most teams expect.

Are Modal, RunPod, or SageMaker better than both for self-hosting?

For teams with no cloud preference and no data-residency constraint, Modal often gets a model serving fastest and RunPod is usually cheaper per GPU-hour than either Google option, but both run your workload in their account, not yours. SageMaker and EKS are the same serverless-versus-dedicated choice under AWS names.

• Modal packages Python functions as containers, scales to zero, and bills GPUs per second: $0.000222 for an L4, $0.000694 for an A100 80GB and $0.001097 for an H100, with CPU and memory billed on top (Modal pricing). Memory snapshots often start initialization-heavy functions 3-10x faster, but Modal states they don't speed up loading weights from storage (Modal docs). Their infrastructure, their bill, their regions.
• RunPod lists L4 pods at $0.59 per hour on Secure Cloud and $0.44 on Community Cloud, A100 80GB from $1.19, and H100 SXM at $3.99 or $2.69 (RunPod pricing). Lambda Labs and Together play in the same market. These are the cheapest raw GPU-hours and the weakest platform guarantees, best for experiments, fine-tuning and batch jobs.
• AWS: SageMaker Serverless Inference does not support GPUs (AWS docs). The Cloud Run role goes to SageMaker real-time endpoints, which can scale to zero with inference components, though AWS notes provisioning from zero takes several minutes and requests fail meanwhile (AWS docs). EKS GPU node groups play the GKE role, with the same $0.10 per cluster-hour control plane fee on standard support (Amazon EKS pricing). On Azure, AML endpoints and AKS GPU node pools are the same pair again.

One of these is plainly the better answer when H100 capacity is unavailable in your GCP region, when the work is research or fine-tuning, or when the team wants zero cloud account setup. They are the wrong answer when regulated or customer data must stay in your VPC, when the model needs private access to internal services, or when you have committed spend with a hyperscaler to burn down.

How do you run dedicated GPU nodes without hiring a DevOps engineer?

If you need a dedicated GPU node pool but have nobody to operate Kubernetes, Qovery closes that gap: it provisions and operates the cluster inside your own GCP, AWS, Azure or Scaleway account, or on an existing Kubernetes cluster, so the GPU nodes, the model weights and the committed-use discounts stay in your name.

To be precise about what we do: Qovery is an internal developer platform that operates the Kubernetes layer, and it is neither a GPU provider nor a model-serving framework. You still buy GPUs from your cloud, and you still run vLLM, TGI, Triton or Ollama in your container.

What Qovery handles in this setup:

• Cluster provisioning and managed upgrades, including GPU node pools on GKE, EKS, AKS or Scaleway Kapsule.
• Git-push deployment of your inference container, with per-environment RBAC.
• Preview environments per pull request, so a model change gets its own short-lived GPU environment.
• Auto-stop for non-production environments, which matters most for GPUs, since an idle L4 staging node bills $0.71 an hour whether anyone uses it or not.

Bring-your-own-cloud is the point. The GPU bill, the Spot savings and the CUDs land in your own cloud account, and the weights never leave your VPC. At Qovery, we built it cloud-agnostic on purpose: the same setup runs on GCP, AWS, Azure, Scaleway or a cluster you already run.

To be fair: a three-person team serving 5k requests a month should use Cloud Run GPU and nothing else. No cluster, Qovery included, is worth operating at that volume.

How do I deploy this from Claude Code or Cursor?

Install the Qovery Agent Skills once and your coding agent can provision the cluster and deploy your inference container into your own cloud account, asking you to approve each action before it runs.

Install the skills globally for any compatible agent (Claude Code, Cursor, Codex, OpenCode, Gemini CLI):

curl -fsSL https://skill.qovery.com/install.sh | bash

Or scope them to the current project:

curl -fsSL https://skill.qovery.com/install.sh | bash -s -- --project

Add the Qovery MCP Server to Claude Code, over OAuth and read-only by default:

claude mcp add --transport http qovery https://mcp.qovery.com/mcp --callback-port 4242

Enable write access only when you want the agent to change infrastructure. You also need to turn on write access in the Qovery Console:

claude mcp add --transport http qovery "https://mcp.qovery.com/mcp?read_write=true" --callback-port 4242

The skills that apply here are qovery-onboard, qovery-deploy, qovery-optimize and qovery-troubleshoot. The MCP Server operates what already exists; the skills do the deploying. Then ask in plain language: "Deploy this vLLM container to a GPU node pool in my GCP account with Qovery."

Docs: AI agent quickstart and Claude Code integration.