GKE fractional GPU access vs dedicated node pools: which actually cuts inference cost for a small ML team?

Key Points:

• Share the GPU when each model uses less than about half of it. GKE GPU time-sharing or NVIDIA MPS lets one NVIDIA L4 node host several small models instead of four near-idle nodes that each bill a full GPU-attached node-hour, 730 hours a month.
• Dedicate the GPU when one model saturates it, needs its full VRAM, or carries a latency SLO. GKE's own docs say memory limits aren't enforced on time-shared GPUs, so one container can grab all the memory and push its neighbours into out-of-memory errors.
• MIG is the only GKE sharing mode with hardware-isolated memory and compute, and it only runs on data-center cards such as the A100 and H100. You buy the expensive card before you can slice it, so MIG is the wrong cost lever for most teams under ten engineers.
• Idle time is the biggest line item. A GPU node that runs around the clock but only serves a 40-hour working week sits idle about 76% of its 730 monthly hours. Scale-to-zero node pools, Spot GPU VMs, committed use discounts and auto-stopped non-production environments save more than any partitioning trick.
• One light endpoint and no Kubernetes today? Cloud Run GPUs, billed per second with scale to zero, beat every GKE setup in this article.

Most small ML teams buy a second GPU while their first one is barely busy. I see it every month: a five-person team runs an embedding model, a reranker and a quantized 7B model, each on its own L4 node, each billing 24/7, and none of them above 20% utilization. The fix is scheduling, and it costs nothing in hardware.

This article compares every real option on Google Cloud, with published prices and the arithmetic shown.

GKE fractional GPU access vs dedicated node pools: which actually cuts inference cost for a small ML team?

Fractional GPU access (GKE GPU time-sharing or NVIDIA MPS) cuts inference cost when a small team serves several models that each use less than roughly 50% of a GPU, sustained, because you stop paying for cards that sit idle. Dedicated node pools cut cost when one model saturates the card, needs its full VRAM, or has a hard latency SLO. In both cases the bigger lever is turning idle GPUs off.

The cost model fits on one line. GKE bills a GPU-attached node-hour (GPU plus host machine type), never a per-inference price, and an average month is 730 hours. An L4 on a g2-standard-4 node costs $0.7068 per hour on demand in us-central1 (Source). A T4 costs $0.35 per hour for the GPU alone (Source) plus $0.19 per hour for an n1-standard-4 host (Source).

The seven options on the table:

• GKE GPU time-sharing
• NVIDIA MPS on GKE
• NVIDIA MIG on GKE
• Dedicated on-demand GPU node pools
• Dedicated Spot GPU node pools
• Cloud Run GPUs
• Vertex AI Endpoints

Three levers sit outside the sharing question and usually save more: GPU node pools with a minimum of zero nodes, Spot GPU VMs, and auto-stopping non-production environments overnight and at weekends.

Scope: every price here comes from Google Cloud's published pricing pages, is us-central1 on-demand unless stated otherwise, varies by region, and was checked on 5 November 2026.

How did we compare GKE GPU strategies for a team with no platform engineer?

We scored each option on six criteria that decide the monthly bill for a team under ten engineers where nobody owns Kubernetes full time: effective GPU utilization, isolation, latency under concurrency, supported GPUs on GKE, operational effort, and the cost floor at zero traffic. Every worked example uses the GPU-attached node-hour (accelerator plus host machine type), because that is what Google Cloud bills.

• Effective utilization: the share of a paid GPU-hour that serves requests. Measure it with DCGM_FI_PROF_SM_ACTIVE or DCGM_FI_PROF_GR_ENGINE_ACTIVE from GKE's DCGM metrics (Source), never with pod count or request count. On time-shared and MPS nodes, use the node-level node/accelerator/duty_cycle metric (Source).
• Isolation: memory limits, compute quota, fault containment, and the blast radius of one noisy neighbour.
• Latency under concurrency: context switching (time-sharing) vs concurrent kernels (MPS) vs hardware partitions (MIG).
• Hardware: which GPUs support each mode on GKE, plus regional availability and accelerator quota, which is often the real blocker.
• Ops effort: node pool flags, GPU driver installation, cluster upgrades, and who gets paged at 2am.
• Cost floor: can it scale to zero, can it run on Spot, and what does a weekend with zero requests cost.

What we did not do: run our own benchmark. This is a comparison of published pricing, documented limits and architecture. Any throughput claim comes from Google or NVIDIA documentation.

Which GKE GPU strategy should you pick? (side-by-side comparison)

For low-traffic, multi-model inference on GKE, rank the options like this: GPU time-sharing first, Spot plus scale-to-zero second, MPS third, dedicated on-demand node pools fourth, MIG last. The order flips when a single model saturates the card or you need hardware-enforced isolation, which puts dedicated node pools and MIG on top.

Which options stack:

• Time-sharing on a Spot node pool with a minimum of zero nodes is a legitimate, cheap combination for internal and non-production inference.
• MIG plus time-sharing, or MIG plus MPS, is documented and recommended by GKE as a way to run several containers per MIG partition (Source).
• Spot plus a strict latency SLO is a combination I would not ship.

Published GPU-attached node prices, us-central1, checked 5 November 2026:

Monthly figures are hourly price × 730. Spot prices are variable and can change up to once a day (Source). GKE adds a cluster management fee of $0.10 per cluster per hour, which is $73 a month, and a $74.40 monthly free tier credit covers one zonal or Autopilot cluster per billing account (Source).

The rule for your design doc: share the GPU below roughly 50% sustained per-model utilization when isolation does not matter, dedicate above it, and turn the node pool off when nobody is using it.

What is GKE GPU time-sharing, and when does it actually lower the bill?

GKE GPU time-sharing lets up to 48 containers share one physical GPU by context-switching between their processes, and it lowers the bill whenever several small models each use a fraction of a GPU and can tolerate some latency jitter. The price of that saving is that GKE does not enforce GPU memory limits between the containers. (Source)

You turn it on per node pool, on GKE 1.23.7-gke.1400 or later (Source):

gcloud container node-pools create shared-l4 \
  --cluster=CLUSTER_NAME \
  --machine-type=g2-standard-8 \
  --accelerator=type=nvidia-l4,count=1,gpu-sharing-strategy=time-sharing,max-shared-clients-per-gpu=4,gpu-driver-version=default \
  --enable-autoscaling --min-nodes=0 --max-nodes=2

Workloads then select the node with cloud.google.com/gke-gpu-sharing-strategy: time-sharing and request nvidia.com/gpu: 1, which now means one share of the GPU.

The line every team should read before co-locating anything, from Google's own limitations section:

Google also says to deploy only workloads in the same trust boundary to a time-shared GPU, and there is no per-container compute quota either.

• Good fit: 3 to 8 small models (embedding models, rerankers, small vision models, a quantized 7B) on one L4 node instead of 3 to 8 nearly idle nodes. Google describes time-sharing as best for "bursty and interactive workloads that have idle periods" (Source).
• Bad fit: latency-SLO endpoints, long-context LLM serving, per-customer isolation, anything where one tenant's OOM must not touch another's.

Worked example 1: four dedicated L4 nodes vs one shared L4 node

1. Dedicated: 4 × g2-standard-4 at $0.706832 per hour (Source) = $2.8273 per hour.
2. × 730 hours = $2,063.95 per month.
3. Shared: 1 × g2-standard-8 (one L4, more host CPU and RAM for four model servers) at $0.853624 per hour (Source) × 730 = $623.15 per month.
4. Difference: $2,063.95 - $623.15 = $1,440.80 per month, before Spot or scale-to-zero. The GKE cluster fee is the same in both cases.

The guardrail: since nothing caps memory, measure peak VRAM per container with nvidia-smi before you co-locate, make sure the sum fits in the L4's 24 GB, and keep batch sizes conservative. The first symptom of over-subscription is a CUDA out-of-memory error in a model nobody deployed that day.

How is NVIDIA MPS different from GPU time-sharing on GKE?

NVIDIA MPS runs CUDA kernels from several containers concurrently on one GPU instead of switching between them, which reduces context switching and gives better behaviour under concurrency than GKE GPU time-sharing. On GKE, MPS divides pinned device memory and active thread percentage per client, but it has no hardware memory isolation and a fatal fault in one client is reported to every client on that GPU.

Enablement uses the same flag with a different value, on GKE Standard 1.27.7-gke.1088000 or later (Source):

--accelerator=type=nvidia-l4,count=1,gpu-sharing-strategy=mps,max-shared-clients-per-gpu=3

What the GKE docs specify (Source):

• MPS works on all NVIDIA GPU types, except G4 machine types with less than one GPU.
• The maximum is 48 clients per physical GPU, or 16 on pre-Volta GPUs.
• By default each client gets 100 / max-shared-clients-per-gpu percent of active threads and total memory / max-shared-clients-per-gpu of pinned device memory. On an L4 with 3 clients, Google's example shows each client getting roughly 33% of threads and 8 GB.
• Memory bandwidth, encoders and decoders are not covered by those limits.
• Pods need hostIPC: true, which Google flags as a security risk.

The difference in one sentence: time-sharing swaps contexts in and out, MPS overlaps kernels. NVIDIA describes MPS as improving GPU utilization and reducing GPU context switching (Source).

The failure mode, in NVIDIA's words: "A fatal GPU fault generated by a Volta MPS client process will be reported to all the clients running on the subset of GPUs in which the fatal fault is contained" (Source). The MPS server then waits for all affected clients to exit before accepting new ones. One buggy model can restart its neighbours.

If latency jitter under time-sharing is your only complaint, try MPS before you buy a second GPU.

When should you use MIG instead of fractional GPU sharing on GKE?

Use NVIDIA MIG on GKE when you need hardware-enforced isolation of GPU memory and compute per workload, because it is the only GKE sharing mode that gives each container its own streaming multiprocessors and memory slice. MIG only runs on data-center GPUs such as the A100 and H100 (not on the T4 or L4), so you pay for the expensive card before you can slice it.

NVIDIA's MIG guide says each instance gets "separate and isolated paths through the entire memory system", with dedicated L2 cache banks, memory controllers and DRAM address buses, and "fault isolation for different clients" (Source).

Partition sizes on GKE (Source):

• A100 40GB: 1g.5gb (7 per GPU), 2g.10gb (3), 3g.20gb (2), 7g.40gb (1).
• A100 80GB: 1g.10gb (7), 2g.20gb (3), 3g.40gb (2), 7g.80gb (1).
• H100 80GB: 1g.10gb (7), 1g.20gb (4), 2g.20gb (3), 3g.40gb (2), 7g.80gb (1).
• Every GPU on every node in a node pool is partitioned the same way, and GKE does not recommend MIG for untrusted workloads.

Why MIG rarely fixes a small team's bill: one a2-ultragpu-1g with an A100 80GB costs $5.0688 per hour (Source). That is 7.2 times the $0.7068 of an L4 g2-standard-4. Split into seven 1g.10gb slices, each slice costs $0.72 per hour and gets 10 GB, while a whole L4 costs $0.71 and gets 24 GB. A quantized 7B inference endpoint fits on an L4.

Where MIG genuinely wins: regulated or multi-tenant workloads, guaranteed VRAM per tenant, mixing training and inference on one expensive card, and shared research clusters. GKE also documents MIG combined with time-sharing or MPS for more containers per partition (Source). I have no published throughput number for that combination, so I won't invent one.

If you don't already own or need A100/H100 capacity, MIG is not your cost lever. If you do, MIG usually beats time-sharing on the same card.

Do dedicated GPU node pools ever win on cost, and how do you make them cheap?

Dedicated GPU node pools win on cost whenever one model keeps the card busy, needs its full VRAM, or carries a hard latency SLO. Above roughly 50 to 60% sustained per-model utilization, sharing adds contention and risk without saving anything. You make dedicated pools cheap with right-sized GPUs, Spot VMs, committed use discounts and a minimum of zero nodes.

Share the GPU when per-model sustained utilization is low and isolation does not matter; dedicate when the model saturates the GPU, needs full VRAM, or has a latency SLO; in both cases the bigger lever is turning idle GPUs off. Measure "sustained" with DCGM_FI_PROF_SM_ACTIVE over a week of real traffic (Source).

• Right-size first. The L4 has 24 GB and the A100 80GB has 80 GB (Source, Source). The A100 node costs 7.2 times more per hour. Most small-model inference fits on the L4.
• Spot VMs. Google advertises discounts of up to 91% (Source). Today the L4 node is about 40% off ($0.4230 vs $0.7068) and the A100 80GB about 41% off (Source). Preemption gives you an optional 120-second notice and a best-effort shutdown period of up to 30 seconds. Spot VMs have no maximum runtime unless you set one (Source). Batch scoring, async queues and staging absorb that. A synchronous user-facing request in flight does not.
• Committed use discounts. Resource-based CUDs on the L4 g2-standard-4 bring the hourly price to $0.4453 (1 year, 37% off) or $0.3181 (3 years, 55% off). On the A100 40GB a2-highgpu-1g they reach 37% and 65% off (Source). Commit only for the always-on floor. If you run on your own cloud account, the commitment is in your name.
• Scale to zero. The cluster autoscaler can take a GPU node pool to zero nodes. A GKE Standard cluster itself never scales to zero, since at least one node must run system Pods (Source), so keep a small CPU pool for those. The cold start adds up to node provisioning, plus automatic NVIDIA driver installation set by gpu-driver-version (Source), plus the image pull, plus model load. Google publishes no total for that sequence, and I won't guess one. Measure it on your own image.

Worked example 2: the idle node

1. A 40-hour working week is about 173 hours a month (40 × 4.33). The remaining 557 of 730 hours, or 76%, serve nobody.
2. L4 g2-standard-4: 557 × $0.706832 = $393.49 per month of idle time.
3. A100 80GB a2-ultragpu-1g: 557 × $5.068798 = $2,821.80 per month of idle time.
4. A single forgotten A100 staging node at minNodes=1 wastes more than the $1,440.80 that consolidating four L4s saves in worked example 1.

Is Cloud Run GPU or Vertex AI cheaper than running inference GPUs on GKE?

For spiky, low-volume inference, Cloud Run GPUs usually cost less than any GKE configuration, because Cloud Run bills per second and scales to zero, while GKE bills each node-hour whether or not a request arrives. Against one always-on L4 node, Cloud Run stays cheaper as long as the instance runs less than about 16 hours a day. You give up multi-model GPU sharing, training on the same cluster and fine-grained cluster networking.

What Google publishes (Source, Source):

• L4 without zonal redundancy: $0.0001867 per second in us-central1, with zonal redundancy: $0.0002909. RTX PRO 6000 is also offered.
• One GPU per instance. An L4 instance needs at least 4 vCPU and 16 GiB of memory.
• GPU services must use instance-based billing, and minimum instances are charged at the full rate even when idle.
• Instances with drivers pre-installed start in about 5 seconds before your container can use the GPU. Model weight loading comes on top of that.

Worked example 3: Cloud Run vs one GKE L4 node

1. Cloud Run L4 instance per second: $0.0001867 (GPU) + 4 × $0.000018 (vCPU) + 16 × $0.000002 (GiB) = $0.0002907 (Source).
2. × 3,600 = $1.0465 per instance-hour.
3. One GKE g2-standard-4 node all month: $515.99 (Source).
4. Break-even: $515.99 ÷ $1.0465 = 493 instance-hours a month, or 493 ÷ 30.4 days = about 16.2 hours a day. Adding a dedicated cluster's $73 fee pushes break-even to about 18.5 hours a day. I excluded the Cloud Run free tier to keep it conservative.

The cold start is the tax: model weights reload on every scale from zero. That breaks a user-facing SLO and does not matter for an internal tool.

Vertex AI Endpoints (Google now documents them under the Gemini Enterprise Agent Platform) bill per node-hour of the deployed machine. A g2-standard-4 prediction node costs $0.81293 per hour in us-central1, versus $0.7068 for the same Compute Engine VM (Source). A deployed model needs at least one replica unless you opt into the Scale To Zero Preview (Source). You get managed serving and give up control and easy per-team cost attribution.

Come back to GKE when you run several models that should share a GPU, train and serve on one cluster, need strict VPC or private networking, or already pay for a Kubernetes setup.

One endpoint, light traffic, no Kubernetes today: Cloud Run GPU is the honest answer, and GKE is overkill.

How do you stop paying for idle GPUs in staging and preview environments?

Delete the idle non-production GPU hours first: a staging GPU endpoint nobody uses after 7pm still bills every node-hour, and an L4 node that runs around the clock for a 40-hour week wastes 557 of its 730 monthly hours, or $393.49. Scheduled auto-stop, scale-to-zero GPU node pools and per-pull-request ephemeral environments save more than any partitioning work.

Ranked by effort-to-saving:

1. Scheduled auto-stop for non-production environments. Staging and dev stop at night and at weekends, and start again in the morning.
2. GPU node pools with --min-nodes=0. Once the workloads are stopped, the autoscaler removes the GPU nodes.
3. Ephemeral preview environments created per pull request and destroyed on merge, so test GPUs exist only while someone reviews.

This is where Qovery fits. Qovery is an internal developer platform that deploys into your own cloud account (AWS, GCP, Azure, Scaleway, or an existing Kubernetes cluster), manages the cluster and its upgrades, and gives your team git-push deployments, per-pull-request preview environments, per-environment RBAC and auto-stop for non-production. Idle GPU environments stop billing overnight, and you don't need a dedicated DevOps engineer to run it.

Bring-your-own-cloud matters more for GPUs than for anything else. Accelerator spend is where committed use discounts, Spot capacity and reservations live, and they only help if the account and the commitment are yours. With Qovery, the GPU bill stays in your own GCP project.

The honest limits:

• Qovery does not partition GPUs for you. Time-sharing, MPS and MIG are GKE node pool settings.
• For advanced GPU scheduling (gang scheduling, queueing, fair-share quotas across teams), NVIDIA Run:ai or Kueue is the right tool, and either beats Qovery at that job.
• With one endpoint and no Kubernetes, Cloud Run GPU is simpler than adding a platform.

How do I set this up with Qovery from Claude Code or Cursor?

Install the Qovery Agent Skills once, then ask your coding agent in plain language to deploy the inference service into your own GCP project and auto-stop the non-production environments. The agent runs the steps and asks you to approve each change before it touches your cloud account.

Install the skills globally for any compatible agent (Claude Code, Cursor, Codex, OpenCode, Gemini CLI):

curl -fsSL https://skill.qovery.com/install.sh | bash

Or scope them to the current project only:

curl -fsSL https://skill.qovery.com/install.sh | bash -s -- --project

Add the Qovery MCP Server to Claude Code over OAuth, read-only by default:

claude mcp add --transport http qovery https://mcp.qovery.com/mcp --callback-port 4242

Or with write access, which you also need to enable in the Qovery Console:

claude mcp add --transport http qovery "https://mcp.qovery.com/mcp?read_write=true" --callback-port 4242

What each skill does: qovery-onboard, qovery-deploy, qovery-troubleshoot, qovery-optimize, qovery-speedup, qovery-preview and qovery-terraform cover setup through infrastructure as code. The MCP Server operates what already exists.

Then ask: "Deploy this inference service to my GCP project with Qovery and auto-stop staging outside working hours."

Docs: AI agent quickstart and Claude Code integration.