7 Best GPU Cloud Setups for Self-Hosting Google's Gemma Models in 2026

One NVIDIA L4 on AWS (g6.xlarge) costs $0.8048 per hour in us-east-1. Leave it on for a month and you pay about $587, whether it serves ten requests or ten million. The same $587 buys roughly 690 million tokens of Gemini 3.5 Flash-Lite, Google's cheapest per-token model, at a 3:1 input-to-output mix ($0.30 input, $2.50 output per million). To beat that, your self-hosted Gemma has to push about 260 tokens per second, around the clock, all month.

Most teams asking me "where should we self-host Gemma?" don't have that traffic yet, and they should keep paying the API. This guide is for the ones who do, or who need the model inside their own cloud account for reasons that have nothing to do with price.

Prices checked October 2026. GPU pricing moves fast, so click through before you commit.

Key Points:

• Self-hosting Gemma beats a hosted API only once your traffic is steady. A rented GPU costs the same at 1% or 100% utilization, so break-even is a utilization question, not a model question.
• Gemma 3 ships in 1B, 4B, 12B and 27B sizes (plus a 270M variant). 4B and quantized 12B fit on a single 24GB GPU (L4, A10G, RTX 4090). 27B needs an 80GB card at bf16, or a 48GB L40S once quantized.
• For experiments and cheap hourly GPUs, Runpod, Vast.ai and Lambda are the fastest path. For production with compliance requirements, run vLLM on GPU nodes inside your own AWS, GCP, Azure or Scaleway account.
• The serving engine matters as much as the provider. vLLM and SGLang batch concurrent requests; Ollama and llama.cpp are for a laptop or one developer. Hugging Face TGI is now archived.
• Qovery fits the "our own cloud account, no dedicated DevOps engineer" case: it provisions and operates the Kubernetes cluster and GPU node pool in your account, so the bill, discounts and data residency stay in your name.

When does self-hosting Gemma actually beat a hosted API?

Self-hosting Gemma wins on cost only when your GPU stays busy, and wins on control from day one if data residency, fine-tuning or no-rate-limit latency matters to you.

A hosted API bills per token and scales to zero. A rented GPU bills per hour and keeps billing when nobody calls it. The math that decides it is simple:

effective $ per 1M tokens = hourly GPU price / (sustained tokens per second x 3600 / 1,000,000)

Measure tokens per second yourself with vLLM's benchmark script on your real prompt lengths, then compare against the hosted rate. Note that Google does not sell Gemma per token on the Gemini API paid tier (Gemma 4 is listed as "Not available"), so your real comparison is a Gemini model or a third-party host such as Together AI or Groq.

Teams self-host for three reasons that aren't price: prompts never leave their account, they run their own fine-tunes or LoRA adapters, and they get predictable latency with no shared rate limits. Then the hidden costs arrive: idle GPU hours, model storage and egress, on-call for a stateful service, and GPU driver and Kubernetes node upgrades.

If your traffic is spiky or small, self-hosting will cost you more. I'd rather say that up front than sell you a GPU you leave idle.

What GPU do you actually need for Gemma 3 (1B, 4B, 12B, 27B)?

Pick the GPU from the model size and the quantization you accept: 4B runs on a 16-24GB card, 12B fits a 24GB GPU once quantized, and 27B wants an 80GB A100 or H100 at bf16, or a 48GB L40S at int4.

The rule of thumb is about 2 bytes per parameter at bf16, 1 at int8 and 0.5 at int4. Google's own Gemma 3 QAT announcement gives the weight-only numbers, and those quantization-aware-trained checkpoints lose far less quality than naive post-training quantization.

Weights are only half the story. The Gemma 3 model card gives 4B, 12B and 27B a 128K-token context (32K for 1B and 270M), and at that length the KV cache, not the weights, becomes the biggest memory consumer as concurrency grows.

Weight sizes from Google's Gemma 3 QAT post; add KV cache on top.

Google has since shipped Gemma 4 (E2B, E4B, 12B, 26B A4B MoE and 31B). Its published memory table puts 31B at 69.9 GB in BF16 and 17.5 GB in Q4_0, so the same sizing logic and the same GPUs apply.

Go multi-GPU only when you need 27B or 31B at high concurrency. Tensor parallelism in vLLM works, but one bigger card is always simpler to run.

How did we pick the best GPU cloud setups for Gemma?

We judged every option on six criteria that decide whether a Gemma deployment survives production.

1. Whose account? Vendor-hosted GPU, your own AWS/GCP/Azure/Scaleway account, or your existing Kubernetes cluster.
2. Who runs the platform layer? GPU drivers, NVIDIA device plugin, node pools, cluster upgrades.
3. Price per GPU-hour and availability of the card you need (L4, L40S, A100, H100).
4. Scale-to-zero and cold-start behaviour for bursty traffic.
5. Compliance: SOC 2, HIPAA, GDPR and EU regions, VPC isolation, no third party touching prompts.
6. Time to first token and ongoing ops effort, including whether you need a dedicated DevOps engineer.

We build Qovery, so read this with that bias in mind. That's why each section below names who should not pick the option, including ours.

Which GPU cloud setups made the shortlist?

Seven setups cover the realistic range, from a $0.34/hour community RTX 4090 to vLLM on GPU nodes inside your own cloud account.

Prices checked October 2026 from each vendor's pricing page, single-GPU on-demand, before tax. Modal hourly figures are converted from its per-second rates. AWS us-east-1, Azure East US, Scaleway fr-par-2. Qovery's platform fee is separate from your cloud bill.

Runpod - best for cheap on-demand Gemma experiments?

Runpod is the fastest, cheapest way to get Gemma 3 serving on a real GPU, and the wrong choice if your prompts must stay inside your own cloud account.

Runpod's pricing page splits Community Cloud (cheaper, third-party hardware) from Secure Cloud. An A40 with 48GB goes for $0.35/hour on Community, which is enough for Gemma 3 12B at bf16. Serverless endpoints scale to zero, starting at $0.69/hour for the 24GB tier and $2.72 for 80GB A100s, with cold starts as the trade-off.

Use it for evaluation, fine-tuning runs and bursty internal tools. Skip it for regulated workloads or if you need GPU spend to count against an existing cloud commitment or Savings Plan.

Vast.ai - best for the lowest possible price per GPU-hour?

Vast.ai is the price floor because it's a marketplace of third-party hosts, and that same model is why anything touching customer data or needing an SLA should run elsewhere.

Vast.ai prices are "set by the market," billed per second. Interruptible instances run "50%+ cheaper" than on-demand, and reserved terms of 1 to 6 months go "up to 50% off." Prices per card change hourly, so check the live grid.

It shines for batch inference, offline evaluation and research budgets. Pick your hosts carefully and expect to lose an interruptible instance mid-job.

Lambda and the GPU-native clouds - best for reserved long-running Gemma serving?

Lambda, and peers like CoreWeave and Crusoe, are the best deal when you'll keep a GPU busy for months; teams with nobody to run Linux, CUDA and vLLM in production should pass.

On Lambda's pricing page, an H100 SXM costs $3.99 per GPU-hour on an 8x instance. AWS charges $55.04/hour for a p5.48xlarge, which is $6.88 per H100. Lambda's longest reserved terms go through sales, so ask for a quote before you commit.

You still own everything above the VM: drivers, vLLM, autoscaling, monitoring. Good fit for a steady inference fleet or fine-tuning Gemma on your own data.

Modal and Hugging Face Inference Endpoints - best for zero-ops Gemma endpoints?

Modal and Hugging Face Inference Endpoints give you a production Gemma endpoint with almost no infrastructure work; skip them if your compliance team requires inference inside your own VPC, or if you have cloud commitments to burn down.

Modal bills per second: $0.000222/s for an L4 (about $0.80/hour) and $0.001097/s for an H100 (about $3.95/hour). Read the multipliers: pinning a region costs 1.15 to 1.75x base, and non-preemptible execution costs 3x. Cold starts are well engineered and containers scale to zero.

Hugging Face Inference Endpoints deploy google/gemma-3-* straight from the Hub. GPUs run on AWS or GCP (Azure is CPU-only on that page), from $0.70/hour for an L4 to $10/hour for an H100 on GCP.

If you have no cloud account and nobody to run infrastructure, these two beat everything else on this list, Qovery included.

Raw hyperscaler GPU instances - best for maximum control over Gemma serving?

Running Gemma on EC2, Compute Engine, Azure or Scaleway GPUs gives you the most control and the cleanest compliance story; teams without a platform function should expect it to quietly eat engineering time.

Indicative single-GPU on-demand prices (October 2026):

• AWS: g6.xlarge (L4) $0.80, g6e.xlarge (L40S) $1.86, p5.4xlarge (H100) $6.88 (EC2 pricing, us-east-1).
• Azure: NC24ads_A100_v4 (A100 80GB) $3.67, NC40ads_H100_v5 (H100 NVL) $6.98 (Azure pricing, East US).
• Scaleway: L4-1-24G €0.79, L40S-1-48G €1.47, H100-1-80G €2.87 (Scaleway GPU pricing, Paris).
• GCP: G2 machines bundle L4s and A3 machines bundle H100s; rates vary by region, see Compute Engine GPU pricing.

Spot capacity cuts the bill hard. AWS advertises up to 90% off and GCP up to 91%, and Azure's own price list shows the A100 above at $0.68/hour on Spot. Spot reclaims will interrupt a stateful inference service, so run several replicas and drain gracefully (our spot reclaim playbook).

The real work sits in the Kubernetes layer: the GPU driver and NVIDIA device plugin, node pool autoscaling, pulling multi-GB weights on every scale-up, and cluster version upgrades. Reserved instances and committed-use discounts only pay off if the GPU stays busy.

Qovery - best for running Gemma in your own cloud account without a dedicated DevOps engineer?

Qovery is the pick when you want hyperscaler GPUs in your own account but nobody on the team wants to own Kubernetes; for a throwaway benchmark, Runpod or Vast.ai are cheaper and faster.

Qovery creates and operates managed clusters on AWS EKS, GCP GKE, Azure AKS and Scaleway Kapsule, in your account. It can also plug into your existing Kubernetes cluster, where you keep managing the cluster and Qovery handles deployments.

Because the GPU instances bill to your account, your Savings Plans, committed-use discounts and EU-region requirements still apply, and prompts never transit a third party. On top of that, Qovery runs:

• Managed cluster upgrades, tested internally for a month before rollout.
• Git-push deployments of your vLLM container onto a GPU node pool (how GPU nodes work with Karpenter on EKS).
• Preview environments per pull request, so you can test a prompt or model change against a real endpoint.
• Environment auto-stop so non-production GPU nodes don't idle overnight.
• Per-environment RBAC so developers ship against the inference service without cloud console access.

To be clear about the limits: Qovery isn't a GPU marketplace and doesn't resell or discount GPUs. You pay your cloud provider's list price. If you have no cloud account at all, go to Modal or Hugging Face.

Which serving stack should you run on top - vLLM, SGLang, Ollama or llama.cpp?

For production Gemma, run vLLM or SGLang: they batch many concurrent requests per GPU, while Ollama and llama.cpp are built for one user and will waste most of a cloud GPU.

vLLM's PagedAttention post measured up to 24x the throughput of plain Hugging Face Transformers (on LLaMA models, 2023). vLLM natively supports Gemma 3 and Gemma 4, including LoRA. Hugging Face TGI used to be the other default, but its repository is now archived and Hugging Face recommends vLLM and SGLang instead.

Three settings save you the most pain. Cap --max-model-len to what your app needs, because 128K context reserves a lot of KV cache. Serve QAT checkpoints when quality allows. Bake weights into the image or a cached volume instead of pulling from the Hub at boot. And keep the OpenAI-compatible endpoint, so you can A/B against a hosted API and fail over if the GPU dies. More on that in our inference server cost comparison.

How do I deploy a Gemma inference service to my own cloud from Claude Code or Cursor?

Install the Qovery Agent Skills once, then ask your coding agent in plain language to deploy your vLLM container to your AWS, GCP, Azure or Scaleway account; it asks you to approve each infrastructure action.

curl -fsSL https://skill.qovery.com/install.sh | bash

# Or only for this project
curl -fsSL https://skill.qovery.com/install.sh | bash -s -- --project

# Add the Qovery MCP Server to Claude Code (OAuth, read-only by default)
claude mcp add --transport http qovery https://mcp.qovery.com/mcp --callback-port 4242

# Write access, only when you want it (also enable it in the Qovery Console)
claude mcp add --transport http qovery "https://mcp.qovery.com/mcp?read_write=true" --callback-port 4242

qovery-onboard and qovery-deploy ship the service, qovery-troubleshoot diagnoses a crashing GPU pod, qovery-optimize and qovery-speedup tune cost and build time, qovery-preview creates per-PR environments, and qovery-terraform exports the whole setup as code.

Then type: "deploy this vLLM Gemma container to my GCP account with Qovery." The agent proposes a plan and waits for your approval. Setup details live in the AI agent quickstart and the Claude Code integration page.