How to Move GPU Inference from Lambda or CoreWeave to Your Own AWS or GCP Account (Without Rewriting the Pipeline)

Key Points:

• If your inference service already runs as a container with a pinned CUDA stack and serves under docker run --gpus all, moving it from Lambda Labs or CoreWeave to your own AWS, GCP, Azure, or Scaleway account is an infrastructure swap. Same image digest, same model weights, same HTTP/gRPC contract, different node pool.
• Five things move: the container image and its CUDA/driver pairing, the GPU SKU mapping (H100 80GB, A100 80GB/40GB, L40S, L4, A10G), model weight storage and cold-start time, autoscaling behavior, and GPU quota approval. Quota is the only step with a multi-day external dependency, and accelerated-instance quota is often zero in a fresh account or project, so file it on day one.
• Four things predictably break: a CUDA toolkit the host driver cannot serve, a missing NVIDIA driver or device plugin on the destination node pool, weights baked into the image, and a readiness probe that fires before the model finishes loading. All four are fixed in configuration, never in inference code.
• Lambda Labs and RunPod are genuinely cheaper per raw H100-hour, and specialized GPU clouds get new SKUs first. Teams migrate anyway because committed-use discounts on existing spend, VPC-private data paths, data residency, a single compliance scope, and utilization control move the total bill more than the hourly rate does. A GPU idle 60% of the time costs more per million tokens at a low hourly price than a well-packed one at list.
• Qovery provisions and operates the Kubernetes cluster inside your own AWS, GCP, Azure, or Scaleway account, or plugs into an existing Kubernetes cluster, so the cloud bill and committed-use discounts stay in your name while developers deploy the same container by git push, with no hand-written manifests and no dedicated DevOps engineer.

Most teams assume that moving GPU inference off Lambda Labs or CoreWeave means rebuilding the pipeline. It does not, provided your inference service is already a container with a pinned CUDA stack that serves correctly on any GPU host with a compatible driver.

I keep having the same conversation with AI founders. They started on a specialized GPU cloud because it was the fastest way to get an H100 in 2024 or 2025, and it was the right call. Then an enterprise customer asks for the workload inside a VPC they can audit, or the CFO notices a large AWS commitment going unused while the GPU bill lands somewhere else. Suddenly "we need to move inference" turns into a quarter-long rewrite in everyone's head.

It is a much smaller job than that. This guide walks through what changes, what breaks, which instance replaces which, the seven steps in order, and an honest look at cost.

Can you migrate GPU inference off Lambda or CoreWeave without re-architecting the pipeline?

Yes, in most cases. If the inference service is already a container with a pinned CUDA version that serves correctly under docker run --gpus all, the serving code, model artifact, and API contract stay byte-identical when you move it to AWS, GCP, Azure, or Scaleway; only node provisioning, driver installation, weight loading, autoscaling, and ingress change.

"Without re-architecting" has a precise meaning here. You ship the same image digest, load the same model artifact URI, expose the same HTTP or gRPC contract, and run the same serving runtime, whether that is vLLM, TGI, Triton, TorchServe, or Ray Serve. If any of those four things changes, you are doing a migration plus a refactor, and you should split them into two projects.

What does change is everything under the container:

• Who provisions the GPU node. Lambda or CoreWeave hands you a machine or a managed Kubernetes node; in your own account, you (or your platform) create the node pool.
• How the NVIDIA driver and the k8s-device-plugin land on the host. Every provider does this differently, covered below.
• Where weights come from. S3, GCS, Azure Blob, or Scaleway Object Storage instead of a provider-local volume.
• How scaling is triggered and how traffic reaches the pod.

Three cases genuinely require code changes. Multi-node inference that depends on InfiniBand and NCCL across hosts needs a re-tuned network layer. A dependency on a provider-managed inference endpoint API (instead of your own container) means you have to build the container first. And code that hardcodes provider-local NVMe scratch paths needs those paths made configurable.

The portability test fits in one line: if docker run --gpus all your-image serves on any GPU host with a compatible driver, you are portable. That works because the NVIDIA Container Toolkit exposes the host's GPUs and driver to the container at runtime; the CUDA user-space libraries travel inside your image.

On timeline, the engineering work for a single containerized service is measured in days. The calendar time is set by GPU quota approval and capacity, which I cover in the steps below, and that is the part you cannot speed up by working harder. The destination is never AWS-only either: GCP, Azure, Scaleway, or an existing self-managed Kubernetes cluster follow the identical path.

What actually breaks when you move a GPU container to AWS, GCP, Azure, or Scaleway?

Four failures account for nearly every broken first deploy of a GPU container in a new cloud: a CUDA toolkit version the host driver cannot serve, no NVIDIA driver or device plugin on the destination node pool, weights baked into the image or local disk, and a readiness probe that fires before the model finishes loading. None of them require touching inference code.

1. CUDA toolkit versus host driver. The CUDA runtime lives in your image; the kernel driver lives on the host. NVIDIA's minor version compatibility rules say an app built with any CUDA 12.x toolkit runs on a driver >= 525, and CUDA 13.x needs a driver >= 580, with caveats: PTX JIT compilation does not work on older drivers, and some newer API calls fail with cudaErrorCallRequiresNewerDriver. Forward compatibility through the cuda-compat package lets a newer toolkit run on an older driver, but only on NVIDIA data center GPUs, select RTX server SKUs, and Jetson. Record the minimum driver your image needs and pin the driver version on the node pool, so a node image upgrade never surprises you mid-week.

2. Driver and device plugin install paths differ per provider. One line each, from each provider's own docs:

• GKE: on recent versions, GKE automatically installs the default NVIDIA driver on GPU nodes; gpu-driver-version accepts default, latest, or disabled.
• EKS: the EKS-optimized AL2023 NVIDIA AMIs ship the driver and container toolkit but not the NVIDIA device plugin, which you install separately; the Bottlerocket NVIDIA variant includes the device plugin too. If you use the NVIDIA GPU Operator, disable its driver install on these AMIs.
• AKS: AKS-managed GPU node pools install and maintain the driver, device plugin, and DCGM exporter; the self-managed path installs only the driver by default.
• Scaleway Kapsule: the NVIDIA GPU Operator is installed by default on every GPU pool, covering driver, container toolkit, and device plugin.

GPUs are only schedulable once a device plugin advertises them. As the Kubernetes GPU scheduling docs put it, after you install the plugin "your cluster exposes a custom schedulable resource such as nvidia.com/gpu", and you request it in limits. The one-line check: kubectl describe node <gpu-node> must show nvidia.com/gpu under Capacity and Allocatable. If it does not, nothing you deploy will ever schedule.

3. Weights in the wrong place. Baking a 16 GB or 140 GB checkpoint into the image makes every pull slow and every node cold start slower, and it ties the image to one model version. Pulling from object storage (S3, GCS, Azure Blob, Scaleway Object Storage) keeps images small and makes the artifact URI the version pin. A shared filesystem (EFS, Filestore, Azure Files) avoids repeated downloads across replicas but adds a throughput bottleneck you have to size. vLLM can also stream Safetensors weights directly from s3://, gs://, or az:// paths with the Run:ai Model Streamer via --load-format runai_streamer.

4. Probes that fire too early. This is the single most common cause of "it worked on Lambda but not on EKS". On a bare GPU VM nothing kills your process while it loads weights for four minutes. On Kubernetes, a liveness probe will. Add a startup probe: liveness and readiness checks are disabled until it succeeds, and the container gets failureThreshold x periodSeconds to start. If you measured a 4-minute load, give it headroom:

startupProbe:
  httpGet:
    path: /health
    port: 8000
  periodSeconds: 10
  failureThreshold: 36   # 36 x 10s = 360s budget for a ~240s measured load
readinessProbe:
  httpGet:
    path: /health
    port: 8000
  periodSeconds: 5

Two smaller things also do not transfer. MIG partitioning, time-slicing, and fractional GPU behavior are configured per provider and per device plugin, so re-create them deliberately (our GPU partitioning guide covers the options). And a public GPU-cloud endpoint becomes VPC-private ingress in your own account, which changes how upstream services resolve the model service.

Which AWS, GCP, Azure, or Scaleway GPU instance replaces the Lambda or CoreWeave node you run today?

Map by GPU model and vRAM per card, never by instance family marketing name. The H100 80GB has a direct equivalent on AWS, GCP, Azure, and Scaleway; the A100, L40S, L4, and A10 are each missing from at least one of them, and cross-node InfiniBand fabric at scale is where Lambda and CoreWeave still lead.

Match on GPU model plus vRAM per card first, then size the vCPU, RAM, and local NVMe around it. The vRAM math decides more than people expect. Weights in memory are roughly parameters x bytes per parameter, before KV cache:

• A 70B model in FP16 needs about 70B x 2 bytes = 140 GB. That does not fit on any single card in this table, so you need tensor parallelism across at least two 80GB GPUs.
• The same 70B model at 8-bit is about 70 GB. It technically fits one 80GB card, with roughly 10 GB left for KV cache, which caps concurrency hard. On an A100 40GB it does not fit at all. Note that FP8 needs Hopper or Ada hardware; A100 does INT8, not native FP8.
• An 8B model in FP16 is about 16 GB, which fits comfortably on a 24GB L4 or A10G with room for KV cache.

Sources: AWS P5, GCP GPU machine types, Azure GPU VM sizes, Scaleway GPU instance types, and NVIDIA datasheets for H100, A100, L40S, and L4.

Single-node 8x H100 with NVLink exists on all four clouds. AWS P5 adds 3,200 Gbps of EFA networking per p5.48xlarge. Where Lambda and CoreWeave still lead is large multi-node InfiniBand clusters you can get on short notice, which matters for training far more than for inference.

Capacity is the other reality. EC2 Capacity Blocks for ML let you reserve GPU instances with a start date up to eight weeks out, and GCP offers future reservations. Regional on-demand H100 capacity is uneven, and spot or preemptible capacity is rarely defensible for latency-bound inference because a reclaim drops live requests. Verify every price and regional availability on the vendor's current page at migration time instead of trusting any blog number, including this one. Our GPU capacity guide goes deeper on where to find it.

What are the seven steps of a GPU inference migration, in order?

A GPU inference migration runs as seven ordered steps, and step one is the GPU quota and capacity request, because it is the only step with a multi-day external dependency you cannot compress. In my experience, accelerated-instance quota is often zero or close to it in a new AWS account or GCP project until you ask.

Step 1: File GPU quota and capacity requests on day one. On AWS, use Service Quotas for "Running On-Demand P instances" or "G and VT instances" (measured in vCPUs). AWS documents that smaller increases are usually auto-approved, larger ones go to Support and "take time to review", and quota requests do not get priority support. On GCP, GPU quota is per region and per GPU model, plus a global "GPUs (all regions)" quota you must raise too; Free Trial billing accounts cannot request changes. On Azure, request vCPU quota for the specific VM family in the Quotas blade. Then start everything else in parallel.

Step 2: Audit the container. Pin the base image CUDA version, record the minimum host driver it needs from the compatibility table, and confirm docker run --gpus all serves on a single test node. vLLM, for example, documents that its default binaries are compiled with CUDA 12.9, with builds for CUDA 12.8 and 13.0, and requires compute capability 7.5 or higher.

Step 3: Move weights to object storage. Upload the checkpoint to S3, GCS, Azure Blob, or Scaleway Object Storage under a version-pinned URI, and load it into a local cache volume at startup. Measure cold start before and after, because that number feeds your startup probe and your autoscaling floor.

Step 4: Create the GPU node pool. Match the GPU type from the table above, add a taint (for example nvidia.com/gpu=present:NoSchedule) with matching tolerations on GPU workloads, and pin the driver version. Before deploying anything real, run a throwaway pod:

kubectl run gpu-check --rm -it --restart=Never \
  --image=nvidia/cuda:12.9.0-base-ubuntu24.04 \
  --overrides='{"spec":{"tolerations":[{"key":"nvidia.com/gpu","operator":"Exists"}],"containers":[{"name":"gpu-check","image":"nvidia/cuda:12.9.0-base-ubuntu24.04","command":["nvidia-smi"],"resources":{"limits":{"nvidia.com/gpu":1}}}]}}'

If nvidia-smi prints the expected GPU and driver, and kubectl describe node shows allocatable nvidia.com/gpu, the pool is ready.

Step 5: Deploy the same image digest, unchanged. Reference it by @sha256: digest, not by tag. Wire ingress, health checks, and the startup probe tuned to the cold start you measured in step 3.

Step 6: Benchmark side by side. Compare p50 and p95 latency, tokens per second, throughput at target concurrency, GPU utilization from DCGM Exporter, and cost per million tokens. Use the same load generator, the same prompts, and the same serving backend in both environments, or the comparison is worthless.

Step 7: Shift traffic 10 / 50 / 100 percent behind a weighted split, and keep the old endpoint warm for a one-week rollback window before you release the Lambda or CoreWeave capacity.

One autoscaling difference deserves its own warning. GPU scale-to-zero recovery takes minutes, not seconds, because node provisioning, image pull, driver readiness, and weight load run serially. Keep a warm floor of at least one replica for latency-sensitive endpoints and reserve scale-to-zero for batch and internal eval workloads.

Is GPU inference cheaper in your own cloud account or on a specialized GPU cloud?

Per raw GPU-hour, specialized GPU clouds like Lambda Labs and RunPod are clearly cheaper than AWS, GCP, or Azure on-demand, and CoreWeave's on-demand H100 rate is close to AWS. Teams migrate anyway because committed-use discounts against existing spend, VPC-private data paths, data residency, a single compliance scope, and utilization control change the total bill more than the hourly rate does.

Credit where it is due. As of the time of writing (October 2026, verify on the live pricing pages):

• Lambda lists the H100 SXM 80GB at $3.99 per GPU-hour on 8x instances and the A100 SXM 80GB at $2.79.
• RunPod lists the H100 SXM at $3.99 per GPU-hour on Secure Cloud and $2.69 on Community Cloud.
• CoreWeave lists an 8x HGX H100 instance at $49.24 per hour on demand, about $6.16 per GPU-hour, and advertises up to 60% off for committed usage.
• AWS lists p5.48xlarge (8x H100) at $55.04 per hour on demand in us-east-1, about $6.88 per GPU-hour, after its June 2025 price cut of up to 45% on P5. See the EC2 on-demand pricing page.
• Azure lists ND96isr_H100_v5 at $98.32 per hour pay-as-you-go in East US, about $12.29 per GPU-hour, per the Azure Linux VM pricing page.
• GCP A3 and Scaleway H100 prices vary by region and change often; check the GCP GPU pricing page and Scaleway GPU pricing directly. Our GPU pricing comparison method explains how to compare them fairly.

Specialized clouds also get new SKUs faster, offer real InfiniBand fabric, and need near-zero setup. They are great until control, compliance, committed discounts, or data gravity matter more than the hourly rate.

Your own account changes the math in four ways. First, discounts: AWS Savings Plans go up to 72% for EC2 Instance Savings Plans and 66% for Compute Savings Plans; GCP resource-based committed use discounts go up to 55% for non-memory-optimized series, and L4-based G2 machines qualify for flexible CUDs at 21% (1 year) or 43% (3 years); Azure Reserved VM Instances go up to 72%. Those are headline maximums, so check the rate for your exact GPU family. Second, existing enterprise agreements and credits apply. Third, private networking to your data. Fourth, one SOC 2, HIPAA, or GDPR scope instead of two vendors to audit.

The cost almost nobody models is idle GPU time. Effective cost per million tokens equals the hourly rate divided by the millions of tokens you actually serve in that hour. Take an illustrative H100 that sustains 1,000 output tokens per second when busy (an assumption for the arithmetic, not a benchmark), so 3.6 million tokens per fully used hour:

The honest reading: if you can keep Lambda GPUs packed, stay on Lambda for cost alone. If you cannot, because traffic is spiky or every team runs its own half-empty endpoint, packing matters more than the vendor's sticker price.

Data gravity is the other quiet cost. If training data, the feature store, and the vector database already live in AWS, every retrieval payload and log line crossing to another cloud pays egress. AWS lists internet egress from us-east-1 at $0.09 per GB for the first 10 TB and $0.085 for the next 40 TB, so 50 TB a month of cross-cloud traffic costs about $4,300 before you count the added latency. Check GCP network pricing and Azure bandwidth pricing for theirs.

Hybrid is a legitimate end state. Plenty of teams burst training on a specialized GPU cloud and run steady-state inference in their own account behind one stable endpoint.

How do you keep the inference pipeline identical while the platform underneath changes?

Freeze the inference contract at three layers - the container image digest, the model artifact URI, and a stable endpoint address - and nothing upstream (orchestration, queues, eval harness, CI) ever learns the GPU moved from Lambda or CoreWeave to your own cloud.

Put the endpoint behind a stable internal DNS name or gateway, such as inference.internal.yourco.com, so no upstream service hardcodes a provider URL. The traffic split in step 7 then becomes a routing change instead of a config change in ten repos.

Keep model artifacts in object storage under version-pinned URIs, and never bake weights into an image you intend to move. Treat the serving runtime (vLLM, Triton, TGI, Ray Serve) as the compatibility boundary: it is what makes the workload portable across clouds, and its own docs define the supported CUDA floor. Our inference server comparison covers how the runtimes differ.

In CI/CD, build once and deploy the identical image digest to both environments for the whole parallel run. Rebuilding per cloud reintroduces exactly the drift you are trying to rule out.

Finally, observability parity. Export DCGM GPU metrics plus latency and token throughput from both environments to the same Prometheus-compatible backend. DCGM Exporter runs as a DaemonSet and serves metrics on a /metrics endpoint; AKS-managed GPU pools even install it for you. Without matching dashboards, the step 6 benchmark is opinion.

Where does Qovery fit in a GPU inference migration?

Qovery provisions and operates the Kubernetes cluster inside your own AWS, GCP, Azure, or Scaleway account, or plugs into an existing Kubernetes cluster, so a team leaving Lambda or CoreWeave keeps the cloud bill and committed-use discounts in its own name and gets git-push deployments without hand-written Kubernetes manifests or a dedicated DevOps engineer.

BYOC is the point. The cloud bill, Savings Plans, and Committed Use Discounts stay in your account, which is usually the whole reason for the migration. Qovery handles cluster provisioning and managed cluster upgrades, node pools, git-push deployments of the same container image, per-environment RBAC, and databases backed by managed cloud services.

For GPUs specifically, on EKS you enable GPU node pools in the cluster settings, choose the instance types, mix on-demand and spot, cap the total GPU count, and declare how many GPUs each service needs; Qovery handles provisioning and scheduling through Karpenter. On the other providers, or on your own cluster, you run your GPU node pool and Qovery deploys onto it.

Two features pay for themselves on GPU spend. Preview environments per pull request give every model or prompt change its own ephemeral environment for eval and regression runs. Environment auto-stop shuts non-production down overnight and on weekends, so staging stops burning H100 hours while nobody is looking.

What Qovery does not do: it does not rewrite your inference code, pick your GPU SKU, or replace your serving runtime. You keep the container and the node type.

And honest alternatives, because they fit some teams better:

• Terraform with EKS or GKE if you already employ platform engineers who want to own every module.
• SkyPilot for opportunistic multi-cloud GPU scheduling, especially batch and training jobs chasing the cheapest available capacity.
• KServe or Ray Serve on self-managed Kubernetes when you want ML-specific serving control (model mesh, canaries per model) and have the team to run it.
• SageMaker Inference or Vertex AI Prediction when you want the cloud provider to own the endpoint entirely and accept its serving contract.

How do I run this migration from Claude Code or Cursor?

Install the Qovery Agent Skills once, then describe the migration to your coding agent in plain language: it connects your cloud account, creates the service, and deploys the same container into your own AWS, GCP, Azure, or Scaleway account, asking you to approve each infrastructure change.

Install the skills globally for Claude Code, Cursor, Codex, OpenCode, Gemini CLI, and other compatible agents:

curl -fsSL https://skill.qovery.com/install.sh | bash

Or scope the install to the current repository instead of your whole machine:

curl -fsSL https://skill.qovery.com/install.sh | bash -s -- --project

Connect the Qovery MCP Server to Claude Code over OAuth. It is read-only by default, so the agent can inspect environments, service state, and deployment status without changing anything:

claude mcp add --transport http qovery https://mcp.qovery.com/mcp --callback-port 4242

Add write access only when you explicitly want the agent to act. It also requires write access enabled in the Qovery Console, so a model cannot talk its way into it:

claude mcp add --transport http qovery "https://mcp.qovery.com/mcp?read_write=true" --callback-port 4242

Which skill does what: qovery-onboard connects your cloud account and cluster, qovery-deploy ships the inference container, qovery-troubleshoot debugs failed pods and probe timeouts, qovery-optimize cuts idle spend, qovery-speedup shortens deploy time, qovery-preview creates ephemeral eval environments, and qovery-terraform exports the setup as code. The skills deploy; the MCP Server operates what is already running.

Then ask in plain language: "deploy this inference container to a GPU node pool in my AWS account with Qovery", and approve the changes it proposes. The full setup is in the AI agent quickstart and the Claude Code integration page.

The short version

Moving GPU inference off Lambda Labs or CoreWeave is an infrastructure swap when the workload is a pinned, portable container. File quota first, fix the driver, device plugin, weights, and probes in configuration, deploy the same digest, benchmark honestly, and shift traffic gradually. Kubernetes is already where most of this lands: the CNCF Annual Cloud Native Survey published in January 2026 reports that 66% of organizations use Kubernetes to host their generative AI workloads.

If you want that cluster running in your own cloud account without building a platform team first, try Qovery.