GPU Quota Request Stuck? How to Burst the Same Training Job onto a Second Cloud in a Day

Key Points:

• A pending GPU quota request is a capacity decision, and resubmitting the same number rarely moves it. Work the unblock order instead: a different region, a different GPU generation, reservation-style capacity (AWS EC2 Capacity Blocks for ML or Google Cloud Dynamic Workload Scheduler), then a burst to a second cloud.
• If your training job is a container that reads from object storage and writes checkpoints back to it, you can usually run it on a second cloud's GPUs the same day. The blocker is almost never the model code. It is the cluster, the driver stack, the image registry and the data path.
• Portability needs exactly three contracts: an OCI image with a pinned CUDA, cuDNN and NCCL stack, S3-compatible object storage for data and checkpoints, and Kubernetes plus the NVIDIA GPU Operator as the scheduler. Everything else belongs in config.
• The six-step burst: confirm capacity, stand up a GPU Kubernetes cluster with the GPU Operator, make the image pullable, stage the dataset once, redeploy the same job definition with only endpoint and resource requests overridden, then resume from the last checkpoint.
• Qovery is the deployment layer that turns the burst into a config change instead of a migration. It runs inside your own AWS, GCP, Azure or Scaleway account, or connects to your existing or rented Kubernetes cluster, so the same job definition, image, env vars, secrets and RBAC can target a second GPU cluster. Qovery does not buy GPU capacity and is not a training scheduler.
• GPU hours are rarely the surprise. A one-time dataset egress (about $171 for 2 TB out of AWS us-east-1 at list), duplicate object storage, and GPU nodes left idle after a failed run are what blow the budget. Keep checkpoints in the burst cloud during the run and ship only the final artifact back.

A GPU quota ticket that has been "pending" for four days is a capacity decision your cloud provider has not made yet, and waiting on it is the most expensive option on the table. Every day the ticket sits, your ML engineers are blocked, the roadmap slips, and the GPUs you need are probably sitting idle in another region or on another provider.

I see this pattern constantly with teams deploying on Qovery. The training code is fine. The container is fine. What is missing is a second place to run it, and the belief that moving clouds means rebuilding the pipeline. It usually does not. This is the runbook I give teams: what to try first, what makes a job portable, which tools solve which part, the six steps, and what it really costs. It works whether your primary cloud is AWS, GCP, Azure or Scaleway, and whether the burst target is another hyperscaler, a GPU specialist, or a Kubernetes cluster you rent.

A quick warning before the numbers: cloud prices and quota policies change. Every figure below links to the vendor page I read on October 29, 2026. Re-check those pages before you sign anything.

Why is my GPU quota request pending, and how long will it actually take?

A GPU quota increase is evaluated against physical regional capacity and your account's history, which is why requests sit for days, come back partially approved, or get approved for a different region or a smaller instance family. None of AWS, Google Cloud or Azure publishes a committed turnaround for large accelerated-instance increases, so plan for "no" and start the parallel path today.

Each provider counts quota differently, and asking in the wrong unit is the fastest way to get a request that comes back too small:

• AWS counts On-Demand quota in vCPUs, per account, per Region, per instance family group. GPUs live under "Running On-Demand P instances" and "Running On-Demand G and VT instances", both with a default of 0. One p5.48xlarge needs 192 vCPUs of P quota, so asking for "8" gets you nothing usable. The Service Quotas guide says larger requests go to Support, "take time to review", "don't receive priority support", and can be approved, denied, or partially approved.
• Google Cloud enforces GPU quota per region and per GPU model (for example NVIDIA_A100_80GB_GPUS), plus a global "GPUs (all regions)" quota on new projects. Preemptible quota (PREEMPTIBLE_NVIDIA_H100_GPUS and friends) is tracked separately and covers Spot and Flex-start VMs. Google's quotas overview says decisions rest on "the availability of resources, the length of time you've used Google Cloud, and other factors".
• Azure enforces regional vCPUs plus VM-family vCPUs per subscription per region, so you need headroom on both your total and on the NC, ND or NV family you want. The automated review "usually happens within a few minutes". When it fails, you get a link to a support request, which has no published timeline.

Expect partial outcomes and recognise them early: approved in a different region, approved for an older accelerator, approved for On-Demand but not Spot, or approved for a fraction of what you asked for.

Before escalating, run this triage:

1. Is the blocker a regional vCPU quota or a per-family (or per-GPU-model) quota?
2. Does the region actually offer the GPU model you requested?
3. Does the account have enough billing history to justify the number?
4. Is a reservation product available instead (covered in the next section)?

A support case that names the workload, instance type, region, GPU count and start date moves faster than resubmitting the same number. Neither is a plan you should block a whole team on.

What should I try on my current cloud before bursting to a second one?

Four in-cloud moves clear most stuck training jobs faster than a cross-cloud burst: change the region, change the GPU generation, buy reservation-style capacity, or run on Spot with checkpoint-resume. If none of them produces usable capacity within 24 to 48 hours, bursting to a second cloud is the cheaper option.

Switch region first. Capacity and quota are both regional. A second region in the same cloud needs zero pipeline changes beyond the storage endpoint and the inter-region transfer cost.

Downshift the accelerator. A10G, L4 and A100 are frequently available when H100 is not. For most fine-tuning and mid-size runs, this changes wall-clock time, not feasibility. On paper, NVIDIA lists 624 BF16 TFLOPS with sparsity for the A100 against 1,979 with sparsity for the H100 SXM, so budget for a run up to three times longer and measure your real step time.

Buy reservation-style capacity. Two products exist precisely because quota and capacity are separate problems:

• AWS EC2 Capacity Blocks for ML reserve P6, P5, P4d and Trainium instances in 1-day increments up to 14 days, then 7-day increments up to 182 days, with up to 64 instances per block and a start time up to 8 weeks out. The key detail: Capacity Blocks "don't count against your On-Demand Instances limits", so they route around the stuck ticket entirely. Cancellations are not allowed.
• Google Cloud Dynamic Workload Scheduler has two modes. Flex-start queues the request and, with queued provisioning on GKE, allocates all requested nodes at once in a single zone; nodes run for at most 7 days and GPU workloads can wait up to 14 days. Calendar mode reserves GPUs for up to 90 days, with the earliest start 87 hours after you submit. Flex-start consumes preemptible quota, which is often easier to get.

Run on Spot with checkpoint-resume. Spot discounts are large: AWS says up to 90%, Google Cloud up to 91%, and Azure up to 90%. Spot only works if your job writes checkpoints every N steps and resumes cleanly, which is exactly the property that makes a cross-cloud burst possible.

Managed training services such as Vertex AI Training or SageMaker Training can sometimes find capacity you cannot, but they trade speed today for a proprietary job spec tomorrow.

My decision rule: multiply blocked engineers by their fully loaded day rate and by the expected days of waiting. Compare that to the hourly price delta of the next available GPU elsewhere, times the run length. The second number is almost always smaller.

What makes a training pipeline portable enough to burst to another cloud?

A training job is portable when it depends on exactly three contracts: an OCI image with a pinned CUDA, cuDNN and NCCL stack, S3-compatible object storage for inputs and checkpoints, and Kubernetes plus the NVIDIA GPU Operator as the scheduler. Everything else is cloud-specific glue that belongs in environment variables, not in the training code.

Image. Pin CUDA, cuDNN, NCCL and framework versions in the image, then check them against the host driver on the target cluster. NVIDIA's minor version compatibility rule says CUDA 12.x applications need a driver of at least 525, and CUDA 13.x needs at least 580. The NVIDIA GPU Operator installs the driver, NVIDIA Container Toolkit, device plugin, DCGM Exporter and MIG Manager on any conformant Kubernetes cluster, so the driver version becomes something you choose rather than inherit.

Data. Use one S3-compatible client with endpoint, bucket, region and credentials injected as env vars. Google Cloud Storage's XML API interoperates with S3 tools through https://storage.googleapis.com and HMAC keys. Scaleway Object Storage supports a subset of the S3 API at endpoints like https://s3.fr-par.scw.cloud. Azure Blob needs an S3-compatible proxy or a second client path behind the same env vars.

Checkpointing. Write checkpoints to object storage every N steps with a deterministic resume path, for example s3://$BUCKET/runs/$RUN_ID/step-*, and have the entrypoint load the latest one on start. This single property turns a migration into a restart.

Scheduler. Use a plain Kubernetes Job, Kubeflow Trainer (the successor to the Training Operator), Ray, Volcano or SkyPilot. Proprietary specs (AWS Batch, SageMaker Training, Vertex AI custom jobs) are the hardest pieces to re-implement on another cloud.

Registry and secrets. Use one registry both clusters can pull from, or mirror the image. Store pull credentials and storage keys once in your deployment layer, never in manifests.

Multi-node NCCL. A multi-node job that runs on cloud A and hangs on cloud B is almost always interconnect env vars, not code. Images tuned for AWS EFA or a specific InfiniBand fabric carry settings that do not apply elsewhere. Unset anything fabric-specific first (a forced NCCL_NET plugin, FI_* variables), then set NCCL_DEBUG=INFO, pin NCCL_SOCKET_IFNAME to the real interface, and set NCCL_IB_DISABLE=1 if the target has no IB or RoCE. Use NCCL_P2P_DISABLE=1 only to diagnose intra-node hangs.

Anti-patterns that break bursts:

• Hardcoded instance types or node names in the job spec
• Cloud SDK calls (boto3, google-cloud) inside the training loop
• Paths that assume a specific CSI driver or a local NVMe layout
• IAM-role-only authentication with no access-key fallback

Which tools let me run the same training job on a second cloud's GPUs?

Four categories solve different halves of the problem: GPU-specialist clouds sell capacity, cross-cloud schedulers decide where a run lands, managed training services optimise for one cloud, and an internal developer platform like Qovery moves your job definition, config and environments between clusters you own. Most teams combine a specialist or second-cloud cluster, Kubernetes, and a deployment layer rather than picking one vendor.

GPU-specialist providers (CoreWeave, Lambda, Nebius, Together AI, Scaleway GPU instances, OVHcloud) are the fastest route to capacity and often cheaper per GPU-hour. You bring your own orchestration, data path, CI and secrets.

Cross-cloud schedulers (SkyPilot, Ray and Anyscale, Flyte, managed Slurm) place runs across clouds. SkyPilot describes itself as "a system to run, manage, and scale AI workloads on any AI infrastructure" and lists Kubernetes, Slurm, AWS, GCP, Azure, OCI, CoreWeave, Nebius, Lambda and more than 15 other providers. These tools are strongest for research-style, short-lived jobs.

Managed training services (SageMaker Training, Vertex AI Training, Azure Machine Learning) are the easiest path inside one cloud and the hardest to burst out of, because the job spec is proprietary.

Qovery is the deployment and environment layer. It runs inside your own AWS, GCP, Azure or Scaleway account, or connects to your existing or rented Kubernetes cluster, so the same job definition, container image, env vars, secrets and per-environment RBAC can be redeployed against a second GPU cluster from the console, API, CLI or a coding agent. The cloud bill, commitments and discounts stay in your name.

To be plain about scope: Qovery does not purchase GPU capacity, does not schedule distributed training, and does not replace SkyPilot or Ray. What it removes is the CI, manifest, secret and environment rebuild that normally turns a burst into a week of work, and it lets a team without a dedicated DevOps engineer run Kubernetes without managing it directly.

How do I actually burst the job to a second cloud in under a day?

A same-day burst is six steps in a fixed order: confirm the exact capacity, stand up or rent a GPU Kubernetes cluster with the NVIDIA GPU Operator, make the image pullable, stage the dataset once, redeploy the identical job definition with only the storage endpoint and resource requests overridden, then resume from the last checkpoint. Steps 1 to 4 are where the hours go; step 5 should take minutes.

Step 1: confirm capacity. Get the GPU model, count, interconnect, driver version and region in writing from the provider or the console, not from a pricing page.

Step 2: stand up the cluster. Use managed Kubernetes (EKS, GKE, AKS, Scaleway Kapsule) or the provider's own cluster. Install the GPU Operator with Helm, adding --set driver.enabled=false if the nodes ship with drivers preinstalled:

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia && helm repo update
helm install --wait --generate-name -n gpu-operator --create-namespace \
  nvidia/gpu-operator --version=v26.7.2

Label and taint GPU nodes so only training pods land there:

kubectl label nodes <gpu-node> workload=training
kubectl taint nodes <gpu-node> nvidia.com/gpu=present:NoSchedule

The job spec then carries the matching toleration and an explicit GPU request:

spec:
  template:
    spec:
      nodeSelector:
        workload: training
      tolerations:
        - key: nvidia.com/gpu
          operator: Equal
          value: present
          effect: NoSchedule
      containers:
        - name: trainer
          image: registry.example.com/trainer@sha256:<digest>
          resources:
            limits:
              nvidia.com/gpu: 8

Step 3: make the image pullable. Mirror the image or point both clusters at one registry, with pull credentials stored once in the deployment layer.

Step 4: stage the data once. Copy the dataset with either tool, both of which accept an S3-compatible endpoint (rclone, AWS CLI):

rclone sync aws:my-dataset scw:my-dataset --s3-endpoint https://s3.fr-par.scw.cloud --transfers 32 --progress
aws s3 sync s3://my-dataset s3://my-dataset-burst --endpoint-url https://s3.fr-par.scw.cloud

Note that aws s3 sync uses one endpoint per command, so copy cloud-to-cloud with rclone or sync to local disk first. Time the transfer and compute the egress bill before the run starts.

Step 5: redeploy. Point the same job definition at the second cluster and override only storage endpoint, bucket, credentials and resource requests. This is the step Qovery collapses: the environment, env vars and secrets already exist, so you clone the environment onto the second cluster and change four values instead of rebuilding CI.

Step 6: resume. Start from the latest checkpoint, compare step time against the spec ratio for the new GPU, and set a hard budget alert before you walk away.

Rollback. When the original quota clears, move back and keep the burst cluster documented as capacity insurance with the GPU node pool scaled to zero.

What does bursting a training job to a second cloud actually cost?

GPU hours are rarely the surprise. The real costs of a cross-cloud burst are a one-time dataset egress out of the primary cloud (about $171 for 2 TB from AWS us-east-1 at list), duplicate object storage while both environments exist, and GPU nodes left running after a failed run, and all three are smaller than a week of blocked engineers.

Egress. At list price, internet egress for the first 10 TB costs $0.09/GB from AWS us-east-1 after 100 GB free, $0.12/GiB on Google Cloud Premium Tier up to 1 TiB and $0.11 from 1 to 10 TiB (North America source), and $0.087/GB on Azure after 100 GB free.

The free-exit programs (AWS, Google Cloud, Azure) cover leaving a provider, not routine bursting. The exception is Google's Data Transfer Essentials, generally available since September 2025, which offers no-charge multicloud traffic between your own workloads in EU, UK and Swiss regions under specific conditions.

Duplicate storage. Keeping 2 TB in a second bucket costs about $46 per month on S3 Standard or about €32 on Scaleway Standard Multi-AZ, plus whatever checkpoints you write. Keep checkpoints in the burst cloud and ship back only the final weights.

Idle GPUs. This is the largest avoidable line item. ClearML's 2024 State of AI Infrastructure survey found only 7% of companies believe their GPUs exceed 85% utilization at peak, and 15% report less than half their purchased GPUs in use. Scale-to-zero node pools and environment auto-stop matter more than the hourly rate.

Worked example for 2 TB of data and 48 GPU-hours on H100 (six hours on one 8-GPU node), at on-demand list prices on October 29, 2026:

Azure's ND H100 v5 rate is published on the Azure Linux VM pricing page, and AKS Standard tier lists at $0.10 per cluster-hour in Azure's retail price list. Even the most expensive line in the GPU row is a fraction of one engineer-week.

How do I avoid being stuck the next time a GPU quota request is denied?

Treat multi-cloud GPU access as a standing capability with four parts: one portable job definition, pre-approved quota in at least two regions or clouds, a second cluster you can scale from zero, and a documented dataset copy procedure with a known duration and price.

• Request quota before you need it in a second cloud or region, and keep that account warm with a small recurring job so there is spend history when you ask for the big number.
• Keep the job definition, image build, env vars and secrets in a deployment layer that can target multiple clusters, so switching is a config change.
• Document the data copy. Knowing the transfer takes three hours and costs $171 is the difference between a decision and a guess.
• Run a quarterly fire drill: one small training run end to end on the backup cluster, with the checkpoint-resume path actually exercised.

This is where Qovery fits day to day. Environments are defined once and deployed into your own AWS, GCP, Azure or Scaleway account or your own Kubernetes cluster, with per-environment RBAC and managed cluster upgrades. Auto-stop schedules shut the standby environment down between drills so the backup GPU capacity does not quietly burn money.

How do I do this with Qovery from Claude Code or Cursor?

Install the Qovery Agent Skills once, then ask your coding agent in plain language to redeploy the training job to the second cluster. The agent proposes each action and you approve it before anything changes.

Install globally for Claude Code, Cursor, Codex, OpenCode, Gemini CLI and other compatible agents. The installer includes qovery-onboard, qovery-deploy, qovery-troubleshoot, qovery-optimize, qovery-speedup, qovery-preview and qovery-terraform:

curl -fsSL https://skill.qovery.com/install.sh | bash

Or scope the install to one repository instead of the whole machine:

curl -fsSL https://skill.qovery.com/install.sh | bash -s -- --project

Add the Qovery MCP Server to Claude Code to inspect what already runs. It is read-only by default, with an OAuth login in your browser:

claude mcp add --transport http qovery https://mcp.qovery.com/mcp --callback-port 4242

Enable write access only when you want the agent to change infrastructure, and only after enabling write access in the Qovery Console:

claude mcp add --transport http qovery "https://mcp.qovery.com/mcp?read_write=true" --callback-port 4242

Then prompt in plain language: "deploy this training job to my Scaleway GPU cluster with Qovery and reuse the env vars from the AWS environment". The agent shows the plan and asks you to approve each step. Skills deploy; the MCP Server operates what already exists. The full setup is in the AI agent quickstart and the Claude Code integration page.