How to Survive a Spot GPU Reclaim Without Losing Your Training Run

A spot GPU reclaim is a scheduled event, and the cloud tells you it's coming: 2 minutes ahead on AWS, about 30 seconds on Azure and on Google Cloud's default setting. To avoid losing a training run, checkpoint full training state to object storage on a time budget, trap SIGTERM and flush an emergency checkpoint before the deadline, and run training as a resumable Kubernetes Job across diversified spot capacity with an on-demand fallback. I have watched teams lose three days of training because nobody wired up a signal handler. The fix takes an afternoon.

GPU capacity is still tight in 2026, and spot is still the pragmatic way to train on a budget. This guide is for the ML or platform engineer who already has a training script and wants it to survive interruption.

Key Points:

• You cannot prevent a spot GPU reclaim, so plan for it. AWS EC2 Spot sends a two-minute interruption notice, Azure Spot VMs give 30 seconds through Scheduled Events, and Google Cloud Spot VMs give a best-effort 30-second shutdown by default (configurable to a 120-second notice). Your job must save state and exit inside that window.
• Checkpoint on a time budget, not on epoch boundaries. Save model weights, optimizer state, LR scheduler, AMP gradient scaler, global step, RNG states, and dataloader position to S3, GCS, Azure Blob, or Scaleway Object Storage every N minutes. Expected lost work is about half the interval, so a 10-minute cadence caps the damage at roughly 5 minutes.
• Trap SIGTERM, flush atomically, exit 0. On Kubernetes, raise terminationGracePeriodSeconds above the 30-second default so the flush finishes before the kubelet sends SIGKILL.
• Diversify capacity. Use multiple GPU instance types, multiple availability zones, a capacity-aware allocation strategy, and an on-demand or reserved fallback pool for the final stretch of a run.
• Run training as a resumable Kubernetes Job in your own cloud account so GPU quota, reservations, checkpoint buckets, and the bill stay in your name. Qovery provisions and operates that cluster on AWS, GCP, Azure, Scaleway, or your own Kubernetes cluster. Qovery runs the cluster, and torchrun elastic, Ray, Kueue, or SkyPilot handle job-level orchestration.

What actually happens when a spot GPU instance gets reclaimed?

The cloud takes the instance back after a short, fixed warning: two minutes on AWS EC2 Spot, 30 seconds on Azure Spot VMs, and a best-effort 30 seconds on Google Cloud Spot VMs by default. When the clock runs out, everything in GPU memory and on local NVMe is gone. Only what you already flushed to object storage or a network volume survives.

The timeline on Kubernetes looks like this:

1. The notice lands in instance metadata, an event stream, or Scheduled Events.
2. A node-level handler (Karpenter, AWS Node Termination Handler, or the cloud's managed equivalent) cordons and drains the node.
3. Your pod receives SIGTERM and the grace period starts.
4. The node terminates, whether your pod finished or not.

What you lose: GPU memory, local scratch NVMe, in-flight gradients, optimizer moments, dataloader position, and RNG state. What survives: anything already committed to Amazon S3, Google Cloud Storage, Azure Blob Storage, Scaleway Object Storage, or a shared network filesystem.

Multi-node distributed training is more fragile. One reclaimed rank breaks the NCCL collective, and every other GPU in the job stalls or crashes unless you use elastic or fault-tolerant training.

Spot is the bottom of the capacity hierarchy. Reserved and committed capacity is guaranteed, on-demand is usually available, and spot is whatever is left. GPU pools are small and heavily contested by on-demand buyers, so they are the pool most likely to be taken back.

The exact endpoints to wire up today:

• AWS: poll http://169.254.169.254/latest/meta-data/spot/instance-action with IMDSv2 (AWS recommends every 5 seconds), or consume the EC2 Spot Instance Interruption Warning event from Amazon EventBridge (AWS EC2 docs).
• Google Cloud: watch the preempted metadata value at http://metadata.google.internal/computeMetadata/v1/instance/preempted, which flips to TRUE (Google Cloud Spot VMs).
• Azure: poll http://169.254.169.254/metadata/scheduledevents?api-version=2020-07-01 with the Metadata:true header and look for EventType: Preempt (Microsoft Learn).

How often do spot GPU instances actually get interrupted?

There is no single published interruption rate for GPUs, and anyone quoting one is guessing. AWS says it reclaims Spot capacity "mainly to repurpose capacity," and that interruptions on price only happen if you set a maximum price (AWS EC2 docs). Check your exact GPU family and region before you plan a run.

Where to look:

• AWS Spot Instance Advisor shows interruption frequency buckets per instance type and region, averaged over the last 30 days (AWS). AWS warns that your actual rate depends on point-in-time capacity.
• AWS Spot placement score rates how likely a Spot request is to succeed in a Region or AZ, and AWS states it "does not provide any guarantees" (AWS EC2 docs).
• Azure shows per-size eviction rates in the portal (quoted per hour, based on the last 7 days) and exposes them through the SpotResources table in Azure Resource Graph (Microsoft Learn).
• Google Cloud does not publish a per-machine-type preemption rate in its Spot VM documentation, so you measure it yourself.

GPU families like p4d, p5, and g5 on AWS, or A100 and H100 shapes anywhere, draw from smaller capacity pools under heavy on-demand pressure. Treat them as volatile, and expect the numbers to shift by region, by AZ, and from week to week.

Design for interruption instead of betting on a low number. With discounts of up to 90%, a reclaim that costs you 5 minutes of work is a great trade. A reclaim that costs you 3 days wipes out the saving.

Before you trust any setup, measure two things for a week: your checkpoint write time at your real model size, and the number of reclaims you actually observe. Size your checkpoint cadence from those two numbers.

What exactly should a training checkpoint contain to be resumable?

A resumable checkpoint holds the full training state, far more than model weights. If you only save model.state_dict(), you resume with a reset optimizer, a restarted learning-rate schedule, and a reshuffled dataset, which is a different training run.

Save all of this:

• Model weights
• Optimizer state (Adam moments are often larger than the weights)
• LR scheduler state
• AMP GradScaler state
• Global step and epoch
• RNG states: Python random, NumPy, torch CPU, torch CUDA
• Dataloader or sampler position
• A config hash, so a resume with mismatched hyperparameters fails loudly

For large models, a single-rank torch.save becomes the bottleneck. Use sharded checkpoints instead: PyTorch Distributed Checkpoint (DCP) with FSDP, or DeepSpeed's save_checkpoint, which saves model, optimizer, and LR scheduler state and must be called on every rank (DeepSpeed).

Asynchronous checkpointing keeps the GPUs busy during the save. PyTorch and IBM Research report that async DCP cut checkpoint downtime for a 7B model from an average of 148.8 seconds to 6.3 seconds, a 23.62x reduction (PyTorch blog). The GPU only waits for the GPU-to-CPU copy, and CPU threads persist the rest.

Cadence math: expected lost work is half the interval. A 10-minute cadence loses about 5 minutes per reclaim on average. Pick the interval from your tolerance and your measured write time, never from "every epoch" when an epoch takes six hours.

Atomicity and retention: write to a temporary key, then commit by writing a latest pointer only after the upload succeeds. Keep the 3 most recent checkpoints. A reclaim in the middle of a write then leaves the previous good checkpoint untouched.

Here is the core of it in PyTorch:

import random, signal, sys, time, numpy as np, torch

stop = False
def on_signal(signum, frame):
    global stop
    stop = True  # set a flag only; never save inside the handler
signal.signal(signal.SIGTERM, on_signal)
signal.signal(signal.SIGINT, on_signal)

def training_state(step, epoch):
    return {"model": model.state_dict(), "optimizer": optimizer.state_dict(),
            "scheduler": scheduler.state_dict(), "scaler": scaler.state_dict(),
            "step": step, "epoch": epoch, "loader": loader.state_dict(),  # torchdata StatefulDataLoader
            "rng": {"python": random.getstate(), "numpy": np.random.get_state(),
                    "torch": torch.get_rng_state(), "cuda": torch.cuda.get_rng_state_all()},
            "config_hash": CONFIG_HASH}

for step, batch in enumerate(loader, start=start_step):
    train_step(batch)
    if stop or time.monotonic() - last_save > CKPT_SECONDS:
        save_atomic(training_state(step, epoch), f"{CKPT_URI}/step-{step}")  # temp key, then commit
        last_save = time.monotonic()
        if stop:
            sys.exit(0)

Where should checkpoints be written so a reclaim cannot eat them?

Write checkpoints to durable object storage in the same region as your GPUs (Amazon S3, Google Cloud Storage, Azure Blob Storage, or Scaleway Object Storage), never only to local NVMe, which dies with the node. Stage locally for speed, then upload, so the GPU pause stays short.

The two-tier pattern works everywhere:

• Local NVMe is the fast staging tier. Write the shard there in seconds.
• Object storage is the durable tier. Upload in a background thread while training continues.

Shared network filesystems (Amazon EFS or FSx for Lustre, Google Cloud Filestore, Azure Files) make multi-node checkpoints simpler because every rank writes to the same path. You pay for that with provisioned throughput or capacity, and high-throughput tiers get expensive fast.

Keep checkpoints in the GPU's region. S3 transfers to AWS services in the same region are free, while inter-region transfer costs $0.01 to $0.02 per GB depending on the region pair (Amazon S3 pricing). A 500 GB checkpoint pulled across regions on every resume adds up, and it slows the restore.

Bucket hygiene: turn on versioning and add lifecycle rules so stale multi-hundred-GB checkpoints expire. Otherwise old checkpoints quietly become your biggest storage line item.

Mount or SDK? Mountpoint for Amazon S3, also available as a CSI driver, gives pods a high-throughput file interface to S3. It can create new files but cannot modify existing ones, which suits write-once checkpoint shards and breaks code that appends or renames. Use the SDK directly when you need explicit atomic commits and retries.

The trap: a checkpoint written only to a PersistentVolumeClaim backed by local or ephemeral node storage does not survive a node reclaim. Check the storage class before you trust a PVC.

How do you catch the interruption notice and flush state in time?

On Kubernetes, the node gets cordoned and your pod receives SIGTERM, so trap it, finish the current step, flush an emergency checkpoint, and exit 0 before the grace period expires. You get one signal and a fixed budget, and the handler belongs in the training loop.

• Set a flag in the handler and do the work in the loop. Saving from inside a Python signal handler, mid-backward pass, produces corrupt or inconsistent state.
• Raise terminationGracePeriodSeconds. Kubernetes defaults it to 30 seconds, then sends SIGKILL (Kubernetes docs). On AWS, set it to around 90 to 100 seconds so it fits inside the two-minute window with margin. On Azure and default Google Cloud Spot VMs, assume about 30 seconds total, no matter what you set.
• Turn the cloud notice into a Kubernetes drain. Karpenter watches an SQS queue fed by EventBridge and "automatically taints, drains, and terminates" nodes ahead of a Spot interruption, once you set --interruption-queue (Karpenter docs). AWS Node Termination Handler does the same for managed node groups, either polling IMDS as a DaemonSet or reading an SQS queue as a Deployment (GitHub). On Google Cloud and Azure, poll the metadata endpoints above or rely on the managed node pool's graceful shutdown.
• Keep the emergency checkpoint small. Write shards in parallel, skip compression, and use async DCP where the framework supports it.
• Make resume idempotent. On start, read the latest committed checkpoint, verify the config hash, and continue from global_step. If no checkpoint exists, start fresh. The same entrypoint handles both cases.

A save that takes longer than the notice window is identical to no save at all. If your 70B checkpoint needs 4 minutes to flush, your emergency checkpoint will never complete on any cloud. In that case, the time-budget checkpoint is your only real protection, and you size the cadence for it.

How do you make Kubernetes restart the run automatically instead of you doing it at 3am?

Run training as a Kubernetes Job (or a JobSet, Kubeflow TrainJob, or Volcano job) with a resume-from-checkpoint entrypoint, so a reclaimed pod is recreated on fresh GPU capacity and continues from the last checkpoint with no human involved. A bare Pod is the wrong unit because nothing recreates it.

• Job settings. The Job controller starts a new Pod when one fails or is deleted, "for example due to a node hardware failure" (Kubernetes docs). restartPolicy must be Never or OnFailure, and backoffLimit defaults to 6. Add a podFailurePolicy rule that ignores the DisruptionTarget condition so spot reclaims don't burn through your retry budget.
• Diversified spot pools. Allow several GPU instance types across several AZs. AWS calls lowest-price the strategy with "the highest risk of interruption" and recommends price-capacity-optimized, while capacity-optimized launches into the most available pools (AWS Auto Scaling docs).
• Taints and tolerations. Taint spot GPU nodes and give only interruption-tolerant training Jobs the toleration. Inference and serving stay on on-demand nodes.
• Elastic distributed training. With torchrun --nnodes=1:4 --max-restarts=3, a failure stops and restarts all workers, and the PyTorch docs tell you to load from checkpoint on every start (PyTorch elastic). Ray Train restarts workers and resumes "by loading the latest checkpoint," with fault tolerance off by default until you set FailureConfig(max_failures=...) (Ray docs).
• Gang scheduling. Kueue adds quota-aware queueing with all-or-nothing admission, and Volcano ensures "all tasks of a job start simultaneously." Without either, a half-scheduled 8-GPU job holds 4 GPUs and does nothing.
• The pragmatic split. Run the bulk of a long run on spot, then move the final stretch or the final fine-tune to on-demand or reserved capacity, where a reclaim costs the most.

Where should you actually run spot GPU training, and where does Qovery fit?

Qovery does not train models, is not a training scheduler, and does not prevent reclaims. It provisions and operates the production Kubernetes cluster on your own AWS, GCP, Azure, Scaleway, or existing Kubernetes cluster, where your spot GPU node pools, checkpoint buckets, and resumable Jobs run. You get Kubernetes without managing it directly and without hiring a platform team.

Why I think this matters for GPUs in particular:

• Bring Your Own Cloud is where the GPU money sits. Capacity reservations, committed-use discounts, Savings Plans, and the cloud bill all stay in your account and your name.
• No cluster babysitting. Qovery handles cluster provisioning, managed upgrades, networking, and per-environment RBAC. ML engineers work with git push and environments instead of node groups and kubeconfigs.
• Idle environments stop. Non-production environments can auto-stop on a schedule, so idle GPU and CPU environments are not billed overnight.
• Preview environments per pull request cover the serving and data-pipeline side of the product, while long training runs as Kubernetes Jobs on the same cluster.
• Clear boundary. Pair Qovery with torchrun elastic, Ray, Kueue, or SkyPilot for job-level scheduling and gang scheduling.

The alternatives each do something better, and you should know which:

• Amazon SageMaker Managed Spot Training does the most for you. It manages Spot interruptions, copies checkpoints to and from S3, and advertises savings of up to 90% (AWS SageMaker docs). It is AWS-only and does not fall back to on-demand.
• SkyPilot is excellent at cross-cloud spot recovery: managed jobs find spot capacity "across regions and clouds" and restart after preemption (SkyPilot docs). Jobs restart from scratch unless you add checkpointing, and you still own whatever clusters it runs on.
• Modal and RunPod are the fastest way to get a GPU running a script, but the GPUs run in their accounts, not yours.
• Run:ai (NVIDIA) and Determined AI (open source, now part of HPE) add GPU scheduling and experiment management on top of a cluster someone still has to operate.
• Raw EKS, GKE, or AKS with Karpenter or the cluster autoscaler gives maximum control and maximum ops burden.

How do I set this up with Qovery from Claude Code or Cursor?

Install the Qovery Agent Skills once, then ask your coding agent in plain language to create the cluster and deploy the workload on your own cloud account. The agent runs the steps and asks you to approve each infrastructure change before it happens.

1. Install the skills globally for Claude Code, Cursor, Codex, OpenCode, or Gemini CLI: curl -fsSL https://skill.qovery.com/install.sh | bash
2. Or scope them to one project: curl -fsSL https://skill.qovery.com/install.sh | bash -s -- --project
3. Add the MCP Server to Claude Code (OAuth, read-only by default): claude mcp add --transport http qovery https://mcp.qovery.com/mcp --callback-port 4242
4. Enable write access only when you want the agent to change infrastructure (you also need to enable write access in the Qovery Console): claude mcp add --transport http qovery "https://mcp.qovery.com/mcp?read_write=true" --callback-port 4242

The skills (qovery-onboard, qovery-deploy, qovery-troubleshoot, qovery-optimize, qovery-speedup, qovery-preview, qovery-terraform) set things up, and the MCP Server operates what already exists.

Then ask: "deploy this training job to my AWS account with Qovery," and approve the plan it proposes. Swap in GCP, Azure, or Scaleway as needed. Full steps are in the AI agent quickstart and on the Claude Code integration page.