How to Move a Databricks or SageMaker Training Pipeline to Kubernetes (Without Losing a Month)

Key points:

• Moving a Databricks or SageMaker training pipeline to Kubernetes means replacing five layers: job orchestration (Argo Workflows, Kubeflow Pipelines, Flyte or KubeRay), distributed training (Kubeflow Trainer with Kueue or Volcano for gang scheduling), the GPU runtime (NVIDIA GPU Operator plus labeled, tainted node pools), data and artifacts (S3, GCS, Azure Blob or Scaleway Object Storage) and experiment tracking (MLflow or Weights & Biases).
• The cost gap is built into the pricing model. Databricks bills DBUs on top of the cloud instance, and SageMaker prices ml.* training instances 15-40% above the same EC2 hardware in us-east-1. On Kubernetes you pay cloud list price, and you keep spot GPUs, GPU sharing, scale-to-zero, Savings Plans and committed-use discounts in your own account.
• Direct GPU access is what makes the move stick: you pin the driver, CUDA and container toolkit together, slice an A100 or H100 into up to seven MIG instances, time-slice GPUs for notebooks and tune NCCL over EFA or InfiniBand for multi-node runs.
• The safe sequence has seven steps: containerize, externalize data paths, provision GPU node pools, prove one GPU job, move orchestration, move tracking, then parallel-run a full retraining cycle. A big-bang DAG rewrite turns a three-week migration into a quarter.
• Do not migrate below the break-even. Under a few hundred GPU-hours a month, with no platform engineer, or with deep Unity Catalog or SageMaker Feature Store coupling, SageMaker Managed Spot Training or Databricks Jobs Compute is still the cheaper total answer. Above it, Qovery runs the cluster layer inside your own AWS, GCP, Azure, Scaleway or existing Kubernetes cluster so the ML team ships training jobs without a dedicated DevOps engineer.

One SageMaker training hour on an ml.g5.xlarge costs $1.408 in us-east-1. The same g5.xlarge, same NVIDIA A10G, costs $1.006 on raw EC2 (both checked November 2026 from the AWS SageMaker price list and EC2 on-demand pricing). That is a 40% premium for identical silicon.

On Azure Databricks, a Jobs cluster on one NC24ads A100 v4 costs $6.67 per hour in East US: $3.67 for the VM plus $3.00 of DBUs (Azure Databricks pricing, checked November 2026). The DBUs nearly double the GPU bill.

Those two numbers are why ML teams call us. This guide is the migration plan we walk them through, including the cases where we tell them to stay put.

Why do teams move training pipelines off Databricks or SageMaker?

Teams move training off Databricks or SageMaker for three reasons: a per-unit markup on GPU compute (DBUs on top of the instance on Databricks, an ml.* premium of 15-40% on SageMaker), no control over the driver and CUDA stack, and proprietary job formats that make every future workload harder to move. Cost usually triggers the project. GPU control is what keeps teams from going back.

• Databricks pricing in one sentence: classic Jobs Compute is billed in DBUs "plus underlying compute costs billed by cloud provider" (Databricks Lakeflow Jobs pricing), so every GPU hour carries two line items. Premium Jobs Compute is $0.30/DBU on Azure East US (Azure Retail Prices API, checked November 2026); legacy Standard tier is $0.10/DBU on AWS and GCP.
• SageMaker pricing in one sentence: training runs on ml.* instances billed at a separate, higher rate than the matching EC2 instance. In us-east-1 that is +40% on ml.g5.xlarge and +15% on both ml.p4d.24xlarge and ml.p5.48xlarge.
• GPU control: pinning driver and CUDA versions, MIG partitioning, time-slicing, NCCL tuning, interconnect choice for multi-node runs, and running a PyTorch build the managed image does not ship yet.
• Lock-in comes in two shapes. Databricks: notebooks, Workflows, DBFS paths, Delta Lake, Unity Catalog and TorchDistributor. SageMaker: Estimators, the Pipelines SDK, input channels, /opt/ml entrypoint conventions and the Model Registry.
• Quota friction: SageMaker tracks training capacity as its own per-instance-type quotas in Service Quotas, separate from your EC2 quotas. Teams with large runs end up negotiating capacity twice.

When not to migrate: fewer than a few hundred GPU-hours a month, no one on the team who can own a cluster, or a pipeline wired deeply into Unity Catalog or SageMaker Feature Store. The break-even is utilization and headcount.

The honest trade: you swap a vendor markup for operating responsibility, and the swap only pays above a utilization threshold.

What exactly do you have to replace when you move training to Kubernetes?

A managed ML platform bundles five layers (job orchestration, distributed training, the GPU runtime, data and artifacts, experiment tracking), while Kubernetes only schedules containers. Map each layer to a named replacement before you write a line of YAML, because the layer you forget is the one that blocks the cutover.

• Job orchestration: SageMaker Pipelines and Databricks Workflows map to Argo Workflows (CNCF graduated, v4.1), Kubeflow Pipelines (Kubeflow is CNCF graduated since July 2026), Flyte (Linux Foundation, backed by Union.ai), Metaflow (Netflix origin, backed by Outerbounds) or KubeRay (Ray project, backed by Anyscale).
• Distributed training: the Estimator distribution config or TorchDistributor maps to Kubeflow Trainer v2.4 and its TrainJob API. It replaced the Training Operator's PyTorchJob and TFJob, which Kueue now lists as deprecated. Trainer v2 still calls itself alpha, so pin the version.
• Gang scheduling and queues: Kueue (v0.20, a kubernetes-sigs project from the Batch Working Group) or Volcano (CNCF incubating, v1.16).
• GPU runtime: the managed AMI or Databricks ML Runtime maps to the NVIDIA GPU Operator (driver, device plugin, container toolkit, DCGM) plus node labels, taints and a MIG or time-slicing profile.
• Data and artifacts: DBFS paths, Delta tables, SageMaker default buckets and input channels map to plain object storage URIs on S3, GCS, Azure Blob or Scaleway Object Storage, plus Feast if you used a feature store.
• Tracking and registry: self-hosted or managed MLflow, or Weights & Biases, replace the SageMaker Model Registry or Databricks-hosted MLflow.
• Identity: IRSA or EKS Pod Identity on AWS, Workload Identity Federation on GKE, Entra Workload ID on AKS. This is the piece that silently fails the very first job with an AccessDenied on the training bucket.
• Serving, in one line: KServe (CNCF incubating), Ray Serve, or a plain container behind a Service. Keep it out of scope for the training migration.

The day ranges are our field estimates for one pipeline and one ML engineer, not a benchmark.

Is GPU training on Kubernetes actually cheaper than SageMaker or Databricks?

Yes, for sustained GPU use: on Kubernetes you pay the raw cloud instance price, while SageMaker charges 15-40% more for the same EC2 hardware in us-east-1 and Databricks adds DBUs that range from +4% on an 8x H100 VM to +82% on a single-A100 VM on Azure. The bigger saving comes from three levers you control on Kubernetes: spot GPUs, GPU sharing and scale-to-zero node pools, all billed to your own account where Savings Plans and committed-use discounts apply.

Spot columns are point-in-time snapshots and move daily. The SageMaker and Azure figures come from each vendor's machine-readable price list.

Two honest readings of that table. SageMaker's premium is a flat tax of 15% or more on every training hour. The Databricks markup depends on the DBU rating of the VM: large 8-GPU nodes carry a small DBU uplift, while small single-GPU nodes can nearly double. Databricks classic compute also runs the VM in your own cloud account, so the instance line already benefits from your discounts. The DBU line does not.

Lever 1, spot GPUs with checkpointing. AWS advertises up to 90% off on Spot Instances, Google up to 91% on Spot VMs including GPUs, and Azure up to 90% on Spot VMs. Real discounts on GPU families are smaller and volatile: our snapshot shows 17% off on p4d and 82% off on the Azure H100 VM.

Lever 2, GPU sharing. MIG gives hard isolation on A100 and H100; time-slicing covers notebooks and dev jobs. Large distributed runs should get whole GPUs.

Lever 3, scale-to-zero and bin-packing. Karpenter deletes empty nodes and consolidates underutilized ones on AWS; Cluster Autoscaler or the managed equivalent does the same on GKE, AKS and Scaleway Kapsule.

The BYOC point: AWS Savings Plans go up to 72% off on-demand, and Google resource-based committed-use discounts cover Cloud GPUs. Those commitments apply to EC2 and Compute Engine spend in your account. SageMaker needs its own separate SageMaker Savings Plan, and DBUs need a Databricks commit.

Be fair to the managed option. SageMaker Managed Spot Training "can optimize the cost of training models up to 90%" with checkpointing to S3 handled for you. If your runs are bursty, single-node and checkpoint well, flip EnableManagedSpotTraining=True and skip the migration. Note that it never falls back to on-demand: if spot capacity is gone, the job waits until MaxWaitTimeInSeconds and stops.

The counterweight: an idle GPU node is the fastest way to erase the saving. One idle p5.48xlarge at $55.04/h costs $1,321 a day and about $40,180 a month (730 hours). One idle Azure ND96isr H100 v5 costs about $71,770 a month. Add the cluster control plane, monitoring, and the engineer time to run all of it.

Rule of thumb: the math flips when GPUs are busy more than about half the month, or when more than one team shares the same node pool.

What does a step-by-step migration plan look like?

Migrate one training step at a time and run it beside the existing pipeline until the metrics match: containerize, externalize data paths, provision GPU node pools, prove exactly one GPU job, move orchestration, move tracking, then parallel-run and cut over. That sequence takes weeks. A big-bang DAG rewrite takes a quarter.

1. Containerize the training code. Lift it out of notebooks or Estimators into a plain train.py and a Dockerfile on a pinned CUDA base image. It still runs on the old platform (SageMaker accepts custom containers, Databricks accepts container services), which makes it the biggest de-risking step.
2. Externalize config and data paths. Replace dbfs:/ paths and /opt/ml/input/data/<channel> conventions with environment variables and object storage URIs. Keep a compatibility shim for one cycle.
3. Provision GPU node pools. Tainted, labeled GPU node groups, the NVIDIA GPU Operator, an autoscaler (Karpenter on AWS, Cluster Autoscaler or the managed equivalent on GKE, AKS and Kapsule), and one spot pool with an on-demand fallback pool.
4. Run exactly one job. A single TrainJob or Argo step. Validate samples per second, NCCL initialization and the checkpoint write to object storage before touching the DAG.
5. Move orchestration. Port the DAG to Argo Workflows, Kubeflow Pipelines or Flyte, and make retries and artifact passing explicit.
6. Move tracking and the registry. Stand up MLflow and dual-log to both systems for the whole parallel run.
7. Parallel run and cut over. Compare eval metrics, wall-clock and cost per completed run across at least one full retraining cycle, then retire the old pipeline.

Rollback rule: keep the managed pipeline runnable for one full retraining cycle after cutover, and keep old artifacts addressable at their original URIs.

These durations are estimates from migrations we have seen, not a benchmark. Add a week if identity and networking belong to another team.

How do you get direct GPU access right on Kubernetes (drivers, sharing, spot, multi-node)?

Direct GPU access on Kubernetes comes down to four decisions: how drivers are managed (NVIDIA GPU Operator or pre-baked node images), how jobs are scheduled (gang scheduling with Kueue or Volcano), how GPUs are shared (MIG or time-slicing), and how interruptions are handled (checkpointing plus an on-demand fallback pool). Get them right and throughput matches the managed platform. Get them wrong and you pay on-demand prices for idle accelerators.

• Drivers. Use the GPU Operator (v26.7 at the time of writing) or bake drivers into the node image, but pin driver, CUDA and container toolkit versions together. Roll upgrades to a canary node pool first.
• Scheduling. A 16-worker job that gets 12 GPUs and waits for 4 more holds 12 idle GPUs. Kueue's all-or-nothing admission or Volcano's gang scheduling prevents that. Add per-team ClusterQueue quotas and cohorts from day one.
• Sharing. NVIDIA's MIG profiles split an H100 80GB into up to seven 1g.10gb instances, or 3x 2g.20gb, 2x 3g.40gb, or one 7g.80gb; the A100 80GB uses the same layout. Time-slicing shares a GPU across more pods but, in NVIDIA's words, offers "no memory or fault-isolation between replicas". Use MIG for small production jobs, time-slicing for notebooks, and neither for multi-GPU training.
• Spot. AWS gives a two-minute interruption notice, Azure 30 seconds via scheduled events, and Google lets you set a 120-second notice or none (the default is 0). Checkpoint to object storage every N steps so that a lost node costs minutes, not hours. Karpenter launches a replacement as soon as it sees the warning.
• Multi-node. On AWS, EFA supports NCCL with OS bypass on p4d and p5, at no extra cost. On Google Cloud, use GPUDirect networking on A3; on Azure, InfiniBand on ND-series VMs. An NCCL ring falling back to TCP can stretch a large run enough to erase the whole cost saving, so run nccl-tests on every new pool.
• Observability. DCGM Exporter feeds Prometheus and Grafana with DCGM_FI_DEV_GPU_UTIL (shipped as DCGM_FI_DEV_GPU_UTIL_RATIO in current default counters) and the more precise DCGM_FI_PROF_GR_ENGINE_ACTIVE. Sustained utilization well under half usually points to a data-loading bottleneck.

The utilization problem is measured. In Microsoft's analysis of its Philly cluster (Jeon et al., USENIX ATC 2019), 8-GPU jobs used GPU cycles 56.9% of the time on average, and 16-GPU jobs only 34.3%. The authors conclude that "GPUs are underutilized in shared clusters." Sharing and bin-packing are how you buy that capacity back.

Who operates the cluster afterwards, and what are the alternatives to Qovery?

Kubernetes fixes GPU access and the vendor markup, but someone still has to upgrade the cluster at 2am, and there are five realistic answers: raw EKS/GKE/AKS with a platform team, a self-managed Kubeflow or Flyte stack, a managed portable runtime such as Anyscale or Union.ai, an internal developer platform such as Qovery inside your own cloud account, or keeping part of the pipeline on SageMaker or Databricks.

The people problem is real. In the CNCF Annual Cloud Native Survey (628 respondents, September 2025), 66% of organizations already host generative AI workloads on Kubernetes, and "lack of training" ranks as the second-biggest challenge at 36%.

Qovery is an internal developer platform that deploys and operates applications inside your own AWS, GCP, Azure or Scaleway account, or on your existing Kubernetes cluster. That is BYOC: the cloud bill, Savings Plans and committed-use discounts stay in your name. What it does: git-push deployments, preview environments per pull request, environment auto-stop for non-production, managed cluster upgrades, per-environment RBAC, and databases backed by managed cloud services.

Be precise about the split. Qovery operates the cluster and the self-service deployment layer; Kubeflow Trainer, Argo Workflows, Ray and MLflow run on top as workloads you control. Qovery is not an MLOps, training-orchestration or experiment-tracking product.

Hybrid is legitimate and common: keep Delta Lake or Unity Catalog as the data layer, move GPU training to Kubernetes, and keep inference wherever it already works.

Decision rule: pick based on how many engineers you can dedicate to the platform, not on which tool has the nicest docs.

What should you check before and after the cutover?

Validate four axes before deleting the old pipeline: numerical parity of the trained model, wall-clock per run, cost per completed run, and GPU utilization measured with DCGM. If any one regresses, fix it first, and keep the managed pipeline runnable for one more retraining cycle.

Compliance is often the reason the move clears review. Training inside your own account and region keeps personal data under your control, which matters in finance, healthcare and the EU, where GDPR restricts transfers of personal data outside the EEA.

How do I run this migration from Claude Code or Cursor with Qovery?

Install the Qovery Agent Skills once and your coding agent (Claude Code, Cursor, Codex, OpenCode or Gemini CLI) can scaffold the containerized training job, deploy it to a cluster in your own cloud account, and troubleshoot the first run, asking you to approve each action before it happens.

Install globally for every compatible agent:

curl -fsSL https://skill.qovery.com/install.sh | bash

Or scope it to one project:

curl -fsSL https://skill.qovery.com/install.sh | bash -s -- --project

Add the MCP Server to Claude Code (OAuth, read-only by default) so the agent can inspect what already runs:

claude mcp add --transport http qovery https://mcp.qovery.com/mcp --callback-port 4242

Enable write access only when you want the agent to change things. You also need to enable write access in the Qovery Console:

claude mcp add --transport http qovery "https://mcp.qovery.com/mcp?read_write=true" --callback-port 4242

The skills that matter here are qovery-onboard, qovery-deploy, qovery-troubleshoot, qovery-optimize, qovery-speedup, qovery-preview and qovery-terraform. The skills deploy; the MCP Server operates what already exists. Then ask in plain language: "Deploy this training container to my AWS account with Qovery" or "Why did this training job fail?"

Docs: AI agent quickstart and Claude Code integration.