How to Share GPUs Fairly Across Teams with Kubernetes Kueue (Instead of Buying More GPUs)

Key Points:

• Kueue holds a Job suspended until its full quota is free, then releases it. Kueue is a Kubernetes-native job queueing controller from the kubernetes-sigs project. It decides when a workload starts and against which quota. kube-scheduler still decides which node each pod lands on.
• Fair GPU sharing takes four objects. You need a ResourceFlavor per GPU type, one ClusterQueue per team with a nominalQuota on nvidia.com/gpu, a shared cohort so idle quota can be borrowed, and a LocalQueue per team namespace that users target with the kueue.x-k8s.io/queue-name label. Three teams fit in under 100 lines of YAML.
• Measure two numbers before you buy hardware. Track average and p95 DCGM_FI_DEV_GPU_UTIL per namespace, plus kueue_pending_workloads and p95 time-to-admission. Low utilization with long queues is a scheduling problem. High utilization with long queues is the only case that justifies new capacity.
• Kueue admits whole workloads and does not partition GPUs. A Workload is only unsuspended when quota for all its pods can be reserved, which prevents two half-placed training jobs from deadlocking. Splitting one physical GPU is the job of MIG, time-slicing or MPS, configured in the NVIDIA device plugin below Kueue.
• Kueue covers admission and fairness, nothing else. Cluster provisioning, GPU node pools, drivers, per-team RBAC and self-service deployment stay your problem. That platform layer is where Qovery fits, running inside your own AWS, GCP, Azure or Scaleway account, or your existing Kubernetes cluster.

Most teams asking for more GPUs have a queueing problem dressed up as a capacity problem. I have watched companies sign multi-year H100 reservations while their existing nodes sat idle every night.

The data backs this up. In ClearML's 2024 survey with the AI Infrastructure Alliance and FuriosaAI, only 7% of companies said their GPU infrastructure exceeds 85% utilization even at peak, and 15% said fewer than half of the GPUs they bought are in use (ClearML, State of AI Infrastructure at Scale 2024). Microsoft's two-month trace of its multi-tenant Philly GPU cluster found gang scheduling and locality constraints were major drivers of queueing delay and lost utilization (Jeon et al., USENIX ATC '19).

This guide shows how to fix that with Kueue. Every YAML block below uses the kueue.x-k8s.io/v1beta2 API from Kueue v0.20.4, checked against the docs on October 31, 2026.

Why do GPUs sit idle while teams complain they cannot get any?

GPUs sit idle because teams pin themselves to dedicated node pools and hold that capacity as insurance, so average utilization collapses while other teams' jobs sit Pending with no ETA. The shortage comes from ownership and missing queueing, and buying hardware makes it worse.

I see the same four anti-patterns in almost every cluster:

1. One namespace per team, pinned to its own GPU node pool. Capacity cannot move between teams even when one pool is empty.
2. nodeSelectors hard-coded in manifests. A job can only run on "its" nodes, so a free H100 next door is invisible to it.
3. "My team paid for these 8 H100s" ownership. Budget lines turn into node pools, and nobody lends what they paid for.
4. Notebooks that reserve a GPU for 8 hours and use it for 20 minutes. The GPU counts as allocated all day.

Plain Kubernetes makes hoarding the rational move. A ResourceQuota is an admission plugin, and when a request exceeds it "the control plane rejects that request with HTTP status code 403 Forbidden" (Kubernetes docs). There is no queue, no ETA and no borrowing. If you give up your quota, you may never get it back.

Vanilla Kubernetes also has no all-or-nothing placement. Two 4-GPU distributed training jobs on a 4-GPU cluster can each get 2 pods placed and wait forever for the rest. The Kueue docs describe this exact deadlock: pods that start "hold onto resources while waiting for the rest" (Kueue all-or-nothing scheduling).

Allocation is also not utilization. A node can be 100% allocated in kubectl describe node and 8% busy. Measure real usage with NVIDIA's dcgm-exporter, whose default counters include DCGM_FI_DEV_GPU_UTIL (GPU utilization in %). Enable its Kubernetes pod mapping (DCGM_EXPORTER_KUBERNETES=true) so each series carries the pod and namespace that holds the GPU (NVIDIA DCGM Exporter docs).

The economics fit in one line. An AWS p5.48xlarge (8x H100) costs $55.04 per hour on demand in us-east-1 (AWS EC2 On-Demand pricing, checked October 31, 2026), and it bills the same at 8% and at 80% utilization. A queue with a guaranteed floor is what makes teams stop hoarding, and that is what Kueue implements.

What is Kubernetes Kueue and what problem does it actually solve?

Kueue is a Kubernetes-native job queueing controller that keeps a workload suspended until its entire resource request fits in an available quota, then unsuspends it so kube-scheduler can place the pods. It adds multi-team quota, borrowing between teams, preemption and whole-workload admission, which ResourceQuota cannot do.

The project describes itself as "a kubernetes-native system that manages quotas and how jobs consume them" and states it "does not replace any existing Kubernetes components" (Kueue overview). It lives in the kubernetes-sigs GitHub org under the Apache 2.0 licence, is developed with WG Batch and SIG Scheduling, and its core APIs are v1beta2 (Kueue on GitHub).

Mechanically, it works in four steps:

1. You create a Job with the kueue.x-k8s.io/queue-name label. Kueue's webhook sets spec.suspend: true (Kueue: Run a Job).
2. Kueue creates a Workload object that summarizes the Job's pod sets and resource requests.
3. When the target ClusterQueue (or its cohort) has quota for the whole Workload, Kueue admits it and injects the flavor's node labels and tolerations.
4. Kueue flips spec.suspend to false, and kube-scheduler places the pods as usual.

The six objects, in plain language:

• ResourceFlavor: what kind of hardware (H100 on-demand, H100 spot, L4).
• ClusterQueue: how much of each flavor a team is guaranteed.
• LocalQueue: the team's entry point inside its namespace.
• Workload: the queued unit Kueue creates for each Job.
• Cohort: the group of ClusterQueues that lend and borrow from each other.
• AdmissionCheck: an external gate, such as ProvisioningRequest for node autoscaling.

Kueue integrates with batch/v1 Job and CronJob, JobSet, Kubeflow Trainer v2 TrainJob, Kubeflow v1 jobs (PyTorchJob, MPIJob, TFJob, JAXJob and more), RayJob, RayCluster and RayService, LeaderWorkerSet, AppWrapper, plain Pods, Deployment and StatefulSet (Kueue integrations index). Deployment support is weaker than Job support. Kueue manages each Deployment pod as an independent Plain Pod, so under quota pressure "the Deployment might run only a subset of Pods" (Kueue: Run Deployment).

Whole-workload admission is the feature that matters for distributed training. By default, Kueue unsuspends a Workload only when quota for all its pod sets can be reserved at once. Quota is an aggregate number though, so Kueue adds waitForPodsReady (evict and requeue if pods are not all ready within a timeout) and Topology-Aware Scheduling, which checks real node capacity before admission (Kueue all-or-nothing scheduling).

Kueue is explicitly not a GPU partitioner (MIG, time-slicing and MPS do that), not a node autoscaler (that is Cluster Autoscaler or Karpenter), not a cost or chargeback tool, and not a replacement for kube-scheduler. Google built the GKE path for its Dynamic Workload Scheduler on Kueue plus the ProvisioningRequest API (Google Cloud blog).

Here is how it compares with the alternatives I get asked about most:

My take: Volcano has the stronger gang-scheduling and HPC heritage, YuniKorn brings deeper queue hierarchies and Spark experience, and Run:ai is the pick when you want fractional GPUs with a vendor support contract. Kueue wins when you want the native option that keeps kube-scheduler, uses a small quota and cohort model, and already speaks the CRDs your ML teams use. If you run a classic Slurm shop with no Kubernetes, stay on Slurm.

How do you set up Kueue for multi-team GPU quota, step by step?

A multi-team GPU setup is five steps: install the NVIDIA GPU Operator so nvidia.com/gpu is countable, install Kueue from its release manifest, declare one ResourceFlavor per GPU type, create one ClusterQueue per team in a shared cohort, and give each team a LocalQueue. The YAML below is copy-pasteable for a 24-GPU cluster of three 8x H100 nodes.

Step 1: make GPUs countable. Install the NVIDIA GPU Operator or the device plugin so nodes advertise nvidia.com/gpu. Label nodes by GPU type and capacity type, and taint them with nvidia.com/gpu:NoSchedule. Kueue can only quota a resource Kubernetes already counts.

Step 2: install Kueue. The current release is v0.20.4 (Kueue releases):

kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/download/v0.20.4/manifests.yaml
kubectl wait deploy/kueue-controller-manager -nkueue-system --for=condition=available --timeout=5m

Step 3: one ResourceFlavor per GPU type. nodeLabels must match your real node labels (on GKE that is cloud.google.com/gke-accelerator, with Karpenter karpenter.sh/capacity-type). Kueue injects the tolerations into admitted pods (Kueue ResourceFlavor):

apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor
metadata:
  name: h100-ondemand
spec:
  nodeLabels:
    gpu-type: h100
    capacity-type: on-demand
  tolerations:
  - key: "nvidia.com/gpu"
    operator: "Exists"
    effect: "NoSchedule"

Repeat for h100-spot, a100-80gb or l4 with the matching labels.

Step 4: one ClusterQueue per team, all in one cohort. Quota cpu and memory alongside nvidia.com/gpu, so a job is never admitted without the CPU its dataloaders need (Kueue ClusterQueue):

apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
  name: research
spec:
  cohortName: gpu-pool
  namespaceSelector:
    matchLabels:
      kubernetes.io/metadata.name: research
  preemption:
    reclaimWithinCohort: Any
    withinClusterQueue: LowerPriority
  resourceGroups:
  - coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
    flavors:
    - name: h100-ondemand
      resources:
      - name: "cpu"
        nominalQuota: 180
      - name: "memory"
        nominalQuota: 1800Gi
      - name: "nvidia.com/gpu"
        nominalQuota: 8
        borrowingLimit: 12

Create applied-ml and inference the same way, changing the namespace selector and limits from the cohort table in the next section.

Step 5: one LocalQueue per team namespace. A data scientist changes exactly one thing in their manifest, the queue label:

apiVersion: kueue.x-k8s.io/v1beta2
kind: LocalQueue
metadata:
  namespace: research
  name: research-queue
spec:
  clusterQueue: research

Submit work. The webhook suspends labelled Jobs for you. Setting spec.suspend: true yourself is harmless and keeps pods from starting if the job is ever created while the webhook is unavailable:

apiVersion: batch/v1
kind: Job
metadata:
  generateName: llm-finetune-
  namespace: research
  labels:
    kueue.x-k8s.io/queue-name: research-queue
spec:
  suspend: true
  parallelism: 2
  completions: 2
  completionMode: Indexed
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: trainer
        image: ghcr.io/your-org/trainer:1.0   # your training image
        resources:
          requests:
            cpu: "16"
            memory: 128Gi
            nvidia.com/gpu: 4
          limits:
            nvidia.com/gpu: 4

Watch it with kubectl -n research get workloads, kubectl -n research describe workload <name> and kubectl get clusterqueue -o wide.

Publish the wait time. Kueue exports kueue_pending_workloads (labelled by cluster_queue and status), kueue_admitted_active_workloads, kueue_admitted_workloads_total and the kueue_admission_wait_time_seconds histogram (Kueue metrics reference). A Grafana panel saying "research waits 12 minutes at p95" is what convinces a team to give up its private node pool:

histogram_quantile(0.95, sum by (le, cluster_queue) (rate(kueue_admission_wait_time_seconds_bucket[1h])))

Common first-run failures:

• The kueue.x-k8s.io/queue-name label is missing, so Kueue ignores the Job.
• ResourceFlavor nodeLabels do not match real node labels, so admitted pods stay Pending.
• Quota covers nvidia.com/gpu but not cpu and memory, so the Workload is inadmissible.
• Jobs are created before the Kueue webhook is ready.
• Job manifests carry their own hard-coded nodeSelector that conflicts with the flavor.

How do cohorts, borrowing and preemption make GPU sharing actually fair?

Fairness comes from putting every team's ClusterQueue in one cohort. nominalQuota is each team's guaranteed floor, borrowingLimit caps how far it can burst into idle capacity, lendingLimit caps what it gives away, and reclaimWithinCohort lets an owner take its floor back by preempting borrowers.

The documented defaults matter, so check them before you copy anyone's YAML (Kueue ClusterQueue):

• nominalQuota: the resources guaranteed to the queue at any time.
• borrowingLimit: if unset, the queue can borrow up to the sum of every other queue's nominal quota in the cohort.
• lendingLimit: if unset, the queue lends out all its unused resources.
• reclaimWithinCohort: defaults to Never. LowerPriority reclaims only from lower-priority borrowers. Any reclaims regardless of priority, and under classic preemption only when the pending workload fits in the queue's own nominal quota.
• withinClusterQueue: defaults to Never. LowerPriority lets a production fine-tune evict the same team's sweep. LowerOrNewerEqualPriority also evicts newer workloads of equal priority.

Priority comes from a WorkloadPriorityClass, attached with the kueue.x-k8s.io/priority-class label:

apiVersion: kueue.x-k8s.io/v1beta2
kind: WorkloadPriorityClass
metadata:
  name: production-finetune
value: 10000
description: "Production fine-tunes beat sweeps"

Classic preemption ranks by priority. Fair Sharing ranks by a share value that measures how much each ClusterQueue borrows, weighted by .spec.fairSharing.weight. You enable Fair Sharing cluster-wide in the Kueue Configuration, not per ClusterQueue, and it does not support borrowWithinCohort (Kueue preemption):

apiVersion: config.kueue.x-k8s.io/v1beta2
kind: Configuration
fairSharing:
  preemptionStrategies: [LessThanOrEqualToFinalShare, LessThanInitialShare]

I use classic priority when teams agree on a ranking of work, and Fair Sharing when the argument is about who has been hogging the cluster.

Preemption only pays off if training checkpoints. A preempted Workload is evicted and requeued, and its pods are terminated. Write checkpoints to object storage at a fixed step interval, handle SIGTERM within the pod's terminationGracePeriodSeconds, and resume from the latest checkpoint on start. Use waitForPodsReady.requeuingStrategy to control backoff, and the kueue.x-k8s.io/max-exec-time-seconds label to cap runaway jobs (Kueue Workload). Our guide to surviving spot GPU reclaim covers the checkpoint side in depth.

Spot capacity plugs in as a second flavor. List h100-ondemand first and h100-spot second, then set flavor fungibility:

spec:
  flavorFungibility:
    whenCanBorrow: TryNextFlavor
    whenCanPreempt: TryNextFlavor

With whenCanBorrow: TryNextFlavor, a job that would have to borrow on-demand quota tries the spot flavor first instead of queueing. The default for whenCanBorrow is MayStopSearch, which borrows from the first flavor.

Here is the cohort I would start with for 24 H100s across three teams:

The worked example: on Saturday, applied-ml and most of inference are idle, so the research sweep borrows 8 GPUs from applied-ml and 4 from inference and runs on 20. On Monday at 9am, applied-ml submits an 8-GPU fine-tune. It fits within its own nominal quota, so on its next scheduling attempt Kueue preempts enough borrowed sweep workloads to free 8 GPUs. The sweep requeues and resumes from its last checkpoint.

For org structures, hierarchical cohorts let a department Cohort sit above team ClusterQueues using spec.parentName, and a Cohort can hold its own shared quota (Kueue Cohort). The Cohort object is part of the v1beta2 API, but the Fair Sharing docs note that the no-preemption-cycle proof does not yet cover the hierarchical case (Kueue Fair Sharing). Start flat and add levels when you have more than one department fighting over one pool.

Does Kueue work for inference, or only for training jobs?

Kueue is built for workloads that finish, such as training, fine-tuning, batch inference and evaluation sweeps. Its Deployment and StatefulSet support is narrower, so long-running online inference belongs on a ClusterQueue with lendingLimit: 0 or a dedicated node pool with a high PriorityClass, never preemptible by training.

Batch and offline inference, evals and dataset generation are just Jobs or JobSets. Queue them exactly like training, and mark them preemptible with a low WorkloadPriorityClass.

Online inference needs guaranteed capacity. The Kueue docs recommend reserving quota for critical serving with lendingLimit so it can scale out fast (Kueue: Run Deployment). Set it to 0 and training can never borrow those GPUs.

Small models waste big GPUs. MIG splits an A100 80GB or H100 80GB into up to 7 isolated 1g.10gb instances (NVIDIA MIG User Guide). Time-slicing and MPS are configured in the NVIDIA device plugin and GPU Operator (NVIDIA GPU sharing). Each method changes what the node advertises, and Kueue quotas whatever units appear. Time-slicing gives no memory isolation, so keep it for dev and small inference.

For multi-node training that must land in one rack or NVLink domain, use Topology-Aware Scheduling. It has been beta since Kueue v0.14 and is enabled by default, using the kueue.x-k8s.io/podset-required-topology annotation on the pod template (Kueue TAS).

Inference endpoints should give GPUs back when idle. Use HPA or KEDA on request rate or queue depth and scale to zero where latency allows, so the cohort can lend that capacity.

For autoscaled clusters, the ProvisioningRequest admission check delays admission until Cluster Autoscaler reports capacity, using classes such as check-capacity.autoscaling.x-k8s.io or best-effort-atomic-scale-up.autoscaling.x-k8s.io (Kueue all-or-nothing scheduling). ProvisioningRequest is a Cluster Autoscaler API. With Karpenter, check your version's support; otherwise Karpenter provisions nodes only after Kueue admits the workload, and waitForPodsReady is your safety net.

Kueue is the wrong tool for single-team clusters with no contention, pure Slurm HPC shops, and serving-only clusters with no batch work.

When is buying more GPUs actually the right answer?

Buy more GPUs only when measured DCGM utilization on the GPUs you own stays high and p95 time-to-admission stays unacceptable after quota, borrowing and preemption are tuned. Every other combination of those two numbers is cheaper to fix in software.

Collect three numbers before any purchase request:

1. Average and p95 DCGM_FI_DEV_GPU_UTIL per team, for example avg by (namespace) (avg_over_time(DCGM_FI_DEV_GPU_UTIL[7d])). With kube-prometheus the pod's namespace often appears as exported_namespace.
2. kueue_pending_workloads and p50/p95 of kueue_admission_wait_time_seconds per ClusterQueue.
3. GPU-hours burned by jobs that were later cancelled or crashed.

My rule of thumb: "low" means average utilization under 50% across a week. Your threshold may differ, but write it down before the budget meeting.

The cheaper levers before capex, by effort: spot or preemptible GPU flavors, right-sizing small models to L4 or A10G instead of H100, MIG for inference, auto-stopping idle dev and staging environments overnight, and moving batch work to a region with available capacity. Our GPU capacity guide covers where to find it.

The idle cost, with the arithmetic. A week has 168 hours. If a team works 10 hours a day for 5 days, 118 hours a week are nights and weekends. That is 118 × 52 / 12 ≈ 511 hours a month, so call it 500. One idle p5.48xlarge at $55.04 per hour costs 500 × $55.04 = $27,520 a month in nights and weekends alone. Three of them, the 24-GPU cluster above, cost $82,560. A p4d.24xlarge (8x A100) at $21.96 per hour costs 500 × $21.96 = $10,980 (AWS EC2 On-Demand pricing, us-east-1, checked October 31, 2026).

Spot is usually the biggest single lever. AWS advertises Spot at up to 90% off On-Demand (AWS EC2 Spot), and Google Cloud lists Spot VMs, GPUs included, at up to 91% off (Google Cloud Spot VMs). Those are ceilings. H100 spot discounts are often lower and change daily, so check current prices in your region before you build a plan on them.

The capex path, honestly: committed-use discounts, Savings Plans and multi-year GPU reservations beat on-demand for steady base load. Size them to your measured floor, the sum of nominal quotas that stay busy, and burst on-demand or spot above it. Running in your own cloud account matters here, because those reservations and discounts stay in your name and follow you, not a vendor.

Who runs the cluster that runs Kueue, and where does the platform layer fit?

Kueue assumes someone already operates a healthy Kubernetes cluster with GPU node pools, drivers, RBAC and a self-service path for teams to ship. For most companies that operational layer, not scheduling, is the real bottleneck, and it is a separate job from Kueue.

What Kueue leaves to you: cluster provisioning and version upgrades, GPU node pool and driver lifecycle, per-team RBAC, environment lifecycle, cost attribution, and a way for data scientists to deploy without filing a ticket. The division of labour in one line: Kueue owns admission and fairness, and the platform owns everything around it.

That platform layer is what we build at Qovery. Qovery deploys and operates your apps inside your own AWS, GCP, Azure or Scaleway account, or on a Kubernetes cluster you already run. Kueue, the NVIDIA GPU Operator and your node pools stay entirely yours, and the cloud bill stays in your name.

The Qovery capabilities that matter to AI teams:

• Managed cluster upgrades, so the Kubernetes version under Kueue does not drift.
• Per-environment RBAC, matching the team boundaries your ClusterQueues encode.
• Git-push deployments for the inference APIs, feature stores and dashboards around your GPU jobs.
• Preview environments per pull request, so a new inference service is tested before it reaches production.
• Environment auto-stop, so GPU-backed non-production environments do not idle overnight.

The boundary, stated plainly: Qovery does not replace Kueue, does not schedule GPUs and does not partition them. You install Kueue on the cluster Qovery manages, or on your own cluster that Qovery connects to. For the broader tool map, see our multi-cloud GPU orchestration comparison.

This combination fits teams shipping AI workloads on their own cloud account who want Kubernetes without operating it directly, and who do not have a dedicated DevOps engineer.

How do I do this with Qovery from Claude Code or Cursor?

Install the Qovery Agent Skills once, and your coding agent can onboard a cluster, deploy the services around your GPU jobs and troubleshoot them in plain language. Every action is scoped and sent to you for approval. Four commands, in this order.

The skills work with Claude Code, Cursor, Codex, OpenCode and Gemini CLI. Install them globally:

curl -fsSL https://skill.qovery.com/install.sh | bash

Or scope them to one repository:

curl -fsSL https://skill.qovery.com/install.sh | bash -s -- --project

Add the Qovery MCP Server to Claude Code. It uses OAuth and is read-only by default:

claude mcp add --transport http qovery https://mcp.qovery.com/mcp --callback-port 4242

Enable write access only when you want the agent to change infrastructure. This also needs write access turned on in the Qovery Console:

claude mcp add --transport http qovery "https://mcp.qovery.com/mcp?read_write=true" --callback-port 4242

The skills (qovery-onboard, qovery-deploy, qovery-troubleshoot, qovery-optimize, qovery-speedup, qovery-preview and qovery-terraform) set things up. The MCP Server operates what already exists.

Then ask in plain language, for example "deploy this inference API to my GCP account with Qovery" or "stop the non-production environments overnight". Details are in the AI agent quickstart and the Claude Code integration page.