How to Autoscale AI Inference Pods on Queue Depth with KEDA (Not CPU)

CPU utilization tells you almost nothing about whether an LLM inference pod is overloaded. The GPU does the matrix math, and the vLLM or Triton process spends most of its life blocked on CUDA calls, so CPU stays flat while requests pile up. If your p99 is timing out and your HPA is sitting at 2 replicas doing nothing, this is why, and queue depth with KEDA is how I fix it.

Key points:

• Autoscale AI inference pods on queue depth, not CPU. A GPU pod batching LLM requests can sit at low CPU while hundreds of requests wait in its internal queue, so a CPU-target HPA never fires and p99 latency breaks first.
• KEDA scales a deployment on the backlog itself: SQS, RabbitMQ or Kafka message count for async inference, or vLLM's vllm:num_requests_waiting gauge scraped by Prometheus for synchronous serving. KEDA graduated from the CNCF on August 22, 2023 (CNCF).
• Set the threshold to the number of queued requests one replica can drain inside your latency SLO. With the default AverageValue metric type, desiredReplicas = ceil(queueDepth / threshold), so a 95-deep queue with threshold 10 gives 10 replicas (KEDA docs).
• Scale-to-zero is the biggest GPU cost lever, and it is not free. A cold start is node provisioning plus a multi-GB image pull plus model load. Use minReplicaCount: 0 for batch, 1-2 warm replicas for interactive endpoints, and an activation threshold so one stray message does not boot a GPU node.
• KEDA scales pods. Karpenter or Cluster Autoscaler scales nodes. You need both, plus Prometheus, the NVIDIA device plugin, GPU taints and workload identity, on EKS, GKE, AKS, Scaleway Kubernetes or a self-managed cluster.

Why does CPU-based autoscaling fail for AI inference pods?

CPU utilization does not track saturation on a GPU inference pod, so a CPU-target HPA scales on noise. The GPU runs the model while the serving process mostly waits on CUDA, and by the time CPU crosses a 70% target, the queue is already deep and p99 has blown past the SLO.

Continuous batching makes it worse. vLLM folds extra concurrent requests into the running batch with almost no extra CPU, so load grows invisibly to the HPA. Google says it plainly in its GKE inference guidance: it does not recommend CPU and memory as the only indicators for GPU inference, because "using CPU metrics alone for autoscaling can lead to suboptimal performance and costs" (Google Cloud).

GPU utilization is a bad trigger too. NVIDIA defines it as the "percent of time over the past sample period during which one or more kernels was executing on the GPU" (NVIDIA nvidia-smi docs). A busy-but-healthy pod and a drowning pod both read close to 100%, and Google notes it "does not measure how much work is being done while the GPU is active" (Google Cloud).

Requests per second gets closer but still misleads. A fixed RPS target encodes an assumption about how fast one replica drains work, and that assumption breaks the day you change the model, the max batch size or the average prompt length.

The failure sequence I have seen in production goes like this: the queue grows, time-to-first-token climbs, clients time out and retry, retries deepen the queue, and the HPA reacts late. The HPA control loop runs every 15 seconds by default, and scale-down uses a 300-second stabilization window (Kubernetes docs), so even a correct signal needs a well-tuned loop. A wrong signal never catches up.

So autoscale on the backlog of work waiting to be done, and leave the resources consumed doing it on your dashboards.

What is KEDA and how does it autoscale Kubernetes pods on queue depth?

KEDA (Kubernetes Event-Driven Autoscaling) is a CNCF Graduated controller that turns an external signal such as queue depth into a Kubernetes autoscaling decision. You declare a ScaledObject, KEDA polls the source every pollingInterval (30 seconds by default), feeds the value to an HPA it creates and owns, and scales the deployment to zero when the queue is empty (KEDA docs).

KEDA ships as three components (KEDA concepts):

• keda-operator watches ScaledObjects, manages the HPA lifecycle and handles the 0 to 1 and 1 to 0 transitions itself.
• keda-metrics-apiserver serves the external metrics to the HPA through the Kubernetes API server.
• keda-admission-webhooks reject broken resources, such as two ScaledObjects targeting the same deployment.

KEDA drives an HPA instead of replacing it. The generated HPA is named keda-hpa-{scaled-object-name} by default, and you tune it through spec.advanced.horizontalPodAutoscalerConfig.behavior (KEDA docs).

The fields that matter for inference, with their documented defaults:

Two per-trigger values do different jobs. The activation threshold decides 0 to 1, and the threshold decides 1 to N, which the HPA executes (KEDA scaling deployments).

For years the stock HPA could not go below 1 replica. Kubernetes 1.37 changed that: the HPAScaleToZero feature gate went beta and is now on by default for object and external metrics (Kubernetes docs). You still need something to serve the queue metric as an external metric, and many managed clusters run older versions, so KEDA remains the practical way to get there today.

KEDA v2.21 lists 80 scaler entries in its catalog, including variants (KEDA scalers), so your broker, Prometheus or cloud queue is almost certainly covered. When a ScaledObject has several triggers, the HPA computes a replica count per metric and takes the largest (Kubernetes docs). That is how you combine queue depth with a latency guardrail.

Which queue depth metric should you scale AI inference on (vLLM, Triton, SQS, Kafka)?

Scale on the metric closest to the work actually waiting. For async or batch inference, that is the broker's message count. For synchronous LLM serving, it is the inference server's own pending-request gauge, which in vLLM is vllm:num_requests_waiting on /metrics.

• vLLM: vllm:num_requests_waiting ("Number of requests waiting to be processed"), vllm:num_requests_running ("Number of requests in model execution batches") and the vllm:time_to_first_token_seconds histogram, all labelled with model_name (vLLM Production Metrics).
• Triton: nv_inference_pending_request_count, the "number of inference requests awaiting execution by a backend", and the cumulative nv_inference_queue_duration_us, served on port 8002 by default (Triton metrics).
• SQS: KEDA counts ApproximateNumberOfMessages plus ApproximateNumberOfMessagesNotVisible by default, because in-flight messages are still being processed. Set scaleOnInFlight: false to count only visible messages (KEDA SQS scaler).
• Kafka: consumer group lag, with lagThreshold defaulting to 10 (KEDA Kafka scaler).

Google's GKE guidance lands in the same place: queue size "directly correlates to request latency" and is "largely workload-agnostic, because queue size is independent of request size, model, or hardware" (Google Cloud). That is why backlog beats RPS. Backlog already reflects arrival rate minus drain rate, so it corrects itself when the model gets slower or a replica lands on a smaller GPU.

Two caveats bite people. On Kafka, KEDA will not scale past the partition count by default, because "extra consumer will have to sit idle" (KEDA Kafka scaler), so size partitions before you size maxReplicaCount. On RabbitMQ and SQS, workers that prefetch messages empty the visible queue before the work is done, so count in-flight messages or you will scale down mid-batch.

How do you write a KEDA ScaledObject for a GPU inference deployment, step by step?

A working queue-depth autoscaler needs four things in order: KEDA installed, the metric source reachable and authenticated, a ScaledObject whose threshold reflects what one replica drains inside your SLO, and a GPU node pool that can actually deliver nodes. Skip the last one and your new pods sit Pending.

Step 1: install KEDA (KEDA deploy docs):

helm repo add kedacore https://kedacore.github.io/charts
helm repo update
helm install keda kedacore/keda --namespace keda --create-namespace
kubectl get pods -n keda

Step 2: wire up the metric source. For vLLM, scrape /metrics with a ServiceMonitor or a Prometheus scrape config. For SQS on EKS, give KEDA an IAM role through IRSA, either on the keda-operator service account (Helm flags --set podIdentity.aws.irsa.enabled=true and --set podIdentity.aws.irsa.roleArn=...) or by reusing your workload's role with identityOwner: workload (KEDA AWS auth). On GKE and AKS, use the GCP and Azure workload identity providers instead.

Step 3: the async ScaledObject on SQS. Parameter names come from the KEDA SQS scaler docs:

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: keda-aws-irsa
  namespace: inference
spec:
  podIdentity:
    provider: aws
    identityOwner: keda   # uses the keda-operator IRSA role
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: batch-embedder
  namespace: inference
spec:
  scaleTargetRef:
    name: batch-embedder          # your GPU Deployment
  pollingInterval: 15
  cooldownPeriod: 300
  minReplicaCount: 0              # batch work can wait for a cold start
  maxReplicaCount: 8              # cap at your GPU quota
  triggers:
  - type: aws-sqs-queue
    authenticationRef:
      name: keda-aws-irsa
    metadata:
      queueURL: https://sqs.us-east-1.amazonaws.com/123456789012/embedding-jobs
      queueLength: "20"           # messages one replica clears inside the SLO
      activationQueueLength: "5"  # do not wake a GPU node for 1-5 messages
      awsRegion: us-east-1
      scaleOnInFlight: "true"     # count NotVisible messages still being processed

Step 4: the synchronous ScaledObject on vLLM. Prometheus scaler fields come from the KEDA Prometheus scaler docs. The third trigger uses metricType: Value because a latency must not be divided by the replica count (KEDA docs):

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-llama
  namespace: inference
spec:
  scaleTargetRef:
    name: vllm-llama
  pollingInterval: 15
  cooldownPeriod: 600
  minReplicaCount: 1              # keep one warm replica for interactive traffic
  maxReplicaCount: 10
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleUp:
          stabilizationWindowSeconds: 0
          policies:
          - type: Pods
            value: 4
            periodSeconds: 30
        scaleDown:
          stabilizationWindowSeconds: 600
          policies:
          - type: Pods
            value: 1
            periodSeconds: 120
  triggers:
  - type: prometheus
    name: waiting
    metadata:
      serverAddress: http://prometheus-operated.monitoring.svc:9090
      query: sum(vllm:num_requests_waiting{model_name="meta-llama/Llama-3.1-8B-Instruct"})
      threshold: "10"
      activationThreshold: "2"
  - type: prometheus
    name: running
    metadata:
      serverAddress: http://prometheus-operated.monitoring.svc:9090
      query: sum(vllm:num_requests_running{model_name="meta-llama/Llama-3.1-8B-Instruct"})
      threshold: "32"             # set to the concurrency one replica holds at your SLO
  - type: prometheus
    name: ttft-p95
    metricType: Value
    metadata:
      serverAddress: http://prometheus-operated.monitoring.svc:9090
      query: histogram_quantile(0.95, sum(rate(vllm:time_to_first_token_seconds_bucket{model_name="meta-llama/Llama-3.1-8B-Instruct"}[2m])) by (le))
      threshold: "2"              # p95 TTFT target in seconds

Step 5: choose the threshold with arithmetic. Load-test one replica and measure how many requests per second it drains at your target latency. Multiply by how long a request may wait before your SLO breaks. Illustrative numbers: a replica drains 2 requests per second and your budget for queue wait is 5 seconds, so the threshold is 2 × 5 = 10 queued requests per replica. KEDA's default AverageValue metric type turns this into desiredReplicas = ceil(currentQueueDepth / threshold), and the KEDA docs give the same example with a queue of 20 and a threshold of 5 producing 4 replicas (KEDA docs). With a queue of 95 and a threshold of 10, you get ceil(9.5) = 10 replicas. Google suggests starting queue-size thresholds between 3 and 5 and raising them until latency hits your target (Google Cloud).

Step 6: tune behavior. Add GPUs fast and remove them slowly. The HPA default scale-up allows 100% or 4 pods every 15 seconds, with no stabilization window, and scale-down waits 300 seconds (Kubernetes docs). For GPUs, I push scale-down stabilization to 600 seconds and remove one pod at a time.

Step 7: the node layer. Karpenter launches nodes for pods the scheduler marks unschedulable (Karpenter docs), and Cluster Autoscaler does the same through node groups. Either way you need a GPU NodePool or node group, an nvidia.com/gpu taint with matching tolerations on the Deployment, the NVIDIA device plugin, and a checked GPU quota for the instance family. KEDA can ask for 10 replicas all day. If the cloud account has quota for 4 GPUs, you get 4.

Step 8: verify before you trust it.

kubectl get scaledobject -n inference
kubectl get hpa keda-hpa-vllm-llama -n inference
kubectl describe scaledobject vllm-llama -n inference   # check Ready and Active conditions

How do you handle GPU cold starts and scale-to-zero without wrecking latency?

Scale-to-zero is the largest GPU cost lever you have, but a cold start stacks node provisioning, a multi-gigabyte image pull, model weight loading and CUDA warmup. Expect minutes rather than seconds, and design around it.

Break the cold start into four parts you can measure on your own cluster: GPU instance launch, NVIDIA driver and device plugin readiness, image pull, and loading weights into VRAM. The image pull alone is easy to measure. AWS measured a 4.93 GB PyTorch image taking 49 seconds to start a pod from Amazon ECR, and 3 seconds once the image was prefetched onto a Bottlerocket data volume (AWS Containers blog). Inference images with CUDA libraries are often bigger, and model weights come on top.

Mitigations, ranked by the impact I have seen:

1. Keep weights out of the pull path. Bake them into a cached image layer, or load them from a pre-warmed volume, EFS, Filestore or Mountpoint for S3.
2. Lazy-load or prefetch images. SOCI on EKS helps most for images above 250 MB compressed (AWS Containers blog), and GKE image streaming lets containers start before the full image lands (Google Cloud blog).
3. Pre-provision headroom with low-priority pause pods that hold a GPU node and get evicted the moment a real replica needs it.
4. Keep a warm floor with minReplicaCount.

Split by pattern: minReplicaCount: 0 for batch and async inference, 1-2 warm replicas for user-facing endpoints, and burst capacity above that from the queue trigger.

Get activationThreshold right. It decides 0 to 1, and KEDA only activates when the value is strictly greater than it (KEDA scaling deployments). The default is 0, which means a single stray message boots a GPU node. Set it to a value that represents real work. If minReplicaCount is 1 or more, activation is ignored entirely.

The stake, in numbers: a g5.xlarge (1 NVIDIA A10G with 24 GB) costs $1.006 per hour on demand in us-east-1 (AWS EC2 pricing, AWS G5). One always-on idle replica costs 730 hours × $1.006 = $734 per month. Scaling it to zero for 12 idle hours a night saves about 360 × $1.006 = $362 per month per replica, before you count larger GPUs.

Guard against thrash. cooldownPeriod only governs the final step to zero (KEDA docs), and the HPA scale-down window governs everything above one. Lengthen both on GPUs, because every flap buys you another full cold start.

KEDA vs HPA vs Karpenter vs KServe: which do you actually need for AI inference?

These are layers, and for inference you almost always run KEDA plus a node autoscaler together. KEDA decides how many pods, the HPA is the mechanism it drives, Karpenter or Cluster Autoscaler decides how many GPU nodes, and KServe or Ray Serve are serving frameworks with their own autoscalers.

The stack, top to bottom: trigger source, KEDA ScaledObject, managed HPA, pending pods, Karpenter or Cluster Autoscaler, cloud GPU quota. Any one of those can block a scale-up, and I debug them in that order.

• HPA with Prometheus Adapter reaches the same signal, but you maintain adapter rules by hand, and scale-to-zero needs Kubernetes 1.37 or later (Kubernetes docs).
• KServe scales on in-flight requests per pod through the Knative autoscaler and supports minReplicas: 0 in Knative mode (KServe docs). Great if you adopt the whole stack. KEDA works with the Deployment you already run.
• Ray Serve targets ongoing requests per replica, supports min_replicas: 0, and asks the Ray Autoscaler for nodes (Ray docs). Best when you already run Ray.
• SageMaker asynchronous inference scales on backlog per instance and can go to zero (AWS docs). Vertex AI scales on CPU, GPU duty cycle or request count, with vLLM queue metrics and scale-to-zero in Preview (Google Cloud). Both trade cluster-level control and cross-cloud portability for convenience.
• NVIDIA Run:ai sits at a different layer again: GPU scheduling and sharing across teams, rather than per-deployment queue scaling.

Skip KEDA when traffic is steady and you serve a single model. A fixed replica count is cheaper to operate than another controller.

Who operates the Kubernetes and GPU layer underneath the autoscaler?

Queue-depth autoscaling only pays off if someone keeps the layer beneath it healthy, and that layer is large. GPU node pools and AMIs, NVIDIA drivers and the device plugin, taints and tolerations, Prometheus storage, KEDA and CRD upgrades, IRSA or workload identity bindings, and a Kubernetes version upgrade every few months. This is where teams without a dedicated platform engineer stall.

This is the part we built Qovery for. Qovery provisions and operates Kubernetes inside your own AWS, GCP, Azure or Scaleway account, or connects to a Kubernetes cluster you already run. The GPU bill, reservations, Savings Plans and data all stay in your name, because the cluster lives in your account.

The ownership split is clean. Your team keeps the ScaledObject, the model, the queue and the SLO. Qovery takes cluster provisioning, managed cluster upgrades, per-environment RBAC and git-push deployments. Environment auto-stop is the non-GPU twin of scale-to-zero: idle preview and staging environments stop on schedule instead of burning nodes overnight.

To be straight about the boundary: Qovery does not replace KEDA, Karpenter or Prometheus. You still install and tune the autoscaler. Qovery removes the cluster operations around it, so a small team can ship GPU inference without hiring someone just to babysit Kubernetes.

How do I set this up with Qovery from Claude Code or Cursor?

Install the Qovery Agent Skills once, then ask your coding agent in plain language to deploy the inference service onto a cluster in your own cloud account. The agent proposes every change and waits for your approval before touching infrastructure.

curl -fsSL https://skill.qovery.com/install.sh | bash

# Or scoped to the current project
curl -fsSL https://skill.qovery.com/install.sh | bash -s -- --project

# Qovery MCP Server in Claude Code (OAuth, read-only by default)
claude mcp add --transport http qovery https://mcp.qovery.com/mcp --callback-port 4242

# Write access, only when you want the agent to change things (also enable it in the Qovery Console)
claude mcp add --transport http qovery "https://mcp.qovery.com/mcp?read_write=true" --callback-port 4242

The skills (qovery-onboard, qovery-deploy, qovery-troubleshoot, qovery-optimize, qovery-speedup, qovery-preview, qovery-terraform) deploy, and the MCP Server operates what already exists. Then ask, for example: "deploy this vLLM service to my AWS account with Qovery and show me the GPU node pool on the cluster". See the AI agent quickstart and the Claude Code integration.