How to Run Falcon, Mixtral, and Other Open-Weight Models on One Kubernetes GPU Cluster

Key Points:

• One GPU node pool can serve Falcon, Mixtral, Llama, and Qwen at the same time. Install the NVIDIA GPU Operator once, then separate models with resource requests, node labels, and namespaces. GPU memory and isolation are the constraints, never Kubernetes.
• Four sharing mechanisms stack: time-slicing (no memory or fault isolation), MIG (hardware isolation on A100, H100, H200, B200), MPS (experimental in the device plugin), and multi-model servers like vLLM, NVIDIA Triton, Ray Serve, or KServe ModelMesh.
• Co-location is memory math: weights + KV cache + 10-20% overhead must fit in VRAM. Mixtral 8x7B keeps all 46.7B parameters resident, so at bf16 it overflows one 80 GB card, while three Llama 3.1 8B servers share one.
• Create a second GPU pool only for a different GPU SKU, purchasing mode, hard isolation boundary, or CUDA driver version. Never for a new model.
• Qovery deploys and operates these workloads in your own AWS, GCP, Azure, Scaleway, or existing Kubernetes cluster, so GPU reservations and committed-use discounts stay yours and idle non-production environments auto-stop.

Can you run Falcon, Mixtral, and Llama on the same Kubernetes GPU cluster?

Yes. One GPU node pool running the NVIDIA GPU Operator can serve Falcon, Mixtral, Llama, Qwen, and an embedding model side by side, separated by resource requests, node labels, and namespaces. There are only four reasons to split into a second pool: a different GPU SKU, a different purchasing mode, a hard tenant isolation boundary, or a different CUDA driver version.

I will say it bluntly: one GPU node pool per model is a copy-paste artifact of vendor quickstarts, not an architecture decision. Every model README ships a Helm chart with its own nodeSelector and a fresh managed node group. Six models later, you have six pools nobody designed, each with an autoscaler floor keeping a GPU node warm, its own slice of fragmented quota, its own driver rollout, and never enough slack to drain.

The math on the floors alone: a Scaleway H100-1-80G lists at about €2.87 per hour in PAR-2 (Scaleway GPU pricing), so one idle node costs roughly €2,090 over a 730-hour month. Six pools is about €12,500 a month before a single token is generated. Idle capacity is the norm, too: in ClearML's 2024 survey, only 7% of respondents reported more than 85% GPU utilization even at peak (ClearML).

The method below has three steps: pick a sharing mechanism, size by VRAM, then schedule with labels, taints, and priority classes. Distributed training, multi-node tensor parallelism, and fine-tuning schedulers are out of scope.

What are the ways to share one GPU across multiple models on Kubernetes?

Four mechanisms exist and they compose: NVIDIA time-slicing, MIG, MPS, and a multi-model inference server. Time-slicing plus a multi-model server is the default for most teams, and MIG is the only one with hardware-enforced memory isolation, on A100, H100, H200, B200 and a handful of other GPUs.

Time-slicing (sharing.timeSlicing.resources with replicas: N) advertises one physical GPU as N schedulable nvidia.com/gpu units. NVIDIA is explicit: "Unlike Multi-Instance GPU (MIG), there is no memory or fault-isolation between replicas" (NVIDIA GPU Operator docs).

MIG partitions the GPU in hardware. An H100 80GB offers profiles such as 1g.10gb (up to 7), 2g.20gb (up to 3), 3g.40gb (up to 2), and 7g.80gb (NVIDIA MIG user guide), and A100, H100, H200, B200, and GB200 all support seven instances (NVIDIA). Repartitioning hurts: "MIG Manager requires that no user workloads are running on the GPUs being configured" (NVIDIA GPU Operator MIG docs).

MPS runs kernels from several processes concurrently. The device plugin supports it via sharing.mps but marks it experimental, caps each client at an equal memory fraction, and rules out combining it with MIG or time-slicing (NVIDIA k8s-device-plugin). A fatal fault in one client can still affect others on that GPU (NVIDIA MPS docs).

Multi-model servers put the sharing logic in the application layer:

• NVIDIA Triton serves every model in a model repository from one server, and instance_group controls how many instances land on which GPU.
• Ray Serve accepts fractional GPUs (num_gpus: 0.5) and runs multiple independent applications on one cluster.
• KServe ModelMesh is built "for high-scale, high-density and frequently-changing model use cases" and acts as a distributed LRU cache of loaded models.
• vLLM serves one model per server, but several vLLM servers can share a GPU when each one caps --gpu-memory-utilization.

Dynamic Resource Allocation (DRA) is the future of shared device requests: its core APIs went GA in Kubernetes v1.34 (Kubernetes blog) and are listed as stable in the docs. Most GPU fleets still use the device plugin, so I do too.

The default I recommend: time-slicing for dev and small models, MIG for production multi-tenant isolation, and a multi-model server for many small models behind one endpoint.

How much GPU memory do Falcon, Mixtral, Llama, and Qwen actually need?

Whether two models can share a card comes down to one calculation: weights + KV cache + roughly 10-20% runtime overhead must fit in VRAM with headroom. Mixtral 8x7B must hold all 46.7B parameters even though only 12.9B are active per token (Mistral AI), so at bf16 it needs about 93 GB for weights alone and spills past a single 80 GB GPU.

The formula, with every result below an estimate rather than a benchmark:

VRAM ≈ params x bytes_per_param
     + KV cache (2 x layers x kv_heads x head_dim x tokens x bytes)
     + 10-20% overhead

Worked example for Llama 3.1 8B, which has 32 layers, 8 KV heads, and a head dimension of 128 (model config). Weights at bf16: 8B x 2 bytes ≈ 16 GB. KV cache per token: 2 x 32 x 8 x 128 x 2 bytes = 128 KiB. Four concurrent 8k-token sequences add about 4 GB. With 15% overhead, the estimate lands around 23 GB, so three of these servers fit on one 80 GB card.

Mixture-of-experts burns people: memory scales with total parameters, throughput with active ones. Mixtral computes like a 13B model and occupies memory like a 47B one.

Long context is the other trap. Llama 3.1 supports 128K tokens (Meta), and at 128 KiB per token one 8k sequence needs about 1 GiB of KV cache while one 128k sequence needs about 16 GiB, more than the weights. Cap --max-model-len to what your product uses.

Quantization is the biggest lever: 2 bytes per parameter at bf16, 1 at FP8, roughly half a byte plus scales at 4-bit. vLLM supports AWQ, GPTQ, FP8, bitsandbytes, GGUF, and more (vLLM quantization docs). Check quality on your own eval set first.

Three recipes you can copy:

1. Three 7-8B models at bf16 on one 80 GB card, each a vLLM server with --gpu-memory-utilization 0.3 and a capped --max-model-len. A fourth fits if you switch them to FP8.
2. Mixtral 8x7B at 4-bit plus an embedding model on one 80 GB card, Mixtral capped at around 0.8 and the embedding server in the remainder.
3. One 7-8B model per MIG 3g.40gb slice, two per H100, with hard memory isolation between them.

Watch the KV cache pre-allocation: vLLM's --gpu-memory-utilization defaults to 0.92 (vLLM engine args), and "vLLM pre-allocates GPU cache using this percentage of memory" (vLLM optimization guide). Two uncapped pods on one time-sliced GPU each try to claim 92% of the card, and the second fails at startup. Set the flag on every server so the values sum below 1.

How do you schedule multiple models on one GPU node pool in Kubernetes?

Keep one node pool and move placement into each Deployment: a nodeSelector on GPU Feature Discovery labels, a toleration for the GPU taint, a right-sized nvidia.com/gpu or nvidia.com/mig-* request, and a PriorityClass. GPU Feature Discovery labels every node with nvidia.com/gpu.product, nvidia.com/gpu.memory, nvidia.com/gpu.count, and MIG labels such as nvidia.com/mig.strategy (GPU Feature Discovery).

Install the NVIDIA GPU Operator once per cluster: one Helm release brings the driver, container toolkit, device plugin, GPU Feature Discovery, DCGM monitoring, and MIG Manager. Then enable time-slicing as in the sharing docs:

apiVersion: v1
kind: ConfigMap
metadata:
  name: time-slicing-config
  namespace: gpu-operator
data:
  any: |-
    version: v1
    flags:
      migStrategy: none
    sharing:
      timeSlicing:
        resources:
          - name: nvidia.com/gpu
            replicas: 4

Wire it in with kubectl patch clusterpolicies.nvidia.com/cluster-policy -n gpu-operator --type merge -p '{"spec": {"devicePlugin": {"config": {"name": "time-slicing-config", "default": "any"}}}}'. Time-sliced nodes get a -SHARED suffix on nvidia.com/gpu.product, so the Deployment selects on that:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llama-31-8b
  namespace: inference
spec:
  replicas: 1
  selector:
    matchLabels: { app: llama-31-8b }
  template:
    metadata:
      labels: { app: llama-31-8b }
    spec:
      priorityClassName: inference-prod
      nodeSelector:
        nvidia.com/gpu.product: NVIDIA-H100-80GB-HBM3-SHARED
      tolerations:
        - key: nvidia.com/gpu
          operator: Equal
          value: present
          effect: NoSchedule
      containers:
        - name: vllm
          image: vllm/vllm-openai:latest
          args: ["--model", "meta-llama/Llama-3.1-8B-Instruct",
                 "--gpu-memory-utilization", "0.3", "--max-model-len", "8192"]
          resources:
            limits:
              nvidia.com/gpu: 1

Confirm the product string with kubectl get nodes -L nvidia.com/gpu.product. GKE adds the nvidia.com/gpu=present:NoSchedule taint automatically when the cluster also has a non-GPU pool (GKE docs); elsewhere, add it yourself. For MIG under mig.strategy=mixed, label the node with nvidia.com/mig.config and request a slice:

resources:
  limits:
    nvidia.com/mig-1g.10gb: 1

The 1g.10gb slice suits the embedding model; a 7-8B model takes nvidia.com/mig-3g.40gb: 1.

• Priority. A high PriorityClass for production inference and a low one for evals, so evals get preempted and never the reverse. Add a PodDisruptionBudget to every serving Deployment.
• Consolidation. Karpenter's default WhenEmptyOrUnderutilized policy removes or replaces nodes to cut cost (Karpenter docs), AKS runs Karpenter through Node Auto Provisioning, and GKE has node auto-provisioning. One shared pool gives the autoscaler something it can actually drain.
• Observability. DCGM exporter metrics (GPU utilization, framebuffer used), per-model latency and queue depth, and OOMKill alerts.
• Weights. Cache weights on a shared PVC or node-local disk via an initContainer so restarts do not re-download tens of GB. Cold start is the hidden cost of scale-to-zero, so measure it.

When should you still create a separate GPU node pool?

Split pools by GPU SKU, purchasing mode, and isolation boundary, never by model. A second pool is justified in four cases: the model needs a different GPU class, tenants must be physically separated, you want a spot pool for batch and evaluation work, or CUDA driver requirements diverge.

Falcon 180B needs multi-GPU 80 GB-class nodes with fast interconnect while a 7B model runs on a 24 GB L4. Some auditors reject MIG and require physical separation. Evals belong on spot, production on reserved. Nodes needing EFA or InfiniBand are their own class.

The rule: one pool per GPU SKU, per purchasing mode, per isolation boundary. If your pool count equals your model count, something went wrong. A pool per team is an org chart, and namespaces, ResourceQuotas, and per-environment RBAC handle it better.

What does this look like on AWS, GCP, Azure, Scaleway, or your own cluster?

The mechanism is identical on every cloud because it is NVIDIA plus Kubernetes. What changes is the instance family, the quota process, and the discount model. That last one is the reason to keep the cluster in your own cloud account: Capacity Blocks, Savings Plans, reservations, and committed-use discounts only apply to GPUs you hold, and they apply to whatever model you schedule on them.

Quota is the real blocker. AWS "Running On-Demand P instances" and "G and VT" quotas default to 0 vCPUs (AWS docs), and GCP grants GPU quota per region and GPU model (Google Cloud docs). One pool means one quota request and one reservation amortized across every model.

This is where Qovery fits. Qovery deploys and operates GPU workloads inside your own AWS, GCP, Azure, or Scaleway account, or on the self-managed Kubernetes cluster most GPU teams already run. You get git-push deploys for serving containers, preview environments per pull request, per-environment RBAC, managed cluster upgrades, and auto-stop for non-production environments, all without a dedicated DevOps engineer. Your reservations and committed-use discounts stay in your name.

vLLM, Triton, Ray Serve, KServe, and NVIDIA Run:ai are complementary: they are the serving and scheduling layer, and Qovery is the deploy-and-operate layer around them (our vLLM vs TGI vs NIM comparison covers the serving side).

Hosted endpoints deserve a fair hearing. For low or spiky single-model volume, Amazon Bedrock, Google Vertex AI, Together AI, or Fireworks AI is simpler and usually cheaper: Together lists Llama 3 8B Instruct Lite at $0.14 per million tokens (Together AI), and Fireworks rents an H100 at $8.00 per hour (Fireworks). A self-hosted GPU costs the same at any load, so it wins once sustained utilization beats your per-token spend, and data residency and fine-tuned weights tip it further. Amazon SageMaker multi-model endpoints solve co-location inside AWS too, with GPU support through Triton on single-GPU instances (AWS docs).

How do I deploy this with Qovery from Claude Code or Cursor?

Install the Qovery Agent Skills once, then ask your coding agent in plain language to deploy each model service to your own cluster. The agent proposes the manifests and the deployment, and you approve every action before it runs.

Install globally for every compatible agent (Claude Code, Cursor, Codex, OpenCode, Gemini CLI):

curl -fsSL https://skill.qovery.com/install.sh | bash
# or scoped to the current project
curl -fsSL https://skill.qovery.com/install.sh | bash -s -- --project

Add the MCP Server to Claude Code (OAuth, read-only by default):

claude mcp add --transport http qovery https://mcp.qovery.com/mcp --callback-port 4242
# write access, which also needs to be enabled in the Qovery Console
claude mcp add --transport http qovery "https://mcp.qovery.com/mcp?read_write=true" --callback-port 4242

The skills (qovery-onboard, qovery-deploy, qovery-troubleshoot, qovery-optimize, qovery-speedup, qovery-preview, qovery-terraform) deploy; the MCP Server operates what exists. Try "deploy this vLLM service for Mixtral to my AWS cluster with Qovery", "why is this GPU pod stuck in Pending", or "auto-stop my staging inference environment at night". Setup details are in the AI agent quickstart and the Claude Code integration page.