Self-hosting an open-weight model like Sarvam AI's Prism on Kubernetes: what it actually takes

Key Points:

• Five components make up the job: GPU node capacity managed by the NVIDIA GPU Operator, an inference server, a weight-loading strategy, autoscaling on queue depth, and immutable per-decision logging.
• Apache 2.0 is the easy part. It allows commercial use, modification and redistribution, includes a patent grant and has no user cap. The work is in the serving stack and the GPU hours under it.
• Prism is a large model. Sarvam's model card puts it at 105B parameters in FP8. It needs one GPU with about 120 GB of free memory (an H200 or B200) and it ships its own serving code, because vLLM and transformers cannot load it.
• For decision scoring, self-host for residency and audit replay, not for cost. On AWS, the smallest on-demand instance with an H200 is an 8-GPU p5en.48xlarge at about $46,206 a month. Against Sarvam's hosted Prism price, that only pays off at hundreds of requests per second, sustained.
• Smaller open-weight scoring models change the maths. A single L4 node (AWS g6.xlarge or Scaleway L4-1-24G) costs about $575-$590 a month and breaks even against a cheap hosted endpoint at roughly 1.2 requests per second, sustained.
• Qovery deploys and operates this stack inside your own AWS, GCP, Azure or Scaleway account, or on a Kubernetes cluster you already run, so the weights, GPU nodes, logs and cloud discounts stay yours.

Apache 2.0 is the easy part. The GPU node group is where the project actually starts.

Sarvam AI released Prism on November 3, 2026 with open weights on Hugging Face. It is a decision model: you give it a context, a question and a fixed set of options, and it returns a calibrated probability for each option in one forward pass. To self-host it on Kubernetes you need five things:

1. A GPU node with enough memory, managed by the NVIDIA GPU Operator.
2. A serving container.
3. A way to get roughly 106 GB of weights onto that node quickly.
4. Autoscaling on queue depth.
5. A tamper-proof log of every decision.

The licence takes five minutes. The rest takes the quarter.

I'll use Prism as the concrete example. The checklist is for the whole class of open-weight scoring models, because next month there will be another one.

What is Sarvam AI's Prism, and why would you self-host it instead of calling an API?

Prism is a 105B-parameter decision model from Sarvam AI, released under Apache 2.0. Teams self-host a model like it for one main reason: the input is often regulated data (KYC files, credit histories, support transcripts, agent traces) that cannot leave their own network.

What the primary sources actually say:

• What it is: a LoRA fine-tune of Sarvam 105B, a mixture-of-experts model with 128 routed experts per layer. The adapter is merged into the weights for serving (Sarvam AI).
• What it returns: probabilities over options you define (choice, yes_no or score questions), never free text. The answer is always one of your options (Hugging Face model card).
• Hardware: "One NVIDIA GPU with FP8 support and about 120 GB of free memory, such as a B200 or H200", plus about 106 GB of disk for the weights (model card).
• Serving: the weights cannot be loaded with AutoModelForCausalLM or served by general-purpose inference servers. You run Sarvam's bundled prism.py (model card).
• Use cases Sarvam tested: prompt-injection guardrails, trace evaluation, support intent routing, model routing and document intelligence. Risk scoring and flagging runs for human review fit the same pattern. Sarvam says it opened the weights so enterprises "can train this calibrated base on their own data and run it on their own infrastructure" (Sarvam AI).

Three reasons to self-host hold up:

• Data residency. GDPR Chapter V only allows transfers outside the EU under specific conditions (GDPR Art. 44). Section 16 of India's DPDP Act 2023 lets the government restrict transfers to notified countries, and sector rules such as RBI's take priority where they are stricter (MeitY).
• Audit replay. You need to re-run a decision from two years ago on exactly the same weights. That is only possible if you hold the weights.
• Vendor continuity. No one can deprecate or retune an Apache 2.0 checkpoint stored in your own bucket.

One common reason is usually wrong: "it will be cheaper." The section on cost does the maths.

On licensing, Apache 2.0 grants commercial use, modification, redistribution and a patent licence (Apache Software Foundation). Meta's Llama licences, by contrast, require a separate licence from Meta for companies above 700 million monthly active users (Llama 3.1 Community License). Gemma ships under its own Gemma Terms of Use with a prohibited-use policy. For a bank's legal team, Apache 2.0 is the shortest conversation they will have this year.

What do you actually need on the cluster to serve a model like Prism?

You need five components, in this order: GPU node capacity, an inference server, a weight-loading strategy, a request and queue layer, and inference-specific observability. Everything else is optimisation.

1. GPU node capacity.

• Create a dedicated GPU node group on EKS, GKE, AKS or Scaleway Kapsule.
• Install the NVIDIA GPU Operator. It manages the driver, the Kubernetes device plugin, the container toolkit, GPU Feature Discovery and DCGM-based monitoring.
• nvidia.com/gpu is an extended resource that only exists once the device plugin is running. You request it in limits, in whole GPUs, and it cannot be overcommitted (Kubernetes docs).
• Taint the GPU nodes (for example nvidia.com/gpu=present:NoSchedule) so a stray cron job never lands on a $60/hour machine.

2. Inference server. Pick by model type:

The original vLLM benchmarks showed up to 24x the throughput of Hugging Face Transformers and up to 3.5x that of TGI (vLLM). For generative workloads, that is the default I'd pick.

Prism is in the last row. It ships prism.py with its own Triton (the GPU kernel language) FP8 kernels, so you wrap Prism.from_pretrained() in a thin HTTP server and stop there. For a scoring model, that is the right amount of engineering.

3. Weight loading. You have three options:

• Bake the weights into the image: instant start, but a 100+ GB image that your registry and nodes will not thank you for.
• Init container pulling from object storage (S3, GCS, Azure Blob or Scaleway Object Storage): simple, and you pay the download on every cold start.
• ReadOnlyMany volume or a node-local cache shared across replicas: the fastest restart, with more moving parts.

Do the cold-start arithmetic. At 1 GB/s of effective throughput (measure yours), 106 GB takes about 106 seconds to download. Add about two minutes to load on a B200 and a few minutes of optional CUDA-graph capture (model card). That is a five-minute pod start before node provisioning. Lazy-loading tools such as GKE image streaming and AWS SOCI help with container layers, but they do not remove the weight download.

4. Request path. Put a Service and an Ingress in front, with authentication at the edge and generous timeouts. Bursty synchronous scoring traffic needs a queue in front: a replica that takes five minutes to start will never absorb a 30-second spike.

5. Observability.

• Generative models: tokens per second and time-to-first-token.
• Every model: p95 latency, plus GPU utilisation and memory from the DCGM exporter.
• Compliance: per-decision logs, covered below.

Sarvam reports a 44 ms median and 420 ms p95 on one B200 (model card), so alert on p95, not on the average.

How much GPU do you need, and what does it cost per month?

A small open-weight scoring model fits on one mid-range GPU for about $575-$735 a month. Prism needs one H200- or B200-class GPU, and on the big clouds those mostly come eight to a machine. A hosted API bills per request, while a GPU bills every hour whether or not a request arrives.

On-demand prices, converted to a 730-hour month:

Prices were checked on November 8, 2026. An 80 GB H100 does not meet Prism's ~120 GB requirement, so the single-H100 row is for smaller models only. GCP (G2 for L4; A3 Ultra and A4 for H200 and B200) and Azure (NVads A10 v5, ND H200 v5) follow the same shape. Check their pricing pages for your region.

Discounts are the lever that moves the maths:

• Spot: AWS Spot is up to 90% off on-demand (AWS) and GCP Spot VMs, GPUs included, are up to 91% off (Google Cloud). Interruptions make it a fit for staging and batch scoring, not for synchronous production.
• Commitments: Savings Plans cut up to 66% (Compute) or 72% (EC2 Instance) (AWS). These discounts only reach you if the cluster runs in your own cloud account.

Break-even, shown:

• Prism, self-hosted vs Sarvam's API. Sarvam charges ₹4 per million input tokens for Prism, with no output tokens (Sarvam pricing). At an assumed ₹88 per dollar (use today's rate), a p5en.48xlarge costs $46,206 x 88 = ₹4,066,135 a month. Divided by ₹4 per million tokens, that is about 1,016,534 million tokens, or roughly 1 trillion input tokens. At 1,000 tokens per decision, that is about 1.02 billion decisions a month, or ~387 requests per second around the clock. Below that, the API is cheaper.
• A small model on an L4 vs a cheap hosted endpoint. Together AI lists Qwen3.5 9B at $0.17 input and $0.25 output per million tokens (Together AI). A request with 1,000 tokens in and 50 out costs $0.00017 + $0.0000125 = $0.0001825. A g6.xlarge at $587.50 a month breaks even at 587.50 / 0.0001825 ≈ 3.2 million requests a month, or ~1.2 requests per second, sustained.

Self-hosting the 105B model is a compliance decision. Self-hosting a 9B model can also be a cost decision.

Scale non-production to zero. A 40-hour working week leaves 128 of 168 weekly hours idle, which is 76% of a staging GPU's bill spent on nobody. Dev and staging inference should never hold a GPU overnight.

Quantization. Methods like GPTQ and AWQ can move a small model to 4-bit or 8-bit weights and onto a cheaper GPU, sometimes onto a CPU. Prism already ships in FP8. Whatever you quantize, benchmark calibration on your own labelled data before you let it score a regulated decision.

What does a minimal production deployment look like, step by step?

A first production deployment takes six steps:

1. Provision a GPU node pool (EKS managed node group, GKE node pool, AKS node pool or Kapsule pool) with the instance type from the sizing table, a nvidia.com/gpu=present:NoSchedule taint and a label.
2. Install the NVIDIA GPU Operator with Helm. Check that its DaemonSets (driver, device plugin, DCGM exporter) are healthy and that the node advertises nvidia.com/gpu.
3. Stage the weights in object storage in the same region. Mount them through a PVC, or pull them in an init container.
4. Deploy the serving container as a Deployment with a nvidia.com/gpu limit, a toleration and a readiness probe that waits for the weights.
5. Expose it through a Service and an Ingress with auth, TLS and long timeouts.
6. Add autoscaling and per-decision logging.

Your serving image and its /ready endpoint are yours to write. Nothing below is a Prism-specific flag.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: scoring-model
spec:
  replicas: 1
  selector:
    matchLabels: { app: scoring-model }
  template:
    metadata:
      labels: { app: scoring-model }
    spec:
      nodeSelector:
        pool: gpu-inference
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      containers:
        - name: server
          image: registry.example.com/scoring-model@sha256:<digest>
          env:
            - name: MODEL_REVISION   # Hugging Face commit hash of the weights
              value: "<weights-commit-sha>"
          ports:
            - containerPort: 8080
          resources:
            limits:
              nvidia.com/gpu: 1
          readinessProbe:            # passes only once weights are in GPU memory
            httpGet: { path: /ready, port: 8080 }
            periodSeconds: 10
            failureThreshold: 3
          startupProbe:              # allows ~10 min for a cold weight load
            httpGet: { path: /ready, port: 8080 }
            periodSeconds: 10
            failureThreshold: 60

Autoscaling. Do not use an HPA on CPU. The GPU saturates while the CPU sits at 10%, and the HPA never scales. Scale on in-flight requests or queue depth instead, with a KEDA Prometheus trigger whose query returns queue length and whose threshold matches what one replica can handle. Set activationThreshold and minReplicaCount: 0 on non-production environments, and production can scale to zero overnight too if latency allows. Budget for the lag: a new GPU node plus a 106 GB weight load is minutes, not seconds, so keep one warm replica in production.

Rollouts and audit. Pin two identifiers together: the container image digest and the weights' commit hash. Log both with every decision. When you move to a new model revision, canary it on a slice of traffic, compare score distributions, then promote it.

How do you keep a self-hosted scoring model auditable and compliant?

For any past decision, you must be able to prove which model revision scored it, on what input, who asked, and who could see the result. Deployment is half the job.

• Immutable per-decision log. Record the input hash, weights commit, container digest, the output probabilities, a timestamp and the caller's identity. Write it to object storage outside the inference cluster, with versioning and object lock turned on.
• Network isolation. Run inference pods in private subnets with no public egress. Reach object storage through private endpoints, and use NetworkPolicies so only named callers can reach the scoring Service.
• Access control. Use per-environment RBAC: a data scientist can redeploy staging but cannot touch the production endpoint or read production decision logs.
• Residency. Pin weights, logs and GPU nodes to one region. In your own cloud account that is a setting. On a vendor's multi-region endpoint it becomes a contract negotiation, which is where GDPR Chapter V and DPDP Section 16 start to matter.
• Human oversight. The EU AI Act classifies systems used "to evaluate the creditworthiness of natural persons or establish their credit score" as high-risk (Annex III, 5(b)). Article 14 requires that people can oversee such a system, override it and stop it. Log the human override next to the model score, and treat the calibrated probability as input to a human decision.

When should you just use a hosted inference API instead?

Use a hosted API when your volume is low, your data is not regulated, or nobody is on call for a GPU node at 3am. Pretending otherwise is how teams burn a quarter on a platform project.

• Low or spiky volume and no residency rule: pay per token. For Prism, the break-even above is hundreds of requests per second.
• No Kubernetes practice in-house: the model is the easy part. The cluster upgrades, GPU driver versions and node autoscaling are not.
• The hybrid that works: prototype against a hosted, OpenAI-compatible endpoint and keep that interface. When volume or compliance justifies it, point the base URL at your in-cluster vLLM, with no application rewrite. For a custom model like Prism, keep a thin client interface of your own so swapping the backend is a config change.

Fair alternatives, each with a real sweet spot:

• Sarvam AI's hosted API: Prism with no GPU, at ₹4 per million input tokens, and the natural first step.
• Together AI: cheap serverless access to many open models.
• Baseten: dedicated deployments of custom models with production support.
• Modal: Python-first serverless GPUs that start fast for bursty jobs.
• Replicate: the quickest path from a model repo to an HTTP endpoint.
• Hugging Face Inference Endpoints: one-click dedicated endpoints for models on the Hub.
• Amazon SageMaker and Google Vertex AI: managed endpoints inside the cloud you already use, with that cloud's IAM and compliance tooling.

How does Qovery fit if you don't want to hand-build the platform around the model?

Qovery deploys and operates this stack inside your own AWS, GCP, Azure or Scaleway account, or on a Kubernetes cluster you already run. You get self-service deployments and managed cluster upgrades without hiring a platform team, and the weights, GPU nodes and logs never leave infrastructure you own.

• BYOC: the cluster, GPU node groups, weight buckets and cloud bill stay in your account, so your Savings Plans and committed-use discounts keep applying.
• Bring your own Kubernetes: if the GPU cluster already exists, on-premises or with any provider, Qovery runs on it rather than forcing a new one.
• Useful for this workload:
  • Git-push deploys of the serving container.
  • A preview environment per pull request, to test a new weights revision in isolation.
  • Environment auto-stop, so staging GPUs don't burn money overnight.
  • Per-environment RBAC on the production scoring endpoint.
  • Managed cluster upgrades.
• The boundary: Qovery is not an inference server. It does not serve, quantize or fine-tune models. You still choose vLLM, Triton or your own prism.py wrapper, the instance type and the batching strategy.

How do I deploy this with Qovery from Claude Code or Cursor?

Install the Qovery Agent Skills once, then ask your coding agent in plain language to deploy the serving container to your own cloud account. The skills cover onboarding, deploying, troubleshooting and preview environments. The MCP Server operates what already exists and is read-only by default.

• Install for every compatible agent (Claude Code, Cursor, Codex, OpenCode, Gemini CLI): curl -fsSL https://skill.qovery.com/install.sh | bash
• Project-scoped instead of global: curl -fsSL https://skill.qovery.com/install.sh | bash -s -- --project
• Add the MCP Server to Claude Code (OAuth, read-only by default): claude mcp add --transport http qovery https://mcp.qovery.com/mcp --callback-port 4242
• Write access, only when you want it (it also has to be enabled in the Qovery Console): claude mcp add --transport http qovery "https://mcp.qovery.com/mcp?read_write=true" --callback-port 4242
• Available skills: qovery-onboard, qovery-deploy, qovery-troubleshoot, qovery-optimize, qovery-speedup, qovery-preview, qovery-terraform. Then ask, for example: "deploy this vLLM service to my AWS account with Qovery."

Docs: AI agent quickstart and Claude Code integration.