Self-Hosting NVIDIA NIM for Agentic Inference: Small Fast Model or Big MoE (GLM-5.3 vs Qwen3.8-Flash-Next) on Your Own Kubernetes Cluster

For agentic inference on your own Kubernetes cluster, self-host Qwen3.8-Flash-Next before GLM-5.3: it activates 6B parameters per token, fits on a 4x H100 or 2x H200 slice, and keeps agent loops fast. Self-host GLM-5.3 only if your own eval proves you need it for long-horizon planning and you can keep a full 8-GPU H200 node busy around the clock. Otherwise, call GLM-5.3 through a hosted API.

Most teams pick the big model first. Then they discover the agent feels slow, and the 8-GPU reservation is permanent. I have watched this happen enough times that I now start every one of these conversations with the smallest model that passes the team's own tool-calling eval.

Model names in this space churn weekly, so here is what I verified on November 9, 2026. Both models exist on Hugging Face: GLM-5.3 (repo created August 25, 2026) and Qwen3.8-Flash-Next (August 24, 2026). Both are validated on NVIDIA's model-free NIM containers per the NVIDIA NIM support matrix. I could not find a dated NVIDIA announcement for November 4, so I am not repeating that date as fact. Check build.nvidia.com and the support matrix yourself before you build on any exact identifier.

Key Points:

• Self-host the smallest model that passes your own tool-calling eval. For agents, tool-call format reliability, context window and how many concurrent loops fit on a GPU decide the outcome, not the headline benchmark.
• Qwen3.8-Flash-Next is the "small fast" option in name only. It has 125B parameters with 6B active, plus a 51B n-gram embedding (Qwen model card). That makes it fast per token, but still a multi-GPU deployment. A truly small 8B dense model like Qwen3-8B fits one 24-48GB GPU.
• GLM-5.3 is roughly 753B parameters and needs an 8-GPU H200 or B200 node at the precisions NVIDIA validates. Below about 28B tokens a month, the hosted API at $1.40 / $4.40 per million tokens (Together AI pricing, checked November 9, 2026) costs less.
• Route instead of choosing. Run a small model as the default worker and escalate to the big one on schema-invalid tool calls, planner steps or context overflow.
• NVIDIA NIM deploys the same way on EKS, GKE, AKS, Scaleway Kapsule or on-prem, through the NIM Operator or the NIM LLM Helm chart, once the NVIDIA GPU Operator is installed.

Which model should you self-host for agentic inference: the small fast one or the big MoE?

Self-host the smallest model that passes your own tool-calling eval, and reserve a large MoE like GLM-5.3 for planning agents that need very long context and multi-step decomposition. Agent loops are dominated by short, repetitive tool calls, so time-to-first-token, output speed and JSON schema adherence shape the experience far more than reasoning depth.

Latency compounds inside a loop. The SWE-agent paper reports successful runs finishing with a median of 12 steps and unsuccessful ones averaging 21. Artificial Analysis measured GLM-4.6 at 34 output tokens/s and Qwen3-Next-80B-A3B at 115 tokens/s (both checked November 9, 2026).

For a step that emits 300 tokens, that gap is 8.8s versus 2.6s. Over 12 to 21 steps, the slower model adds 74 to 130 seconds to every task. Users feel that.

Format adherence is the failure mode that kills agents. The Berkeley Function Calling Leaderboard team found that chat models "tend to generate malformed function calls in which parameters can be extracted but not executable", plus type errors like a float passed where an int is required. Reliability also drops on repetition: tau-bench found gpt-4o succeeded on under 50% of tasks, with pass^8 under 25% in retail.

My five-minute checklist before picking a model:

• Median prompt length per step, including re-sent tool results.
• Model calls per completed task, measured from your own traces.
• Peak concurrent agent sessions, because concurrency fills GPU memory.
• Strictness of your tool schemas: nested objects and enums break small models first.
• Acceptable p95 latency per step, and whether any session exceeds 100k tokens.

The big MoE earns its GPUs on long-context planning, multi-file code edits, ambiguous task decomposition and sessions past 100k tokens. If none of those describe your agent, start smaller.

What is NVIDIA's model-free NIM catalog, and how is it different from a packaged NIM?

NVIDIA NIM is a set of inference microservices shipped as containers that expose an OpenAI-compatible API, deployable with an official Helm chart or the NIM Operator. A model-free NIM runs "any supported model in NIM LLM and VLM without a model-specific container image" (NVIDIA NIM docs), so adding a model becomes a config change.

The components, as of NIM 2.1.5 and NIM Operator v3.1.4 (checked November 9, 2026):

• Container image. NIM 2.x ships "one container, one backend": vLLM or SGLang (NIM overview). The current release bundles vLLM 0.31.0 and SGLang 0.5.21 (NIM release notes).
• Model source. Set NIM_MODEL_PATH to an hf://, ngc://, s3://, gs:// or local path, and "NIM then generates a manifest at startup" (model-free NIM docs).
• NIM Operator CRDs. NIMCache, NIMService, NIMPipeline and NIMBuild (NIM Operator docs). The NGC API key became optional in v3.1.3 (Operator release notes).
• Helm chart. It requires Kubernetes v1.26 or later and an NGC API key to pull the chart (NIM Helm docs).

For our two models, the support matrix lists GLM-5.3 on nvcr.io/nim/nvidia/sglang-model-free-nim:2.1.5, with an FP8 checkpoint on H200 and an NVFP4 checkpoint on B200. Qwen3.8-Flash-Next runs on a custom tag, vllm-model-free-nim:2.1.4-qwen3.8-flash-next. Only GLM-5.3-Flash is a hosted endpoint on build.nvidia.com; neither GLM-5.3 nor Qwen3.8-Flash-Next is.

OpenAI compatibility is the part I care about most. The OpenAI SDK, LangChain, LlamaIndex and the OpenAI Agents SDK all work by changing base_url, so a model swap never touches application code.

Licensing has two layers. On the NVIDIA side, the Certified Feature Branch "is available at no charge", while the Production Branch "requires an active NVIDIA AI Enterprise subscription" (NIM offerings). That subscription lists at $4,500 per GPU per year (NVIDIA AI Enterprise pricing). On the model side, GLM-5.3 ships under a custom "glm-5.3" license and Qwen3.8-Flash-Next under "qwen-community-1.0", not Apache 2.0. Have legal read both.

I covered the image names, YAML and troubleshooting in detail in our model-free NIM guide.

How do GLM-5.3 and Qwen3.8-Flash-Next compare for agent workloads?

GLM-5.3 is a roughly 753B-parameter MoE with a 1M-token context that needs a full 8-GPU H200 or B200 node. Qwen3.8-Flash-Next is a 125B MoE with only 6B active parameters and a 262k native context that fits on 4x H100 or 2x H200. A dense 8B model like Qwen3-8B is the real "small" option and runs on one 24-48GB GPU.

Dense versus MoE in one paragraph: total parameters decide VRAM for weights, active parameters decide compute per token. That is why an MoE model can be fast per token and still expensive to host. NVIDIA's heuristic for weight memory is total parameters x bytes per parameter / tensor-parallel size, with 1 byte for FP8 and 0.5 for NVFP4 (NIM memory troubleshooting).

The VRAM figures below are my estimates from that heuristic, not NVIDIA-published minimums. Specs come from each model's Hugging Face card and config.json, checked November 9, 2026.

KV cache: what actually fills the GPU

In agent workloads, concurrency fills the GPU before weights do. The per-token KV cache formula, from the PagedAttention paper, is:

KV bytes per token = 2 (K and V) x layers with KV cache x KV heads x head_dim x bytes per element

Worked out at BF16 from each config.json:

• Qwen3-8B: 2 x 36 layers x 8 KV heads x 128 x 2 bytes = 144 KiB per token. A 32k session needs about 4.8GB, and a 128k session about 19GB.
• Qwen3.8-Flash-Next: only 12 of 48 layers use full attention, with 2 KV heads of 256 dimensions, so 24 KiB per token. That is about 0.8GB at 32k and 3.2GB at 128k, plus a fixed-size state for the 36 linear-attention layers.
• GLM-5.3: uses MLA, so each layer caches a compressed latent of 512 + 64 values instead of full heads. Across 78 layers that is about 88 KiB per token, or roughly 2.9GB at 32k and 11.8GB at 128k.

On one L40S, Qwen3-8B at FP8 leaves roughly 30GB for KV cache after weights and runtime overhead. That is only about six concurrent 32k-token sessions. This is why hybrid-attention models like Qwen3.8-Flash-Next are so attractive for agents: their cache stays small as transcripts grow.

Quantization. Red Hat ran over half a million evaluations on quantized LLMs. On coding tasks, 8-bit models (FP8 and INT8) recovered 99.9% of baseline accuracy and 4-bit models 98.9%. Those were Llama 3.1 models on general and coding benchmarks, not function calling, so rerun your tool-call eval after quantizing.

Thinking modes. Hybrid reasoning modes add output tokens to every step. I found no published figure for how many tokens GLM-5.3 or Qwen3.8-Flash-Next add, so measure it. Inside a tight tool loop, I default to non-thinking mode and reserve thinking for planner steps.

What GPU and Kubernetes setup do you actually need to run NVIDIA NIM on your own cluster?

A small 8B-class model needs one 24-48GB GPU per replica. Qwen3.8-Flash-Next needs four 80GB GPUs or two 141GB GPUs with tensor parallelism. GLM-5.3 needs a full 8-GPU H200 or B200 node. Everything else (node pools, GPU Operator, weight caching, autoscaling, observability) is identical across model classes, and that is where teams lose weeks.

Prices below are on-demand Linux rates checked November 9, 2026.

The cluster checklist, the same for every model:

• GPU stack. The NIM Operator "requires NVIDIA GPU Operator version 24.3.0 or higher" (NIM Operator platform support). Request nvidia.com/gpu, and taint GPU nodes so CPU workloads never land there.
• Weight cache. Without a cache, "each NIM microservice instance downloads its own copy of the model when it starts" (NIM Operator caching docs). Back NIMCache with a PVC on network storage. At 753GB for GLM-5.3, this decides your cold start.
• Warm floor. Keep at least one replica running for anything user-facing. Pulling a 13GB image plus hundreds of GB of weights takes minutes, never seconds.
• Autoscaling signal. Scale on queue depth or concurrency with KEDA, never on CPU. An inference pod pegs the GPU while CPU looks idle. I wrote up the pattern in KEDA queue-depth autoscaling for inference pods.
• Observability. NIM serves the backend's native Prometheus metrics at /v1/metrics on port 8000, prefixed vllm: or sglang: (NIM observability docs). Alert on vllm:time_to_first_token_seconds, vllm:inter_token_latency_seconds, vllm:num_requests_waiting and vllm:kv_cache_usage_perc (vLLM metrics), plus DCGM_FI_DEV_FB_USED_BYTES from dcgm-exporter 4.9.0.

To be fair: one model, one GPU, one team on EKS or GKE is well served by the NIM Operator plus Helm, and needs nothing more.

Is self-hosting NVIDIA NIM cheaper than calling a hosted API for agent workloads?

Self-hosting is cheaper only past a utilization crossover, because a reserved GPU costs the same at 5% and 95% busy. For GLM-5.3 on an 8x H200 node, that crossover sits around 28 billion tokens a month, which is roughly 10,500 tokens per second sustained 24/7.

The formula I use, so you can rerun it when prices move:

break-even tokens per month (millions) = (GPU node $/hour x 730) / blended hosted $ per million tokens
blended $ per million = (input share x input price) + (output share x output price)
1B tokens per month ≈ 380 tokens per second, sustained 24/7

Agents are input-heavy because every step re-sends the transcript and tool results. Anthropic reports agents "typically use about 4x more tokens than chat interactions", and multi-agent systems about 15x (Anthropic engineering). The table below assumes a 10:1 input-to-output ratio. That ratio is my illustrative assumption, so plug in your own.

At 1B tokens a month, every hosted option wins by a wide margin. The break-even volumes also assume your GPUs can actually serve that many tokens, so validate throughput with your own traces before you trust them.

What moves the crossover:

• Commitments. AWS EC2 Instance Savings Plans go "up to 72%" (AWS Savings Plans), and these discounts stay in your own account.
• Spot. Spot capacity is up to 90% off on AWS and up to 91% off on Google Cloud (Google Spot VMs). It fits batch agents, not user-facing ones.
• Batching and FP8. Higher concurrency and FP8 raise tokens per GPU-hour without changing the bill.
• Hidden costs. Idle GPUs at night and on weekends, cross-AZ traffic, engineer time, model upgrade churn, and NVIDIA AI Enterprise at $4,500 per GPU per year if you need the Production Branch. That is $36,000 a year for one 8-GPU node.

Auto-stopping dev and staging GPU environments outside working hours removes most of the idle spend from non-production.

Cost is not the only reason to self-host. Data residency, keeping prompts and tool outputs inside your VPC, and pinning a model version a provider cannot deprecate are all valid reasons on their own.

How do you route between a small and a large model instead of picking one?

Run the small model as the default agent worker and escalate to the large model only when a step fails validation or is explicitly a planning step. Every NIM speaks the same OpenAI-compatible API, so the router is a thin service in front of two Kubernetes deployments rather than an application rewrite.

Escalation triggers that work in practice:

• Invalid tool call. The JSON does not parse, or it fails schema validation.
• Loop detected. The same tool is called twice with identical arguments.
• Planner step. The orchestrator marks the step as decomposition or planning.
• Context overflow. The transcript exceeds a threshold, for example 100k tokens.
• Retry budget spent. The small model already failed N times on this step.

The pseudo-logic, short enough to adapt:

def complete(step, messages, tools):
    if step.kind == "plan" or token_count(messages) > 100_000:
        return call("large", messages, tools)

    for attempt in range(2):
        reply = call("small", messages, tools)
        call_ok = reply.tool_call and validates(reply.tool_call, tools)
        if call_ok and not repeats_last_call(reply, messages):
            return reply
        messages = messages + [error_feedback(reply)]

    return call("large", messages, tools)  # escalate, and log the trace for your eval set

Keep both models behind one internal endpoint, for example http://llm-router.inference/v1. Application code should never hardcode a model name, so a model swap stays a deployment change.

Evaluate on your own task set. Capture 100-300 real agent traces, replay them against both models, and compare task success rate and cost per completed task. Per-token price and leaderboard rank, taken alone, will mislead you.

The research backs the direction. RouteLLM reported cost reductions of over 85% on MT Bench while achieving 95% of GPT-4's performance. FrugalGPT matched GPT-4 with up to 98% cost reduction. Your numbers will differ, but if the small model handles most steps, the large model's GPU pool shrinks to a fraction of a single-model deployment.

How do you run a NIM endpoint in your own cloud account without a dedicated DevOps engineer?

The hard part of production NVIDIA NIM is the platform around inference: GPU node pools, networking and TLS, secrets, per-team environments, cluster upgrades and turning idle GPUs off. Qovery runs that layer inside your own AWS, GCP, Azure or Scaleway account, or on your existing Kubernetes cluster, so the GPU bill, the committed-use discounts and the model weights stay inside your perimeter.

To be precise: Qovery does not serve models. NVIDIA NIM, vLLM or SGLang does that. Qovery is the operational layer that deploys and governs the NIM workload and the gateway, router and agent services around it.

What that gives a team running agents on GPUs:

• Your account, your discounts. Savings Plans and CUDs apply directly, and your compliance posture does not change.
• RBAC per environment, so only the right people can touch the GPU workloads.
• Auto-stop for non-production environments, which kills the overnight GPU bill in dev and staging.
• Managed cluster upgrades and git-push deployments for the router and agent services.
• A preview environment per pull request for the gateway, router and agent code in front of the model.
• Bring your own Kubernetes, which matters if you already bought GPU nodes on-prem.

Plain Helm plus the NIM Operator is fine for one model and one team. KServe, BentoML, Ray Serve and plain vLLM are solid serving layers. Bedrock, Vertex AI, Together AI and Fireworks AI are cheaper below the crossover. The platform argument starts when several teams each need their own environment, their own model version and their own GPU budget.

How do I deploy and operate this from Claude Code or Cursor?

Install the Qovery Agent Skills once, and your coding agent can scaffold, deploy and troubleshoot the services around your NIM endpoint in your own cloud account, asking you to approve each action. Add the Qovery MCP Server when you want the agent to inspect or operate what is already running.

Install globally (works with Claude Code, Cursor, Codex, OpenCode and Gemini CLI):

curl -fsSL https://skill.qovery.com/install.sh | bash

Or scope it to one project:

curl -fsSL https://skill.qovery.com/install.sh | bash -s -- --project

Add the MCP Server, which uses OAuth and is read-only by default:

claude mcp add --transport http qovery https://mcp.qovery.com/mcp --callback-port 4242

Enable write access only when you want it. You also need to enable write access in the Qovery Console:

claude mcp add --transport http qovery "https://mcp.qovery.com/mcp?read_write=true" --callback-port 4242

The skills are qovery-onboard, qovery-deploy, qovery-troubleshoot, qovery-optimize, qovery-speedup, qovery-preview and qovery-terraform. The MCP Server operates what already exists. Then ask in plain language: "deploy this inference gateway to my GCP account with Qovery and give it a preview environment per pull request."

Setup details are in the AI agent quickstart and the Claude Code integration page.