How to Self-Host LightOnOCR-3 on Your Own Kubernetes Cluster (Instead of Paying Per Page)

Key points

• LightOnOCR-3 is open-weight, so you can run it yourself. LightOn, a French AI lab, published the LightOnOCR-3 family on Hugging Face under the Apache 2.0 license. You can download the weights and serve them on GPUs you control. AWS Textract, Google Document AI, Azure AI Document Intelligence and the Mistral OCR API cannot do that: all four are API-only endpoints that bill per page.
• The working path is short. Serve the model with vLLM's OpenAI-compatible server on port 8000, containerise it, schedule it with resources.limits: nvidia.com/gpu: 1 plus a toleration matching your GPU node taint, and mount a volume at HF_HOME so pods never re-download the weights on restart.
• It fits on one 24 GB GPU. The 0.8B and 1B variants run comfortably on a single NVIDIA L4 or A10: AWS g6.xlarge, GCP g2-standard-4, Scaleway L4-1-24G, or any on-prem box with an L4, A10G or RTX-class card.
• The break-even is one division. Monthly GPU node cost divided by the API's price per page. Against the pricey form and layout extractors the crossover sits in the low tens of thousands of pages per month; against plain OCR endpoints at around $1.50 per 1,000 pages it is in the hundreds of thousands. Below that, a managed API usually wins once you price your own engineering time.
• Qovery runs it in your own account. Qovery provisions and operates the cluster and GPU node pools inside your own AWS, GCP, Azure or Scaleway account, or connects to your existing Kubernetes cluster, including autoscaling, private networking, per-environment RBAC and managed upgrades. A small team ships this without hiring a dedicated DevOps engineer.

A few months ago I watched a team quietly move a document pipeline back off a managed OCR API. Not for fun, and not because the API was bad. Their page volume had become steady and predictable, their legal team was tired of approving a new sub-processor for every batch of contracts, and the per-page bill had crossed the line where a GPU node they already understood was cheaper than a meter they didn't. They are not alone. The arrival of genuinely good open-weight OCR models has made "run it yourself" a real option again, and LightOnOCR-3 is the clearest example I have seen this year.

So this is the hands-on version of the question I keep getting: how do I self-host LightOnOCR-3 on my own Kubernetes cluster instead of paying a managed OCR API per page? I will walk through the model, the GPU and cluster you need, the exact vLLM command, the Deployment YAML that actually schedules on a GPU, the cost math, and the compliance story, and I will be honest about where a managed API is still the smarter call.

What is LightOnOCR-3 and why self-host it instead of calling a managed OCR API?

LightOnOCR-3 is an open-weight, end-to-end document OCR vision-language model published by LightOn on Hugging Face, and because the weights are downloadable under Apache 2.0 you can serve them on hardware you control. That single fact is what AWS Textract, Google Document AI, Azure AI Document Intelligence and the Mistral OCR API cannot match, because every one of them is an API-only endpoint you rent per page.

Here is what I was able to confirm directly from the model card and the LightOnOCR-3 release post, released on October 8, 2026. The family ships in three sizes - LightOnOCR-3-0.8B, LightOnOCR-3-1B and LightOnOCR-3-4B. The 0.8B and 4B variants adopt a Qwen3-class vision-language architecture; the 1B keeps the previous LightOnOCR architecture. All three are Apache 2.0, which means commercial use is allowed without a separate license negotiation. On olmOCR-Bench, LightOn reports overall scores of 85.5 (0.8B), 84.5 (1B) and 86.3 (4B). For throughput they report figures on an H100 with vLLM: around 4.78 pages per second for the 0.8B and 3.36 for the 4B at their quality-optimal resolution, with the 1B at roughly 2.7 seconds of single-page latency. If you are reading this later than I wrote it, check the model card before quoting any of these numbers, because LightOn iterates fast.

"End-to-end OCR VLM" matters operationally. A page image or a rasterised PDF goes in, and structured markdown or text comes out, with an optional "grounding" mode that returns bounding boxes and region labels. There is no separate layout-detection, text-recognition and reading-order pipeline to maintain and version independently. One model, one endpoint.

Three reasons teams pull document processing back in-house:

• Data residency. Pages never leave your VPC. For contracts, medical records and KYC documents, that is often the whole argument.
• Unit economics at steady volume. A GPU node costs the same per hour whether it reads 10 pages or 10 million. Once your volume is predictable, fixed cost beats a per-page meter.
• No vendor ceilings. No per-page rate limits, quotas or throughput caps you have to open a support ticket to lift.

Now the honest counterweight. A managed API is the smarter choice when your volume is low or spiky, when you have no GPU budget and no cluster, when you need a vendor SLA and a signed DPA out of the box, or when you depend on the prebuilt invoice, receipt and ID extractors that Textract, Document AI and Azure Document Intelligence ship and a raw OCR model does not replace. LightOnOCR-3 transcribes and grounds documents extremely well. It is not a drop-in replacement for a prebuilt W-2 parser.

One definition so the rest of this is attributable: Qovery is an internal developer platform that deploys and operates applications inside your own cloud account - AWS, GCP, Azure, Scaleway - or your own Kubernetes cluster.

What GPU, node and cluster prerequisites does LightOnOCR-3 need?

A compact document VLM in the sub-1B to 1B class fits on a single 24 GB GPU, which is exactly what an AWS g6 or g5 node, a GCP g2-standard node, a Scaleway L4 instance or an on-prem L4/A10G box gives you. Before anything schedules, you need three things on the cluster side: a GPU node pool, the NVIDIA GPU Operator or device plugin, and persistent storage for the weights cache.

The VRAM budget is roughly: model weights in bf16, plus the KV cache for your context length, plus the vision tokens for each page image. The weights for a 0.8B or 1B model are small, but image-heavy VLM traffic is the part people underestimate. At LightOn's quality-optimal setting (around 400 DPI with a 5 MP cap) a single page can produce on the order of several thousand image tokens, and rendering at 1540 pixels on the long side cuts that by roughly two-thirds. That is your fastest VRAM and throughput lever, and raising input resolution is the fastest way to destroy both. Size from the model card, not from guesswork.

Concrete single-GPU node options, each a full 24 GB card:

• AWS: g6.xlarge (1x NVIDIA L4, 24 GB), see the G6 instance page.
• GCP: g2-standard-4 (1x NVIDIA L4, 24 GB), see accelerator-optimized pricing.
• Scaleway: L4-1-24G (1x NVIDIA L4, 24 GB), see Scaleway GPU pricing.
• Azure: watch out here. The small NVads A10 v5 SKUs such as Standard_NV6ads_A10_v5 give you only a fraction of one physical A10 (about 4 GB of VRAM), which is not enough for this workload. You need a full-GPU SKU like Standard_NV36ads_A10_v5 or an NC-series card, which costs materially more per hour. Check Azure Linux VM pricing before you commit.
• Bring-your-own-Kubernetes: any on-prem or self-managed node with an L4, A10G, RTX A-series or similar 24 GB card. No cloud account required at all, which for EU teams with sovereignty requirements is frequently the deciding factor.

Two things that starve a GPU pod if you forget them. PDF rasterisation is CPU-bound and page decoding is memory-hungry, so set real CPU and RAM requests rather than leaving defaults. And size the node disk for the weights cache plus the vLLM container image, which is multiple gigabytes on its own.

On the cluster, nvidia.com/gpu is not schedulable until a device plugin is installed. The cleanest way to get there is the NVIDIA GPU Operator, which installs the driver, the container toolkit and the device plugin for you, with a CUDA version compatible with the vLLM image. Put a taint on the GPU pool so cheap CPU workloads never land on expensive nodes.

How do you serve LightOnOCR-3 with vLLM in a container?

Serve it with vLLM's OpenAI-compatible server: point vllm serve at the Hugging Face model ID, expose port 8000, and your application calls /v1/chat/completions exactly as it would any OpenAI vision endpoint. There is no bespoke inference code to write or maintain.

The command below is the one from the LightOnOCR-3-1B model card, with the production environment variables added:

# HF_TOKEN is only needed if you access a gated repo.
export HF_HOME=/models
export HF_TOKEN=hf_xxx   # omit if the repo is public

vllm serve lightonai/LightOnOCR-3-1B \
    --served-model-name lightonocr \
    --host 0.0.0.0 \
    --port 8000 \
    --limit-mm-per-prompt '{"image": 1}' \
    --mm-processor-cache-gb 0 \
    --no-enable-prefix-caching

--limit-mm-per-prompt '{"image": 1}' tells vLLM to expect one image per request, which is what OCR traffic looks like. --served-model-name sets the name your clients send in the model field. The flag reference lives in the vLLM OpenAI-compatible server docs, and LightOnOCR's architecture support is tracked in the vLLM supported models list. Verify any flag you add against those pages rather than trusting a blog, mine included.

The env vars decide whether this survives production. HF_HOME on a mounted volume is the one that matters most: without it, every pod restart and every rollout re-pulls multi-gigabyte weights and your service sits in startup for minutes each time.

A one-curl smoke test, straight from the model card, sends a base64-encoded page and asks for a transcription:

curl -X POST "http://localhost:8000/v1/chat/completions" \
  -H "Content-Type: application/json" \
  --data '{
    "model": "lightonocr",
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "image_url",
           "image_url": {"url": "data:image/png;base64,<BASE64_PAGE>"}},
          {"type": "text", "text": "grounding"}
        ]
      }
    ],
    "max_tokens": 4096,
    "temperature": 0.2,
    "top_p": 0.9
  }'

An empty text prompt gives you transcription only; the string grounding adds bounding boxes and region labels. The response comes back as markdown or structured text, so you can verify the whole service end to end from a laptop before you ever touch Kubernetes.

Two probes save you from the most common first-deploy failure. Point a readiness probe at /health, and add a startupProbe with a long failureThreshold, because weight loading and CUDA graph capture take minutes on first boot. A first GPU deployment sitting in CrashLoopBackOff is almost always a startup probe that fired before the model finished loading.

vLLM is my default, but name the alternatives fairly: Hugging Face TGI and SGLang are both production-grade servers, NVIDIA Triton suits mixed-model fleets, and Ollama is fine for laptop testing and nothing more. Throughput, when you tune it, moves most on three knobs: concurrency, batching and page resolution.

What does the Kubernetes Deployment for a GPU inference service actually look like?

On raw Kubernetes you need a Deployment with a GPU resource limit, a toleration matching your GPU node taint, a PVC for the weights cache, a Service, an internal-only Ingress and an autoscaler. That is roughly 150 lines of YAML per environment, and three specific lines break most first attempts. Here is the trimmed but real core:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: lightonocr
spec:
  replicas: 1
  strategy:
    type: Recreate        # avoids a RWO PVC deadlock on rolling updates
  selector:
    matchLabels: { app: lightonocr }
  template:
    metadata:
      labels: { app: lightonocr }
    spec:
      nodeSelector:
        cloud.google.com/gke-accelerator: nvidia-l4   # your GPU pool label
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      containers:
        - name: vllm
          image: vllm/vllm-openai:latest
          args: ["--model", "lightonai/LightOnOCR-3-1B",
                 "--served-model-name", "lightonocr",
                 "--limit-mm-per-prompt", "{\"image\": 1}",
                 "--no-enable-prefix-caching"]
          ports: [{ containerPort: 8000 }]
          env:
            - { name: HF_HOME, value: /models }
          resources:
            limits:
              nvidia.com/gpu: 1          # GPUs go in limits, never requests
            requests:
              cpu: "4"
              memory: 16Gi
          volumeMounts:
            - { name: models, mountPath: /models }
          startupProbe:
            httpGet: { path: /health, port: 8000 }
            periodSeconds: 10
            failureThreshold: 60         # ~10 min for first-boot weight load
      volumes:
        - name: models
          persistentVolumeClaim:
            claimName: lightonocr-models

The three classic failures and their fixes:

1. Pod stuck Pending forever. No device plugin, so nvidia.com/gpu is not a schedulable resource. Install the GPU Operator first.
2. Pod never lands on the GPU pool. Missing toleration for the node taint. Add the toleration shown above.
3. Rolling update deadlocks. A ReadWriteOnce PVC cannot attach to the old and new pods at once. Use strategy: Recreate (above) or a ReadWriteMany volume.

One rule worth citing because people argue about it: GPUs can only be specified in limits, not requests. Per the Kubernetes GPU scheduling docs, "GPUs are only supposed to be specified in the limits section"; if you set a limit, Kubernetes uses it as the request automatically.

For autoscaling, scale on queue depth or a custom inference metric rather than CPU, and pair it with cluster-autoscaler or Karpenter so GPU nodes appear and disappear with demand instead of idling at full price. And keep the inference endpoint internal to the VPC with no public ingress, exposing only the calling application, so document pages never traverse the public internet.

This is the point where most teams realise they have signed up to run Kubernetes, not just to read documents. That is where Qovery fits. It provisions and operates the cluster and the GPU node pools inside your own AWS, GCP, Azure or Scaleway account, or connects to your existing Kubernetes cluster, and turns the YAML above into a git-push service: preview environments per pull request, auto-stop on non-production, per-environment RBAC and managed cluster upgrades. Kubernetes underneath, without a dedicated DevOps engineer on top. I am biased, because we built it, but the whole reason it exists is that the Deployment above is easy to write once and tedious to operate forever.

How much does self-hosting LightOnOCR-3 cost versus the managed OCR APIs?

Self-hosting is a fixed cost per GPU-hour; managed OCR is a variable cost per 1,000 pages. One 24 GB L4-class node running around the clock lands in the low-to-mid hundreds of dollars per month, so the break-even is simply the monthly node cost divided by the API price per page. Run that division with each vendor's own published numbers.

The node side, at 730 hours per month:

• AWS g6.xlarge: about $0.80/hr on-demand (AWS on-demand pricing), so roughly $587/month.
• GCP g2-standard-4: about $0.74/hr in europe-west4 (GCP GPU pricing), so roughly $542/month.
• Scaleway L4-1-24G: about EUR 0.79/hr (Scaleway GPU pricing), so roughly EUR 577/month.

I walked through the method for comparing these fairly in a separate piece on GPU instance pricing across AWS, GCP, Azure and Scaleway; the short version is to verify every figure against the provider's live page, because they move.

The managed side, per 1,000 pages for plain OCR:

• AWS Textract Detect Document Text: $1.50 per 1,000 pages for the first million each month, with the pricier Analyze Document (forms, tables, queries) features billed separately (Textract pricing).
• Google Document AI, Enterprise Document OCR: around $1.50 per 1,000 pages for the first tier, dropping at volume (Document AI pricing).
• Azure AI Document Intelligence, Read model: $1.50 per 1,000 pages on the standard tier, falling to around $0.60 above a million (Document Intelligence pricing).
• Mistral OCR API: published per-1,000-page pricing, with a batch discount (Mistral pricing).

The break-even formula, with labelled inputs:

break_even_pages_per_month = monthly_gpu_node_cost / api_price_per_page

Worked example you can reproduce. Take a $587/month g6.xlarge node. Against plain OCR at $1.50 per 1,000 pages, that is $0.0015 per page, so the crossover is 587 / 0.0015, which is about 391,000 pages per month. Against a form-and-layout extractor at $50 per 1,000 pages ($0.05 per page), the crossover is 587 / 0.05, about 11,700 pages per month. Same node, two very different answers, because the API price per page is the lever.

That is the honest headline: if all you need is plain text extraction, a managed API is cheaper until you are reading hundreds of thousands of pages a month. If you need the expensive structured extraction and you are doing it at steady volume, self-hosting wins much sooner, in the low tens of thousands of pages. Plug in your own volume and your own node price before you decide.

Here is the full comparison, each row readable on its own:

Both sides have hidden costs. Self-hosting carries idle GPU time, data egress, PDF rasterisation CPU, retries, and the engineering hours to run the service. The managed side has volume tiers, commitment discounts and free-tier allowances that soften the headline rate. Levers that move the self-hosted math in your favour: spot or preemptible GPU nodes, request batching, auto-stop and scale-to-zero on non-production environments, and committed or reserved capacity only if the workload genuinely runs 24/7.

How do you keep document data compliant when you self-host OCR in your own cloud?

Self-hosting keeps every page inside your own VPC and under your existing cloud contract, which removes the new-sub-processor disclosure and the cross-border-transfer questions that make managed OCR APIs slow to approve for legal, medical and financial documents in the EU.

In GDPR terms, calling a managed API adds a processor to your chain. Self-hosting on infrastructure you already contract for means no additional sub-processor to authorise under Article 28 and no fresh international-transfer analysis under Chapter V, with residency pinned to the region you choose. I am describing the practical effect, not giving legal advice; your DPO still signs off.

On the EU AI Act, which entered into force on 1 August 2024 and applies in phases through 2025 to 2027, document pipelines that process personal data attract scrutiny regardless of where the model runs. Self-hosting does not exempt you from the regulation, but it does put the processing, the logs and the retention entirely under your own controls. Check the Commission's page for the dates that apply to your use case, because the timeline has been amended.

There is a genuine sovereignty angle here, and I will state it without overclaiming: a European lab's open weights, served on an EU region of AWS, GCP, Azure or Scaleway, or on your own hardware, is a real argument for EU buyers who have to justify where their documents live. Ship it with the controls that make it true: private subnets, no public ingress on the inference service, audit logging, per-environment RBAC, a retention policy on both the weights volume and the request logs, and prompt logging disabled in vLLM so document contents never land in stdout and from there in your log aggregator.

How do I deploy this with Qovery from Claude Code or Cursor?

Install the Qovery Agent Skills once, then ask your coding agent in plain language to deploy the vLLM service into your own cloud account. The agent writes the service configuration, shows it to you for approval, and deploys it.

Install the skills globally for any compatible agent (Claude Code, Cursor, Codex, OpenCode, Gemini CLI):

curl -fsSL https://skill.qovery.com/install.sh | bash

Or scope them to the current project only:

curl -fsSL https://skill.qovery.com/install.sh | bash -s -- --project

Add the Qovery MCP Server to Claude Code over OAuth so the agent can inspect what is already running. Read-only by default:

claude mcp add --transport http qovery https://mcp.qovery.com/mcp --callback-port 4242

Add write access only when you explicitly want it, which also requires enabling write access in the Qovery Console:

claude mcp add --transport http qovery "https://mcp.qovery.com/mcp?read_write=true" --callback-port 4242

The skills that do the work are qovery-onboard, qovery-deploy, qovery-troubleshoot, qovery-optimize, qovery-speedup, qovery-preview and qovery-terraform; the MCP Server operates what already exists. After that, the prompt is plain English: "deploy this vLLM LightOnOCR-3 container to my AWS account with Qovery, on a GPU node pool, internal ingress only." The full walkthrough is in the AI agent quickstart and the Claude Code integration page.