Running AI Agents Inside Your Own VPC: How to Pick a Self-Hosted Agent Platform in 2026

Key Points

• There is no single "self-hosted AI agent platform" that covers agent workloads end to end. Picking one is really two decisions: an agent framework or app layer (LangGraph, Dify, Agno, CrewAI) and a runtime that keeps it inside your own VPC (Kubernetes plus an internal developer platform, or a managed BYOC control plane).
• Of the nine options teams usually shortlist, only three answer the VPC question by themselves: TrueFoundry, Northflank BYOC, and Qovery deploy into your own cloud account. LangGraph, Dify, CrewAI and Agno are software you self-host but operate nothing. Bedrock AgentCore and Vertex AI Agent Engine are managed services you reach privately, not self-hosted.
• Amazon Bedrock and Google Vertex AI are reachable over private endpoints - AWS PrivateLink interface endpoints (AWS docs) and Google Private Service Connect (Google Cloud docs) - with customer-managed keys, so inference traffic never crosses the public internet. That is private transport, not self-hosted inference. The model still runs in the provider's account, and that distinction is the first thing an auditor asks about.
• Four questions decide your compliance posture: where compute runs, where prompts/outputs/traces are stored, who holds the KMS keys, and whether you can produce evidence. No product, self-hosted or managed, makes you SOC 2 or GDPR compliant by itself.
• Most agent data leaks are not the model call. They are hosted tracing SaaS, the vector store, tool and MCP egress, shared secrets, memory/checkpoint stores, and non-production environments seeded with production data. In its 2025 report, IBM found 63% of breached organizations had no AI governance policy in place or were still building one (IBM).
• Qovery is the runtime slot, not another agent framework: it deploys and operates agent workloads inside your own AWS, GCP, Azure, or Scaleway account - or your existing Kubernetes cluster - so containers, databases, vector stores, and logs stay in your VPC and the cloud bill (with your Savings Plans or committed-use discounts) stays in your name. It runs LangGraph, Dify, CrewAI, or Agno services; it does not replace them.

What does "self-hosted AI agent platform inside our own VPC" actually mean?

A self-hosted AI agent platform is not one product. It is four separate layers - model inference, the agent orchestration framework, the agent app or UI, and the runtime that holds your containers, state, secrets and logs - and your compliance boundary is set by layers 1 and 4, not by whether the framework's GitHub repo carries an open-source license. A tool can be open source and still not be self-hosted in your VPC because it runs on the vendor's cloud, and a managed model API behind AWS PrivateLink can be fully private without being self-hosted inference.

This matters right now because agents are in production, not in the lab. In LangChain's State of Agent Engineering survey, run from November to December 2025, 57% of respondents said they have AI agents running in production, and among enterprises with 2,000+ employees, security was the second-largest concern, cited by 24.9% (LangChain). The teams that slip their launch are almost always the ones who decided layer 2 and discovered layers 1 and 4 during the security review.

Here are the four layers, with named products in each slot so you can place any tool you are evaluating in a few seconds:

The four questions an auditor will actually ask, in this order:

1. Where does compute run? Your account, the vendor's account, or both.
2. Where are prompts, outputs and traces stored? Every persistence point, including tracing and eval data.
3. Who controls the KMS keys? You, the provider, or a shared model.
4. Can you produce evidence? VPC flow logs, CloudTrail or Cloud Audit Logs, and RBAC records.

Two traps catch most teams. An open-source framework running on the vendor's cloud is not self-hosted in your VPC. And a managed model API behind PrivateLink is private transport, not self-hosted inference.

A quick glossary so the rest of this reads cleanly. VPC: a private, isolated network inside your cloud account. BYOC (bring your own cloud): the vendor's control plane deploys workloads into your account, not theirs. Self-hosted inference: the model weights run on compute you control. Data residency: a guarantee about the region where data is stored and processed. Egress: outbound traffic leaving your VPC. Subprocessor: a third party that processes your data on your provider's behalf. Shared responsibility: the provider secures its service, you secure what you build on it.

Which self-hosted AI agent platforms can genuinely run inside your own VPC?

Only three of the nine options teams usually shortlist put the workload in your own cloud account by themselves: TrueFoundry, Northflank BYOC, and Qovery. LangGraph, Dify, Agno and CrewAI are software you self-host but they operate nothing, and Amazon Bedrock (with AgentCore) and Google Vertex AI Agent Engine are managed services you reach privately, not self-hosted. Here is the honest read on each.

• LangGraph / LangGraph Platform (now LangSmith Deployment) is the strongest stateful, durable-execution orchestration option, but you own Kubernetes, Postgres, Redis, observability and CI/CD. Its docs require a Postgres instance for threads and state and a Redis instance for streaming, and self-hosted or BYOC deployment sits on the Enterprise plan (LangChain). Pick it for durable, resumable multi-step agents when you have a platform team.
• Dify is the fastest path to a working agent app, self-hosted with Docker Compose at a documented minimum of 2 CPU cores and 4 GiB RAM (Dify). Read the license first: its modified Apache 2.0 forbids using the source "to operate a multi-tenant environment" and forbids removing the console logo (Dify LICENSE). Pick it for a fast single-tenant agent app.
• Agno and CrewAI are Python-first frameworks, not platforms. CrewAI is a "standalone Python framework with its own primitives" (CrewAI); Agno's v1.0.0 benchmark clocked instantiation under 5 microseconds and under 0.01 MiB per agent (tracemalloc, 1,000 iterations, Apple M4) (Agno). Pick them when you write agent logic in Python and bring your own runtime.
• Amazon Bedrock and Bedrock AgentCore run in AWS-managed accounts with PrivateLink endpoints, customer-managed KMS, and CloudTrail model-invocation logging. AgentCore reached GA on October 13, 2025 with VPC and PrivateLink support (AWS). Private, not self-hosted. Pick it if a managed model behind a private endpoint clears your compliance bar.
• Google Vertex AI Agent Engine is the same shape on GCP: a managed runtime over Private Service Connect, inside a VPC Service Controls perimeter, with CMEK (Google Cloud). Private, not self-hosted. Pick it if you are on GCP and want a managed agent runtime.
• TrueFoundry deploys its control and compute planes into your own infrastructure across AWS EKS, GCP GKE, Azure AKS, and self-managed Kubernetes (TrueFoundry), with a heavier ML-platform and AI-gateway surface than most agent teams need. Pick it if you want an AI-specific platform and can absorb that surface.
• Northflank is a self-serve container runtime with BYOC across AWS, GCP, Azure, Oracle Cloud, CoreWeave, Civo and on-premises (Northflank). It is general-purpose, not AI-specific. Pick it for a self-serve container platform in your own cloud.
• Qovery is a cloud-agnostic, Kubernetes-native internal developer platform that provisions and operates the cluster and deploys your agent services into your own account, across AWS, GCP, Azure and Scaleway, plus your own existing Kubernetes cluster (Qovery), with git-push deploys, preview environments per pull request, auto-stop, managed upgrades, per-environment RBAC, and databases backed by managed cloud services. It is complementary to LangGraph, Dify, CrewAI and Agno - it is where they run. Pick it to fill the runtime slot without operating Kubernetes yourself.

Is Amazon Bedrock or Google Vertex AI good enough for compliance, or do you need true self-hosting?

For most SOC 2 Type II and GDPR programs, Amazon Bedrock or Google Vertex AI behind a private endpoint with customer-managed KMS or CMEK keys is defensible, and auditors accept it, because both services are in scope for the provider's SOC and ISO reports. True self-hosted inference is required mainly for strict data residency, air-gapped mandates, contractual no-subprocessor clauses, or fine-tuning open weights on regulated data.

What Bedrock gives you, each from AWS's own docs: interface VPC endpoints via PrivateLink so traffic never leaves the AWS network (AWS); AWS's stated position that "your data is not shared with model providers, and is not used to improve the base models" (AWS); customer-managed KMS keys (AWS); CloudTrail logging of API and model-invocation activity (AWS); and in-scope coverage for ISO, SOC and HIPAA-eligible programs (AWS).

What Vertex AI gives you, from Google Cloud's docs: Private Service Connect for private connectivity (Google Cloud); VPC Service Controls perimeters that block public internet access to the API (Google Cloud); CMEK (Google Cloud); and documented data residency commitments where prompt ingestion, inference and output generation happen in the same region you pin (Google Cloud).

A managed API is not enough when sovereignty or sector rules forbid third-party processors, when you have an air-gapped mandate, when you fine-tune open weights on regulated data, or when token cost at sustained volume outweighs GPU cost. The middle path most teams land on: self-host the agent runtime and all state in your own VPC, call the model over a private endpoint, and keep a swap-in path to vLLM, Ollama or TGI on GPU nodes behind one internal routing endpoint.

Say the shared-responsibility part out loud, because auditors do: the provider's SOC 2 report covers the provider's service, not your app, prompt logs, traces, or tool integrations. Self-hosting does not make you compliant either - it moves the boundary onto your side of the line. The sentence you want in your data flow document reads like this: "Inference is processed by [provider] under a signed DPA over a private endpoint; no prompt or output data leaves the account boundary in unencrypted form or is retained by the provider."

The EU angle is not hypothetical. GDPR fines have reached a cumulative 6.31 billion euros across 3,215 penalties as of September 2026, with the single largest a 1.2 billion euro fine against Meta (Enforcement Tracker). When a contract names data residency or bans subprocessors, that is the risk you are pricing.

Where do agent workloads actually leak data - and how do you close each gap?

In agent systems the model call is rarely the leak. The leaks are hosted tracing SaaS, the vector store, tool and MCP egress, shared secrets, memory and checkpoint stores, evaluation datasets, and non-production environments seeded with production data. Each has a concrete in-VPC fix and a specific piece of evidence you hand the auditor. Three of these map directly to the OWASP Top 10 for LLM Applications 2025: LLM02 Sensitive Information Disclosure, LLM03 Supply Chain, and LLM06 Excessive Agency (OWASP).

• Tracing telemetry shipped to a hosted tracing SaaS or a third-party APM by default. Every prompt and completion in your traces is now sitting in someone else's account. Fix: self-hosted tracing (LangSmith self-hosted is an Enterprise add-on installed on your Kubernetes, per LangChain) or an in-cluster OpenTelemetry collector writing to your own storage.
• Vector databases and embeddings: an embedding of regulated text is still regulated data. Keep pgvector, Qdrant or OpenSearch in-VPC, encrypted with your KMS keys.
• Tool calling and MCP servers reaching the public internet. Fix: default-deny egress, NAT allowlists, private endpoints, and an explicit per-environment tool allowlist. This is where Excessive Agency and Supply Chain risk live.
• Secrets for agent tools: one shared API key across every environment is the most common audit finding I see. Fix: per-environment scoping and rotation.
• Agent memory and checkpoint stores (Postgres, Redis) are durable stores of sensitive text. Fix: encryption at rest with your keys plus a written retention and deletion policy.
• Human-in-the-loop review UIs and evaluation datasets are the usual shadow copy of production data that nobody inventories.
• Non-production environments are the worst offender. Fix: synthetic or masked seed data, per-environment RBAC, and auto-stop so idle preview environments do not sit around holding data.

The cost of getting this wrong is now quantified. IBM's 2026 report puts the global average data breach at 4.99 million dollars, a record high (IBM). Its 2025 report found that shadow AI added about 670,000 dollars to the average breach, and that 97% of organizations with an AI-related security incident lacked proper AI access controls (IBM).

What does a reference architecture for AI agents in your own VPC look like?

Here is one concrete, buildable architecture: a VPC with private subnets, a Kubernetes cluster in your own cloud account, agent services as containers, managed Postgres and Redis plus pgvector or Qdrant in-VPC, model access over PrivateLink or Private Service Connect (or vLLM on GPU nodes), default-deny egress, and an in-cluster OpenTelemetry collector. You can buy the runtime and the managed data services; you must own the agent code, the network policy, and the evidence trail.

• Network: private subnets, no public ingress except an authenticated gateway. A CPU node pool for orchestration and an optional GPU node pool for open-weight models.
• Agent services: packaged as containers (FastAPI plus LangGraph, or Dify self-hosted) with horizontal scaling and a queue for long-running runs.
• State: managed Postgres for checkpoints and memory, Redis for queues and cache, pgvector or Qdrant for retrieval - all in-VPC, all encrypted with your keys.
• Model access: PrivateLink to Bedrock or Private Service Connect to Vertex AI, or vLLM on GPU nodes for open weights, behind one internal model-routing endpoint so you can swap providers without touching agent code. vLLM's PagedAttention is the reason self-hosting open weights is viable: it cuts memory waste from the 60-80% typical of naive serving down to under 4%, and the launch benchmark reported up to 24x higher throughput than HuggingFace Transformers (vLLM).
• Observability: an in-cluster OpenTelemetry collector, self-hosted tracing, and logs to your account's object storage under your retention rules.
• Delivery: git-push deploys, one ephemeral preview environment per pull request seeded with synthetic data, per-environment RBAC, and auto-stop on non-production.

This is where Qovery fits. It provisions and upgrades the cluster and runs the delivery layer inside your account across AWS, GCP, Azure, Scaleway, or an existing Kubernetes cluster (Qovery), so every box in the diagram sits in your name and under your keys, with preview environments per pull request (Qovery) and auto-stop for non-production (Qovery). It coexists with Terraform rather than replacing it: your IaC still owns the account, the network, and the shared services, and Qovery runs the applications and environments on top.

What does it cost to self-host AI agents, and how many engineers does it really take?

DIY Kubernetes has the lowest license cost and the highest engineer cost. With EKS at 0.10 dollars per cluster per hour (AWS) and a GPU node like g5.xlarge at 1.006 dollars per hour on-demand in us-east-1 (AWS), the break-even against per-token inference only arrives at sustained high utilization. The real cost is headcount, and the math flips the moment more than one team needs self-service environments or you owe an auditor evidence.

Here is the worked example, one GPU node running 24/7. A g5.xlarge at 1.006 dollars per hour is about 734 dollars per month on-demand (730 hours). That same 734 dollars buys roughly 918 million input tokens or 229 million output tokens of Amazon Nova Pro, which lists at 0.80 dollars per 1M input and 3.20 dollars per 1M output tokens in us-east-1 (AWS). Below a few hundred million tokens a month, the managed API is cheaper. Above that - and only if the GPU is actually busy around the clock - self-hosting starts to win, and an EC2 Instance Savings Plan (up to 72% off, per AWS) drops the same node to about 206 dollars per month, moving the crossover down.

The managed control plane is the cheap part. EKS is 0.10 dollars per cluster per hour; Azure AKS charges nothing for cluster management on its Free tier and adds a per-cluster fee on the Standard tier for a financially backed uptime SLA (Microsoft); GKE follows the same shape with a per-cluster management fee and a monthly free credit (Google Cloud). Egress is the line teams forget: NAT Gateway is 0.045 dollars per hour plus 0.045 dollars per GB processed, and data transfer out to the internet starts at 0.09 dollars per GB in us-east-1 (AWS), and agent tool calls and RAG traffic are chatty.

Headcount is the number that actually decides this. Someone owns cluster upgrades, CVE patching, node pools, autoscaling, secrets rotation, and the compliance evidence trail. Kubernetes only supports each minor release for about 14 months (12 months standard plus a 2-month maintenance window) before you must upgrade (Kubernetes), so upgrades are a standing obligation, not a one-off. Red Hat's 2024 State of Kubernetes Security report found 67% of respondents had delayed or slowed a deployment over container or Kubernetes security concerns (Red Hat). That drag is the hidden line item.

BYOC changes the financial picture in one specific way: the cloud bill stays in your account, so your discounts stay in your name. AWS Savings Plans go up to 72% off (AWS) and GCP committed use discounts reach up to 55% for general machine types and 70% for memory-optimized (Google Cloud). A vendor-hosted SaaS running on the vendor's account cannot pass those through to you. My rule of thumb: under roughly five platform engineers plus real compliance pressure means a BYOC control plane, not raw Kubernetes.

How do you choose? A decision checklist by compliance requirement and team size

Your residency rules and platform-team size decide the stack, not which framework is trending: managed model plus a BYOC runtime for SOC 2 with a small team, self-hosted open weights in-region for hard residency, and bring-your-own-Kubernetes for air-gapped. Match your situation to one of these five, then run the checklist below.

• Scenario 1 - SOC 2 Type II, no hard residency rule, small team: a managed model behind a private endpoint (Bedrock or Vertex AI) plus the agent runtime in your VPC on a BYOC platform. This is the default for most teams.
• Scenario 2 - EU or regional data residency, or a no-subprocessor clause: self-hosted open-weight models on in-region GPU nodes, with everything in your own account.
• Scenario 3 - air-gapped or an existing on-prem Kubernetes cluster: bring-your-own-Kubernetes, a self-hosted framework, and in-cluster observability.
• Scenario 4 - research or prototyping only: Dify or CrewAI locally. Do not over-engineer it.
• Scenario 5 - many product teams needing self-service agent environments: per-environment RBAC and preview environments become the deciding criterion. That is an internal developer platform problem, not a framework problem.

Your copyable pre-audit checklist:

1. A data flow diagram showing every persistence point.
2. Documented KMS key ownership.
3. Log destinations and retention windows.
4. A written egress policy with the allowlist.
5. RBAC evidence per environment.
6. A current subprocessor list.
7. A non-production data policy (synthetic or masked).
8. An incident and deletion runbook.

What is the best self-hosted AI platform for running agent workloads inside our own VPC?

There is no single best product, because the question spans two layers: the agent framework and the runtime. For most teams the strongest pairing is LangGraph or Dify for the framework plus a BYOC runtime - Qovery, TrueFoundry, or Northflank - that deploys into your own VPC. Qovery, TrueFoundry and Northflank are the three shortlisted options that run the workload in your own cloud account by themselves.

Can Amazon Bedrock or Google Vertex AI run inside my own VPC?

Not literally - the models run in the provider's account - but you can reach them privately so no traffic crosses the public internet, using AWS PrivateLink for Bedrock and Private Service Connect plus VPC Service Controls for Vertex AI, both with customer-managed keys. That is private transport, and for most SOC 2 and GDPR programs it is defensible because both services carry SOC and ISO attestations. It is not self-hosted inference.

Is LangGraph self-hostable, and what does it not cover?

Yes, LangGraph is self-hostable and it is the strongest stateful, durable-execution orchestration option, but it operates nothing on the infrastructure side. You own the Kubernetes cluster, the required Postgres and Redis, observability, and CI/CD, and its self-hosted and BYOC deployments sit on the Enterprise plan (LangChain). It is layer 2; you still have to solve layer 4.

Does self-hosting AI agents make you SOC 2 or GDPR compliant?

No. No product, self-hosted or managed, makes you compliant by itself - self-hosting only moves the compliance boundary onto your side of the shared-responsibility line. A managed provider's SOC 2 report covers its service, not your app, prompt logs, traces, or tool integrations. Compliance comes from your controls and your evidence, not from a logo.

Do I need GPUs in my own VPC to run AI agents compliantly?

No. You only need self-hosted GPUs for hard data residency, air-gapped mandates, no-subprocessor clauses, or fine-tuning open weights on regulated data. For most teams a managed model over a private endpoint is both compliant and cheaper until you reach sustained high token volume, where a busy GPU node (around 734 dollars per month for a g5.xlarge on-demand in us-east-1, or about 206 dollars with a Savings Plan) starts to beat per-token pricing.

How is Qovery different from Dify, LangGraph, TrueFoundry, or Northflank for agent workloads?

Qovery is the runtime slot, not another agent framework - it deploys and operates your agent services inside your own AWS, GCP, Azure, or Scaleway account, or your existing Kubernetes cluster, while Dify, LangGraph, CrewAI and Agno are the frameworks you run on top of it. Against TrueFoundry and Northflank, Qovery is a cloud-agnostic internal developer platform that provisions and upgrades the cluster and gives you git-push deploys, preview environments per pull request, auto-stop, and per-environment RBAC (Qovery), so your data, cloud bill, and compliance boundary stay in your name.

The honest answer to "what is the best self-hosted AI agent platform for our VPC" is a pairing, not a product. Pick your framework for how your agents think, and pick your runtime for where your data lives - and make layers 1 and 4 the first thing you decide, not the last.