Bring Your Own Kubernetes (BYOK)

The Qovery Operator connects a Kubernetes cluster you already run to Qovery. You create the cluster in the Qovery Console, install the Operator once with Helm, and Qovery then installs and updates the Qovery stack on the cluster from a platform profile.

The Qovery Operator installation is in Preview and available to every organization on the Enterprise plan. Current limitations lists what the Preview does not cover yet. Other providers and on-premise clusters use the installation with the Qovery CLI.

How it works

Qovery never stores the kubeconfig of your cluster. You install the Operator yourself, with your own kubectl and helm access.
The Operator runs in the qovery namespace. It opens an outbound connection to Qovery and runs the Qovery deployments as Kubernetes Jobs inside your cluster. These Jobs run with cluster-admin rights.
The platform profile of the cluster lists the components Qovery installs and their settings. The list can grow as Qovery adds components, and the profile page of the cluster always shows the current one. Today, the profile holds:
- Qovery stack: the Qovery agent, the shell agent, the Qovery priority classes and the Qovery StorageClasses used by container databases and service storage (for example aws-ebs-gp3-0, gcp-pd-ssd, scw-sbv-ssd-0). A StorageClass that already exists with the same name is kept as it is, and Qovery does not change the default StorageClass of the cluster.
- Log infra: Loki and Alloy, for application logs.
- Gateway API: Envoy Gateway, which exposes your applications.
- DNS and certificates: cert-manager and ExternalDNS, for the Qovery-provided domain and its TLS certificates.
Application images are built inside your cluster and pushed to a registry Qovery creates from your cloud credential: Amazon ECR on AWS, Artifact Registry on GCP, Scaleway Container Registry on Scaleway.

Prerequisites

A Qovery organization on the Enterprise plan

A running Kubernetes cluster on AWS (EKS), GCP (GKE) or Scaleway (Kapsule)

kubectl access to the cluster with cluster-admin rights and helm on your machine, only to install the Operator once. Later Operator updates go through Qovery.

At least one node with 4 vCPU and 8 GiB of memory free, for example an 8 vCPU node. Each build runs in a builder pod that requests the CPU and RAM of the service build settings, 4000m and 8 GiB by default, so a 4 vCPU node cannot host it.

A cluster that does not already run cert-manager, ExternalDNS or Envoy Gateway

A default StorageClass (kubectl get storageclass), unless Loki stores logs in object storage. Since EKS 1.30, EKS creates gp2 without marking it as the default.

On EKS, the Amazon EBS CSI driver add-on with its IAM role. The Loki pvc storage, container databases and service storage all use it. GKE and Kapsule include their CSI driver.

The cluster can create a public load balancer for a Service of type LoadBalancer, which Envoy Gateway uses to expose your applications. On EKS, the public subnets must carry the kubernetes.io/role/elb=1 tag.

For services with autoscaling, metrics-server (kubectl get apiservice v1beta1.metrics.k8s.io). GKE and Kapsule include it. On EKS, add the metrics-server add-on.

A working cluster DNS (CoreDNS) that resolves public names

Outbound internet access from the cluster to Qovery and to public container registries

A Qovery cloud credential for the provider of the cluster. You can create it during the installation.

The preflight checks verify several of these prerequisites at each deployment of the cluster.

Credential permissions

The cloud credential lets Qovery create the cluster registry, push the images it builds and clean them up. It does not need access to the Kubernetes cluster itself.

AWS
GCP
Scaleway

The credential only needs Amazon ECR. Do not use the Qovery CloudFormation stack offered in the credential form, because it grants the permissions of a cluster managed by Qovery.Recommended: an IAM role that Qovery assumes through STS

Create an IAM role with this trust policy, where <ORGANIZATION_ID> is the ID of your Qovery organization, shown in the Console URL (console.qovery.com/organization/<ORGANIZATION_ID>/…):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::283389881690:role/qovery-deployer",
          "arn:aws:iam::283389881690:user/qovery-deployer"
        ]
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringLike": {
          "sts:ExternalId": "<ORGANIZATION_ID>:*"
        }
      }
    }
  ]
}

Qovery sends the external ID <organization id>:<cluster id>, and <organization id>:00000000-0000-0000-0000-000000000000 when it checks the credential. The <ORGANIZATION_ID>:* condition accepts both.

Attach this permissions policy to the role, where <ACCOUNT_ID> is your AWS account ID and <CLUSTER_REGION> the region of the cluster. Keep the us-east-1 statement even if the cluster runs in another region, because Qovery checks the credential there. The ClusterRegistry statement only covers repositories whose names start with z or qovery-mirror-, the prefixes Qovery uses (z<first 8 characters of the cluster ID>-…). Do not give your own repositories in that region these prefixes.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "EcrLogin",
      "Effect": "Allow",
      "Action": "ecr:GetAuthorizationToken",
      "Resource": "*"
    },
    {
      "Sid": "CredentialCheck",
      "Effect": "Allow",
      "Action": "ecr:DescribeRepositories",
      "Resource": "arn:aws:ecr:us-east-1:<ACCOUNT_ID>:repository/*"
    },
    {
      "Sid": "ClusterRegistry",
      "Effect": "Allow",
      "Action": [
        "ecr:DescribeRepositories",
        "ecr:CreateRepository",
        "ecr:PutLifecyclePolicy",
        "ecr:TagResource",
        "ecr:ListTagsForResource",
        "ecr:DescribeImages",
        "ecr:BatchDeleteImage",
        "ecr:DeleteRepository",
        "ecr:BatchCheckLayerAvailability",
        "ecr:InitiateLayerUpload",
        "ecr:UploadLayerPart",
        "ecr:CompleteLayerUpload",
        "ecr:PutImage",
        "ecr:BatchGetImage",
        "ecr:GetDownloadUrlForLayer"
      ],
      "Resource": [
        "arn:aws:ecr:<CLUSTER_REGION>:<ACCOUNT_ID>:repository/z*",
        "arn:aws:ecr:<CLUSTER_REGION>:<ACCOUNT_ID>:repository/qovery-mirror-*"
      ]
    }
  ]
}

Set the maximum session duration of the role to at least 4 hours, for example 8 hours. Qovery requests 4-hour sessions, and with the default of 1 hour the credential check fails with DurationSeconds exceeds the MaxSessionDuration.
aws iam update-role --role-name <ROLE_NAME> --max-session-duration 28800
In the credential form, set Authentication type to Assume role via STS, skip the CloudFormation step, then fill in Name and Role ARN.

Alternative: access keysCreate an IAM user with the same permissions policy, then set Authentication type to Static credentials in the credential form and paste its access key and secret key.For both

The EKS node role must be able to pull from ECR in the same account, for example with the AmazonEC2ContainerRegistryPullOnly managed policy.
The policy above covers only ECR. If Terraform services use the cluster credentials, add the permissions they need to the same role or user.

Qovery creates, reads and deletes Artifact Registry repositories and images in the project of the credential, in the cluster region. The setup commands shown in the credential form grant the permissions of a cluster managed by Qovery. For BYOK, grant only the roles below.

Grant the service account the Artifact Registry Administrator role (roles/artifactregistry.admin) on the project.
With a JSON key, Qovery checks the key by listing the Cloud Storage buckets of the project, so also grant a role with the storage.buckets.list permission, for example Storage Bucket Viewer (roles/storage.bucketViewer). A Workload Identity Federation credential does not need it.
With Workload Identity Federation, run the setup command of the credential form (see the GCP cluster credentials), then replace the Qovery custom role it binds by the Artifact Registry role:
gcloud projects remove-iam-policy-binding <PROJECT_ID> --member serviceAccount:<SERVICE_ACCOUNT_EMAIL> --role projects/<PROJECT_ID>/roles/qovery_role gcloud projects add-iam-policy-binding <PROJECT_ID> --member serviceAccount:<SERVICE_ACCOUNT_EMAIL> --role roles/artifactregistry.admin
The setup command also lets the service account through the organization policy constraints/iam.allowServiceAccountCredentialLifetimeExtension. Keep that exception, because Qovery requests 4-hour tokens.
Qovery creates no image pull secret for Artifact Registry. The GKE node service account must be able to pull from Artifact Registry in the project of the credential, for example with the roles/artifactregistry.reader role.

Qovery creates, reads and deletes Container Registry namespaces and images in the project of the API key, in the cluster region.

Give the API key the ContainerRegistryFullAccess permission set on the project.
Qovery checks the key by listing the Container Registry namespaces of the fr-par region, whatever the cluster region.

Loki object storage

Skip this section if Loki stores logs in a persistent volume, the default.

Create the bucket and its identity before you deploy the profile, then fill in their fields on the Loki page of the profile. Choose the storage before the first deployment, because changing it later does not move the logs already stored.

AWS: S3 bucket and IAM role

Create the bucket in the region of the cluster, since Loki uses that region. Loki reaches the bucket through IRSA, so your EKS cluster needs an IAM OIDC provider. Create an IAM role with this permissions policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket"],
      "Resource": "arn:aws:s3:::<BUCKET_NAME>"
    },
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::<BUCKET_NAME>/*"
    }
  ]
}

And this trust policy, for the loki service account of the qovery namespace:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Federated": "arn:aws:iam::<ACCOUNT_ID>:oidc-provider/<OIDC_PROVIDER>" },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "<OIDC_PROVIDER>:sub": "system:serviceaccount:qovery:loki",
          "<OIDC_PROVIDER>:aud": "sts.amazonaws.com"
        }
      }
    }
  ]
}

<OIDC_PROVIDER> is the OIDC issuer of the cluster without https://, for example oidc.eks.eu-west-3.amazonaws.com/id/EXAMPLE. On the profile, fill in S3 bucket name and Loki IAM role ARN.

GCP: GCS bucket and service account

Loki reaches the bucket through GKE Workload Identity. Enable it on the cluster and its node pools.

Create the bucket and a GCP service account, and grant the service account the roles/storage.objectAdmin role on the bucket.

Let the loki service account of the qovery namespace act as it:

gcloud iam service-accounts add-iam-policy-binding <SERVICE_ACCOUNT_EMAIL> --role roles/iam.workloadIdentityUser --member "serviceAccount:<PROJECT_ID>.svc.id.goog[qovery/loki]"

On the profile, fill in GCS bucket name and GCP service account email.

Scaleway: S3-compatible bucket

Create the bucket and an API key that can read and write it.

Create the credentials Secret in the qovery namespace, with the keys S3_ACCESS_KEY_ID and S3_SECRET_ACCESS_KEY. The credentials stay in your cluster and never reach Qovery.

read -r -p "Access key: " S3_ACCESS_KEY_ID
read -r -s -p "Secret key: " S3_SECRET_ACCESS_KEY; echo
kubectl -n qovery create secret generic loki-s3-credentials --from-literal=S3_ACCESS_KEY_ID="$S3_ACCESS_KEY_ID" --from-literal=S3_SECRET_ACCESS_KEY="$S3_SECRET_ACCESS_KEY"
unset S3_ACCESS_KEY_ID S3_SECRET_ACCESS_KEY

Create the namespace first if the Operator is not installed yet: kubectl create namespace qovery.

On the profile, fill in S3-compatible bucket name, S3-compatible endpoint (an https:// URL, for example https://s3.fr-par.scw.cloud), S3-compatible region (for example fr-par) and S3-compatible credentials Secret (loki-s3-credentials).

Create the cluster

Open the cluster creation page

In the Qovery Console, go to Clusters and create a new cluster. Under Connect existing cluster (BYOK), select AWS, GCP or Scaleway.

Add cluster page with the Connect existing cluster (BYOK) section

Fill in the general information

Cluster name: the name of the cluster in Qovery.
Production cluster: off by default. Turn it on for a production cluster.
Credentials: select a cloud credential of the provider, or create one.
Region: the region of your cluster. On Scaleway, the list shows zones. Select a zone of the cluster region, for example fr-par-1 for a cluster in fr-par.

Click Continue.

Configure the Qovery Operator

Leave the settings as they are, unless you want the Operator and its Jobs pinned to specific nodes. To pin them, fill in Node group label key and Node group label value with the label of your node group or node pool. The default key is the EKS one. Use the key of your cluster:

Cluster	Node label key
EKS managed node group	`eks.amazonaws.com/nodegroup`
EKS with Karpenter	`karpenter.sh/nodepool`
GKE	`cloud.google.com/gke-nodepool`
Scaleway Kapsule	`k8s.scaleway.com/pool-name`

If those nodes are tainted, add their taints under Node group taints to tolerate. These settings place only the Operator and its Jobs, not the platform components. You cannot use kubernetes.io/arch as a key, because Qovery sets the architecture of each build itself.Click Continue and Qovery creates the cluster. It creates the registry repositories in your cloud account at the first deployment that needs them.

Install the Qovery Operator

Download the Helm values file values.yaml. It contains the credential of the cluster, so keep it private and delete it once the Operator is installed.
From the folder of the values file, with a kubeconfig pointing to your cluster, run the Helm command shown in the Console.

Keep the page open. Once the Operator connects to Qovery, the Console opens the profile of the cluster and shows the Operator status at the bottom of the profile sidebar.

Install Qovery Operator step with the Helm values file and command

Configure and deploy the platform profile

On the profile page, review the settings of each component. If you change a setting, click Save and deploy. To keep the defaults, open the Manage deployment menu of the cluster and click Deploy. Qovery installs the components of the profile through the Operator. Follow the progress in the cluster logs.

Choose the Loki storage before the first deployment. The default, pvc, needs a default StorageClass. Object storage (s3 on AWS, gcs on GCP, s3Compatible on Scaleway) needs a bucket and an identity you create, see Loki object storage, and is required for high availability. Changing the storage later does not move the logs already stored.

Once the deployment succeeds, you can deploy your applications on the cluster.

Preflight checks

Before each deployment of the cluster, Qovery checks your configuration and the cluster itself. The results appear in the cluster deployment logs, under Control-plane preflight and In-cluster preflight checks, with a recommended action for each check that fails. The checks cover:

the Operator connection, the platform profile and its required inputs, and the DNS provider configuration;
access to the Kubernetes API, the Kubernetes permissions of the deployment Jobs, and target namespaces still being deleted;
the cluster DNS, and access from the cluster to Qovery, to the Helm chart registries and to the container registries;
Helm releases, CRDs or a cert-manager installation that conflict with the profile;
a default StorageClass, when a component of the profile needs one.

A blocking check that fails stops the deployment before Qovery changes anything in the cluster. The other checks only add a warning to the logs, and the deployment continues. Today, the checks on the Operator connection, the platform profile, its inputs and the DNS provider block, and so do Kubernetes API access and target namespaces still being deleted. The other in-cluster checks only warn.

Current limitations

DNS: applications are exposed on the domain provided by Qovery. The DNS provider of the cluster cannot be changed.
Observe and metrics: Observe and the Monitoring graphs of the cluster and its services are not available on BYOK clusters yet, and Qovery installs no metrics stack on them. If you need them, tell us, since we prioritize BYOK features from customer requests.
Existing installations: Qovery does not adopt an existing cert-manager, ExternalDNS or Envoy Gateway installation. Use a cluster without them.
Removal: deleting the cluster in Qovery does not uninstall anything from the cluster. See Remove the cluster.

Need a component, a DNS provider or a cloud provider that is not covered yet? Contact Qovery Support through your usual channel: in-app chat, shared Slack or Microsoft Teams channel, or email.

Remove the cluster

Deleting the cluster in Qovery removes only its Qovery configuration. The Operator and the platform keep running in your cluster, and the Envoy load balancer keeps its public IP and keeps incurring cloud costs until you uninstall the gateway releases (step 2).

In Qovery, delete the environments of the cluster, then delete the cluster.

Uninstall the gateway releases while envoy-gateway still runs:

helm -n qovery uninstall qovery-cluster-gateway qovery-gateway-class

If envoy-gateway is already gone, the GatewayClass keeps its finalizer. Remove it with:

kubectl patch gatewayclass qovery-public-gateway --type=merge -p '{"metadata":{"finalizers":null}}'

List the other releases of the qovery namespace with helm -n qovery list and uninstall each one with helm -n qovery uninstall <release>, qovery-operator last. Then delete the namespace with kubectl delete namespace qovery.
Delete the CRDs that the platform installed (Gateway API, Envoy Gateway and cert-manager), if nothing else uses them.
Delete the registry repositories Qovery created in your cloud account: the z<first 8 characters of the cluster ID>-… and qovery-mirror-… ECR repositories, the Artifact Registry repositories or the Scaleway Container Registry namespaces. Mirror repositories of deleted services can remain, so check for them too.
If Loki used object storage, delete the bucket and its identity.

Troubleshooting

The Operator does not connect

Check that the Operator pod is running: kubectl -n qovery get pods.
Read its logs: kubectl -n qovery logs deployment/qovery-operator --tail=100.
A gateway communication error or transport error at startup is often a cluster DNS issue. Check that a Pod resolves the Qovery endpoint:
kubectl -n default run qovery-dns-preflight --rm -i --restart=Never --image=busybox:1.36 -- nslookup engine.qovery.com
The lookup must return an address. A timeout or no servers could be reached means the cluster DNS must be fixed first.
Check that the cluster can reach the internet.

Loki stays Pending

With the pvc storage, the Loki volume needs a default StorageClass and, on EKS, the EBS CSI driver. Check kubectl get storageclass and, on EKS, kubectl -n kube-system get pods -l app.kubernetes.io/name=aws-ebs-csi-driver. To mark a class as default, run kubectl annotate storageclass <NAME> storageclass.kubernetes.io/is-default-class=true, then deploy the cluster again. With object storage, Loki has no volume. Check the bucket and identity fields of the Loki page against Loki object storage.

A build stays on Provisioning builder

No node has enough free CPU and memory for the builder pod. kubectl -n qovery get pods shows a build-… pod in Pending, and kubectl -n qovery describe pod <pod> reports Insufficient cpu or Insufficient memory. Add a node with at least 4 vCPU and 8 GiB free, or lower CPU and RAM in the build settings of the service.

The credential is refused when the cluster is created

Qovery checks that the credential can use the registry of the provider in the cluster region before it creates anything. Compare the credential with Credential permissions.

Overview

Account & Organization

Clusters & Cloud

Source Control

Projects & Environments

Services

Deployment & CI/CD

Environment Variables & Secrets

Observability

Notifications

Networking & Advanced

Bring Your Own Kubernetes (BYOK)

How it works

Prerequisites

Credential permissions

Loki object storage

Create the cluster

Preflight checks

Current limitations

Remove the cluster

Troubleshooting

​How it works

​Prerequisites

​Credential permissions

​Loki object storage

​Create the cluster

​Preflight checks

​Current limitations

​Remove the cluster

​Troubleshooting

How it works

Prerequisites

Credential permissions

Loki object storage

Create the cluster

Preflight checks

Current limitations

Remove the cluster

Troubleshooting