Webinar · Oct 20: The migration takes 2 weeks. Deciding to do it takes 6 months.

Qovery API unavailable after an Envoy Gateway upgrade

On 8 October 2026, an upgrade of Envoy Gateway from 1.8 to 1.9 on the cluster that hosts the Qovery Control Plane made the Qovery API unreachable twice, for about 11 minutes each time. Environment and cluster deployments were blocked until we rolled the cluster back to Envoy Gateway 1.8.3.

Date
October 8, 2026
Status
Resolved
Downtime
22 min
Affected
Qovery API and Console, Environment deployments, Cluster deployments

Impact

During both windows, the Qovery API did not answer. The Console, the CLI, the Terraform provider and the MCP server all depend on it, so none of them could read or change your setup.

Environment and cluster deployments could not start. Cluster deployments that were already running lost contact with the Control Plane and failed. We redeployed every affected cluster on 8 October.

Applications already running on your clusters kept serving traffic. They run in your own cloud account and do not need the Control Plane to answer requests.

Timeline (UTC)

  1. 07:08We deploy Envoy Gateway 1.9.1 to the Control Plane cluster.
  2. 09:21The first configuration translation error is recorded. The Qovery API stops answering.
  3. 09:33Rollback to Envoy Gateway 1.8.3 complete. Service restored after about 11 minutes.
  4. 10:42We deploy a targeted patch for the configuration incompatibility found in Envoy Gateway 1.9.
  5. 10:53The patch does not fix the whole problem. We roll back to 1.8.3 again and service is restored after 11 minutes.

Cause

Envoy Gateway 1.9 worked on our development and staging environments, but failed on the production cluster that hosts the Qovery Control Plane.

A configuration patch we apply to Envoy no longer matched the configuration that 1.9 generates, so Envoy rejected it. We found and fixed that incompatibility. The second incident showed that another issue, specific to production, remained.

What we are changing

  • Reproduce the remaining issue outside the Control Plane cluster before trying again.
  • Test the complete Control Plane deployment workflow on the new version before the next production attempt.
  • Run the next production attempt during a maintenance window announced in advance, outside European and US working hours.
  • Keep the rollback to Envoy Gateway 1.8.3 ready for the whole rollout.

The Envoy Gateway 1.9 rollout is paused on the Control Plane cluster. We will announce the maintenance window for the next attempt on our status page and in the Console before it starts.

Last updated October 8, 2026

Live status on status.qovery.com