Qovery API unavailable after an Envoy Gateway upgrade
On 8 October 2026, an upgrade of Envoy Gateway from 1.8 to 1.9 on the cluster that hosts the Qovery Control Plane made the Qovery API unreachable twice, for about 11 minutes each time. Environment and cluster deployments were blocked until we rolled the cluster back to Envoy Gateway 1.8.3.
- Date
- October 8, 2026
- Status
- Resolved
- Downtime
- 22 min
- Affected
- Qovery API and Console, Environment deployments, Cluster deployments
Impact
During both windows, the Qovery API did not answer. The Console, the CLI, the Terraform provider and the MCP server all depend on it, so none of them could read or change your setup.
Environment and cluster deployments could not start. Cluster deployments that were already running lost contact with the Control Plane and failed. We redeployed every affected cluster on 8 October.
Applications already running on your clusters kept serving traffic. They run in your own cloud account and do not need the Control Plane to answer requests.
Timeline (UTC)
- 07:08We deploy Envoy Gateway 1.9.1 to the Control Plane cluster.
- 09:21The first configuration translation error is recorded. The Qovery API stops answering.
- 09:33Rollback to Envoy Gateway 1.8.3 complete. Service restored after about 11 minutes.
- 10:42We deploy a targeted patch for the configuration incompatibility found in Envoy Gateway 1.9.
- 10:53The patch does not fix the whole problem. We roll back to 1.8.3 again and service is restored after 11 minutes.
Cause
Envoy Gateway 1.9 worked on our development and staging environments, but failed on the production cluster that hosts the Qovery Control Plane.
A configuration patch we apply to Envoy no longer matched the configuration that 1.9 generates, so Envoy rejected it. We found and fixed that incompatibility on 8 October. The second incident showed that another issue, specific to production, remained.
On 9 October we found it. The Control Plane uses Envoy policies that run Lua code. Envoy Gateway 1.8 allowed Lua by default; 1.9 no longer does, and our configuration did not turn it back on. We reproduced the failure on a test environment, and confirmed that the fix restores traffic with Envoy Gateway 1.9 and the same policies in place.
What we are changing
- Deploy the fix to the Control Plane during a scheduled maintenance window, outside European and US working hours.
- Test the complete Control Plane deployment workflow on the new version before each production step.
- Keep the rollback to Envoy Gateway 1.8.3 ready for the whole rollout.
The fix ships during a scheduled maintenance on Saturday 10 October, 08:00 to 09:00 Paris time. No impact is expected on your side. Details on our scheduled maintenance page. View the schedule
Last updated October 9, 2026
Live status on status.qovery.com