After the eMudhra CA Key Compromise: How to Audit and Rotate Certificate Chains Across Every Kubernetes Cluster

Key points:

• The incident is real and confirmed. On 17 October 2026 a researcher showed that the private keys of eight eMudhra intermediate CAs under emSign ECC Root CA - G3 were "effectively all-zeroes apart from the last byte" and recoverable near instantly. eMudhra revoked them with reason caCompromise on 18 October, and Mozilla and Google removed the G3 root (Mozilla Bugzilla 2083449, Google Security Blog).
• Browsers were protected fast; your pods were not. Google says CT-enforcing clients could not have been attacked. Go services, Python requests, Java and most in-cluster clients do not enforce Certificate Transparency, so they trust whatever their image's CA bundle trusts until you patch it.
• cert-manager and Crossplane internal CAs are not affected. Their webhook CAs are self-signed and cluster-local. What you patch is (1) any leaf certificate issued by eMudhra, usually through an ACME ClusterIssuer, (2) every trust store your workloads read (trust-manager bundles, base images, Go toolchain, certifi, JDK), and (3) Crossplane provider images that call cloud APIs.
• Rotation order matters. Inventory, stop eMudhra issuance, re-issue leaves with cmctl renew, then push patched trust bundles and images, then verify against a live test endpoint. A second deadline is close: Chrome stops trusting new certificates under emSign Root CA - G1 whose earliest SCT is after 16 November 2026, 23:59:59 UTC.
• This is a fleet problem. One cluster is an afternoon. Twenty clusters across three clouds is a week, unless node upgrades and redeploys are already one action. Qovery runs that layer on your own AWS, GCP, Azure or Scaleway accounts or existing clusters.

A certificate authority's intermediate keys were so weak that anyone could compute them from the public certificate. In Mozilla's words, each key was "effectively all-zeroes apart from the last byte." If you run Kubernetes, your first question is probably the one I got from three platform teams this month: "our cert-manager and Crossplane deployments need patching, what is the safe way to do it across all our clusters?"

This can happen to any CA, and the people at eMudhra are now dealing with the worst week a PKI team can have. I wish them a fast and clean recovery. What follows is a runbook you can execute today, built only on what was publicly reported and on commands that exist.

What exactly happened with eMudhra, and what is confirmed?

eMudhra's eight G3 intermediate CA private keys were weak enough to recover from their certificates, the intermediates were revoked as compromised on 18 October 2026, and the emSign ECC Root CA - G3 was removed from Mozilla's and Google's root stores. Everything below comes from primary sources: the Mozilla incident bug, the Google Security Blog, eMudhra's statements in that bug, and vendor advisories.

The root cause, per eMudhra's own comment in the Mozilla bug, traces to the 30 April 2025 key generation ceremony: a test of a wrap/unwrap step created a second, weaker key pair, and "these second pair of weaker keys were used by mistake to create the csr (because the key labels were similar)." The keys were P-384, and Mozilla's write-up says they were "effectively all-zeroes apart from the last byte" and recoverable "near instantly via the baby-step giant-step discrete log algorithm." From the ceremony to the browser blocks, that is an exposure window of about 17 months.

Three numbers help size the blast radius:

• About 30,000 certificates needed migration, and eMudhra said "about 97% came via ACME" (Mozilla Bugzilla 2083449). ACME is exactly how cert-manager talks to a CA, so if any of your clusters used eMudhra, it was almost certainly through a cert-manager ACME issuer.
• 107,107 certificates were logged to CT between 20 and 21 October, after revocation, including 106,050 proof-of-concept certificates whose only SAN is google.com (Mozilla Bugzilla 2083449). People recovered the keys and forged certificates in public.
• No TLS certificate chaining to the six TLS intermediates was in CT before the block, so Google states that "no fraudulent certificates issued by these CAs could have been used against CT-enforcing clients" (Google Security Blog).

There is no public report of malicious use of the keys against a specific organization. I would not assume any, and I would not assume none either for clients that ignore CT.

Do cert-manager and Crossplane themselves need patching after the eMudhra compromise?

Neither cert-manager nor Crossplane has a code vulnerability from this incident. You patch three things around them: certificates they issued from eMudhra, the trust stores of the pods that make TLS calls, and the binaries built with an unpatched Go toolchain. Their own webhook PKI is self-signed and never chained to eMudhra.

Here is how each piece maps to the incident:

I checked the Crossplane source to be sure. The initializer in internal/initializer/tls.go generates a self-signed CA with NotAfter: time.Now().AddDate(10, 0, 0), stores it in the crossplane-root-ca Secret, issues crossplane-tls-server and crossplane-tls-client from it, and writes the CA into the caBundle of Crossplane's webhook configurations and CRD conversion webhooks. Provider revisions get their own <revision>-tls-server and <revision>-tls-client Secrets. None of that touches a public CA.

Crossplane's exposure is the provider's outbound TLS. A provider holds cloud credentials and calls cloud API endpoints. A client that still trusts the G3 hierarchy and does not check revocation or CT could be served a forged certificate for that endpoint by anyone in the network path. Go does not enforce CT, which is why the Go team shipped a hard denylist.

How do I inventory every certificate chain across all my Kubernetes clusters?

Run six read-only checks per cluster: issuers, Certificates, TLS Secrets, CA bundles in ConfigMaps, webhook caBundles, and the chains your endpoints actually serve. Loop them over every kube context and write the output to a file per cluster. Everything below uses kubectl, jq, openssl and cmctl.

Start with a context loop. Every later snippet runs inside it:

for ctx in $(kubectl config get-contexts -o name); do
  echo "=== $ctx ==="
  export KUBECONTEXT="$ctx"
  # paste the checks below, using: kubectl --context "$KUBECONTEXT" ...
done

1. Which issuers point at eMudhra?

List every Issuer and ClusterIssuer with its backend. Any ACME server URL on an eMudhra or emSign host is your first finding:

kubectl get clusterissuers,issuers -A -o json | jq -r '
  .items[] | [.kind, (.metadata.namespace // "-"), .metadata.name,
    (if .spec.acme then "acme " + .spec.acme.server
     elif .spec.ca then "ca " + .spec.ca.secretName
     elif .spec.vault then "vault " + .spec.vault.server
     else (.spec | keys | join(",")) end)] | @tsv'

eMudhra's ACME directory URLs are issued per account and profile through its CERTInext console (emSign docs), so match on the hostname rather than a fixed URL.

2. Which Certificates depend on those issuers?

kubectl get certificates -A -o json | jq -r '
  .items[] | [.metadata.namespace, .metadata.name,
    .spec.issuerRef.kind + "/" + .spec.issuerRef.name,
    .spec.secretName, (.status.notAfter // "-"),
    ([.status.conditions[]? | select(.type=="Ready") | .status][0] // "-")] | @tsv'

For any suspicious one, cmctl status certificate <name> -n <namespace> shows the CertificateRequests, events and the Secret it writes.

3. Which TLS Secrets carry an emSign chain, including ones cert-manager never touched?

Hand-made Secrets, Helm-chart defaults and certificates copied from another system are where incidents hide. This scans the full chain in tls.crt and the optional ca.crt of every kubernetes.io/tls Secret:

kubectl get secrets -A --field-selector type=kubernetes.io/tls -o json \
| jq -r '.items[] | [.metadata.namespace, .metadata.name,
         (.data["tls.crt"] // ""), (.data["ca.crt"] // "")] | @tsv' \
| while IFS=$'\t' read -r ns name crt ca; do
    for blob in "$crt" "$ca"; do
      [ -z "$blob" ] && continue
      echo "$blob" | base64 -d \
      | openssl crl2pkcs7 -nocrl -certfile /dev/stdin 2>/dev/null \
      | openssl pkcs7 -print_certs -noout \
      | grep -qiE 'emsign|emudhra' && echo "HIT secret $ns/$name"
    done
  done

cmctl inspect secret <name> --namespace <ns> gives a readable view of a single hit, including validity and issuer.

4. Which ConfigMaps hold a CA bundle that still trusts emSign?

trust-manager targets, Helm values that embed CA bundles, and Crossplane's registryCaBundleConfig all land in ConfigMaps:

kubectl get configmaps -A -o json | jq -c '
  .items[] | {ns: .metadata.namespace, name: .metadata.name,
              pem: ([(.data // {})[] | select(contains("BEGIN CERTIFICATE"))] | join("\n"))}
  | select(.pem != "")' \
| while read -r cm; do
    jq -r .pem <<<"$cm" \
    | openssl crl2pkcs7 -nocrl -certfile /dev/stdin 2>/dev/null \
    | openssl pkcs7 -print_certs -noout \
    | grep -qi 'emSign ECC Root CA - G3' \
    && echo "HIT configmap $(jq -r '.ns + "/" + .name' <<<"$cm")"
  done

Expect kube-root-ca.crt to show up in every namespace with the cluster CA only. That one is fine.

5. What CA does each webhook and APIService trust?

Webhook caBundle fields tell the API server which CA to trust when it calls your admission webhooks, CRD conversion webhooks and aggregated APIs. They should all be private CAs. This prints the subject of every bundle so you can confirm it:

{
  for kind in validatingwebhookconfigurations mutatingwebhookconfigurations; do
    kubectl get "$kind" -o json | jq -r --arg k "$kind" '
      .items[] | .metadata.name as $n | .webhooks[]?
      | select(.clientConfig.caBundle) | [$k, $n, .clientConfig.caBundle] | @tsv'
  done
  kubectl get crd -o json | jq -r '.items[]
    | select(.spec.conversion.webhook.clientConfig.caBundle)
    | ["crd", .metadata.name, .spec.conversion.webhook.clientConfig.caBundle] | @tsv'
  kubectl get apiservices -o json | jq -r '.items[] | select(.spec.caBundle)
    | ["apiservice", .metadata.name, .spec.caBundle] | @tsv'
} | while IFS=$'\t' read -r kind name bundle; do
    subj=$(echo "$bundle" | base64 -d | openssl x509 -noout -subject -enddate 2>/dev/null | tr '\n' ' ')
    echo "$kind $name -> $subj"
  done

You will see cert-manager's webhook CA, Crossplane's crossplane-root-ca, and whatever your policy engine and service mesh use. Also note which ones are managed by cert-manager's CA injector through the cert-manager.io/inject-ca-from, inject-ca-from-secret or inject-apiserver-ca annotations. The injector refreshes those caBundle fields automatically when the source changes. The others are refreshed by their own operators, or by you.

6. What chain do your endpoints actually serve?

Secrets can be correct while a load balancer still serves an old certificate. Check the wire:

kubectl get ingress -A -o jsonpath='{range .items[*]}{range .spec.tls[*]}{range .hosts[*]}{@}{"\n"}{end}{end}{end}' \
| sort -u | while read -r host; do
    chain=$(echo | openssl s_client -connect "$host:443" -servername "$host" -showcerts 2>/dev/null \
      | openssl crl2pkcs7 -nocrl -certfile /dev/stdin 2>/dev/null \
      | openssl pkcs7 -print_certs -noout | grep -i issuer | tail -1)
    echo "$host  $chain"
  done

Repeat for Gateway API listeners and for anything terminated outside the cluster (cloud load balancers with uploaded certificates, CDNs, API gateways).

How do I find workloads whose trust store still trusts the compromised chain?

Check three things per image: the CA bundle file, the Go toolchain version of Go binaries, and language-specific bundles (certifi, the JDK cacerts). Then confirm with a live test from inside the pod against a test host that serves a certificate from one of the compromised intermediates.

List the images running in a cluster:

kubectl get pods -A -o jsonpath='{range .items[*]}{range .spec.containers[*]}{.image}{"\n"}{end}{end}' | sort -u

For each image, extract the system bundle and look for the G3 root:

cid=$(docker create "$IMAGE")
docker cp -L "$cid":/etc/ssl/certs/ca-certificates.crt ./bundle.pem && docker rm "$cid" >/dev/null
openssl crl2pkcs7 -nocrl -certfile bundle.pem | openssl pkcs7 -print_certs -noout \
  | grep -i 'emSign ECC Root CA - G3' && echo "still trusts G3: $IMAGE"

Then the language layers:

Finally, the live test. The researcher's site, linked from the cert-manager trust-manager docs, serves certificates from the compromised intermediates. A certificate error means the client is protected; a successful response means it still trusts the chain:

kubectl exec -n "$NS" deploy/"$APP" -- \
  curl -sS -o /dev/null https://dv-g3b-emudhra.we-have-a-problem.com/ \
  && echo "VULNERABLE: $NS/$APP" || echo "protected: $NS/$APP"

This tests the image's curl and system bundle, not your Go or Java binary. For those, the version checks above are the real answer.

What is the safe rotation order for certificates and trust bundles?

Inventory, stop new eMudhra issuance, re-issue leaf certificates, then push patched trust stores and images, then verify. The normal rule for any CA rotation is "add the new trust, move the leaves, remove the old trust." With a compromised root you skip the wait on the last step: G3 leaves are already untrusted by browsers, so removing G3 from your bundles breaks nothing that is not already broken.

A few details that save hours:

• Step 2 can hit CA rate limits. If you renew hundreds of Certificates against a public ACME CA at once, stage it by namespace or label. Check your new CA's published rate limits first.
• Changing issuerRef on a Certificate triggers re-issuance; changing the Issuer's own spec does not. That is why cmctl renew is in step 2.
• Watch the CertificateRequests too. kubectl get certificaterequests -A shows denied or failed orders right away.

How do I fix trust-manager bundles?

Upgrade trust-manager so useDefaultCAs: true resolves to a CA package without the G3 root. The trust-manager docs note that the eMudhra issue "led to an emergency distrust of a CA certificate", that trust-manager exists to provide "a single place to update trusted CA certs across a cluster", and that the "trust-manager v0.26.0 Helm chart installs a fixed package by default." The recommended fixed package is quay.io/jetstack/trust-pkg-debian-trixie:20250419-deb13u1.0 or newer, set through .defaultPackageImage.repository and .defaultPackageImage.tag if you pin it.

helm upgrade trust-manager oci://quay.io/jetstack/charts/trust-manager \
  --install --namespace cert-manager --wait

Then check that each Bundle target was rewritten (kubectl get bundles) and that consuming pods reload it. Most applications read their CA file at startup, so a rollout restart of consumers is part of this step. If you also publish JKS or PKCS#12 targets through additionalFormats, Java pods get the same fix.

What about Crossplane specifically?

For Crossplane, upgrade providers and core to releases built on a patched Go toolchain, and check any custom CA bundle you configured. Do not rotate the Crossplane webhook CA for this incident; it is self-signed and unrelated.

• Providers. Each provider pod is a Go binary with its own image trust store. Read the release notes of the provider versions you run and move to builds whose Go version is at or above 1.26.10 or 1.27.3. Update the Provider package reference and let the revision roll.
• Package pulls. If you set registryCaBundleConfig in the Crossplane Helm chart to fetch packages from a private registry, the referenced ConfigMap is a CA bundle. Check 4 above covers it. If it is a copy of a system bundle, regenerate it.
• Webhook CA (only if you rotate for another reason). The Crossplane initializer only generates a new CA when the crossplane-root-ca Secret is missing or incomplete, and leaves client and server Secrets alone while they hold data. A rotation therefore means removing all of them together and restarting the Crossplane pod so its init container regenerates them and re-injects caBundle. Test that in staging first.

For cert-manager itself, the same logic applies: upgrade to a release built with a patched Go toolchain, because the controller makes outbound TLS calls to your ACME server.

Which tools help audit certificate chains across a Kubernetes fleet?

No single tool covers the whole job. cmctl and trust-manager cover cert-manager's world, image scanners cover base images and Go versions, CT monitoring covers what was issued for your domains, and a fleet platform covers rolling the fix everywhere. Here is how they split:

How do I run this rotation across many clusters without a platform team?

Turn the fix into a redeploy. The commands above are the easy part. The slow part is getting rebuilt images and new node images onto 15 clusters in three clouds, with an audit trail, without someone SSH-ing into anything.

This is where I am obviously biased, because it is what we built. At Qovery, we operate Kubernetes in your own AWS, GCP, Azure or Scaleway account, or on clusters you already run. The control plane is ours, the data plane stays yours. For an incident like this one, that means:

• Managed cluster upgrades. Node image updates roll through Qovery-managed clusters as one action per cluster, so the node CA bundle fix does not depend on someone remembering every node pool.
• Fleet-wide redeploys from git. Once CI rebuilds your images on patched base images and Go, every environment that tracks that branch redeploys. Environment cloning lets you test the rotation on a copy of production first.
• cert-manager with Let's Encrypt by default. Qovery-managed clusters issue service domain certificates through cert-manager and Let's Encrypt, so they were never on the eMudhra hierarchy. You can still use your own ClusterIssuer when you need one.
• Audit logs and RBAC. Every redeploy and configuration change is attributed, which is what your post-incident review will ask for.

Qovery does not scan your images or certificates, and it does not replace cmctl or trust-manager. You still run the inventory above. We have upgraded hundreds of clusters this way, and incident patches go fastest on fleets where upgrades are already routine. More on that in how we upgrade hundreds of Kubernetes clusters, and on a similar "patch the fleet in 24 hours" exercise in our ingress-nginx CVE mitigation runbook. If mTLS between services is part of your rotation, our piece on Kubernetes mTLS from manual logic to ambient mesh covers where those certificates live.

How do I do this with Qovery from Claude Code or Cursor?

If you want a coding agent to help you redeploy and check environments across clusters, these are the only commands to run:

curl -fsSL https://skill.qovery.com/install.sh | bash

# 2. Or for the current project only
curl -fsSL https://skill.qovery.com/install.sh | bash -s -- --project

# 3. Qovery MCP Server in Claude Code, OAuth, read-only by default
claude mcp add --transport http qovery https://mcp.qovery.com/mcp --callback-port 4242

# 4. Write access, only when you explicitly want it (also enable it in the Qovery Console)
claude mcp add --transport http qovery "https://mcp.qovery.com/mcp?read_write=true" --callback-port 4242

Commands 1 or 2 install the skills (qovery-onboard, qovery-deploy, qovery-troubleshoot, qovery-optimize, qovery-speedup, qovery-preview, qovery-terraform), and the agent asks you to approve each action. Command 3 is read-only through an OAuth login, which is enough to list clusters, environments and deployment status. Use command 4 only when you want the agent to trigger redeploys. Then ask, for example: "list every environment across my clusters and redeploy the ones running images built before 20 October." Start from the AI agent quickstart or the Claude Code integration.