Repository navigation
Recover a GatewayClass wedged in Terminating - #5292
electricjesus wants to merge 1 commit into
Conversation
Deleting the GatewayAPI CR marks the GatewayClass for deletion through the ownerRef we set on it. Envoy Gateway's gateway-exists finalizer then blocks that delete while any Gateway still uses the class, so it never completes. Envoy Gateway keeps managing a class in that state but skips its parametersRef, because it only resolves the ref when DeletionTimestamp is nil. Gateway Deployments fall back to the upstream Envoy image with no imagePullSecrets, and the class is still reported Accepted=True, so nothing surfaces the fault. Re-creating the CR restores the EnvoyProxy but cannot clear a DeletionTimestamp. Watch GatewayClass, clear the finalizers on a terminating one so the delete can finish, and report it on TigeraStatus rather than healing silently.
There was a problem hiding this comment.
Pull request overview
Fixes a Calico Enterprise Gateway API edge case where a GatewayClass can become stuck in Terminating (blocked by Envoy Gateway’s finalizer), causing Envoy Gateway to skip parametersRef resolution and potentially roll gateway Deployments back to an upstream Envoy image without expected pull secrets.
Changes:
- Adds a lazy watch on
GatewayClassresources so the controller reconciles when a class becomes wedged inTerminating. - Introduces reconcile logic to “unwedge” terminating GatewayClasses by clearing finalizers and surfacing the repair via TigeraStatus.
- Adds a unit test covering the wedged-Terminating repair path and its status reporting.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| pkg/controller/gatewayapi/gatewayapi_controller.go | Adds GatewayClass watch wiring and implements the “unwedge terminating GatewayClass” repair logic surfaced through status. |
| pkg/controller/gatewayapi/gatewayapi_controller_test.go | Adds a unit test validating that a terminating GatewayClass is released and the repair is reported via SetDegraded. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| blocking := 0 | ||
| for j := range gateways { | ||
| if string(gateways[j].Spec.GatewayClassName) == name { | ||
| blocking++ | ||
| } | ||
| } | ||
|
|
||
| reqLogger.Info("GatewayClass is stuck terminating; clearing finalizers so it can be re-created", | ||
| "gatewayClass", name, "deletionTimestamp", gc.DeletionTimestamp, | ||
| "finalizers", gc.Finalizers, "blockingGateways", blocking) | ||
|
|
||
| patched := gc.DeepCopy() | ||
| patched.Finalizers = nil | ||
| if err := r.client.Patch(ctx, patched, client.MergeFrom(gc)); err != nil { |
|
Heads up - operator development on master has moved into the Calico monorepo at projectcalico/calico, under The move shifted every file down a directory and changed the import prefix, so a plain cherry-pick won't apply. This gets most of the way: Anything the PR touched outside the operator tree ( |
Description
Deleting the
GatewayAPICR marks theGatewayClassfor deletion, since we set a controller ownerRef on it. Envoy Gateway'sgateway-exists-finalizerblocks that delete while any Gateway still uses the class, so it never finishes and nothing repairs it.Envoy Gateway keeps managing a terminating class but skips its
parametersRef, because it only resolves the ref whenDeletionTimestampis nil (internal/provider/kubernetes/controller.go:366, EG v1.8.3). Gateway deployments fall back to the upstreamdocker.io/envoyproxy/envoyimage with noimagePullSecrets, and the class still reportsAccepted=True. Re-creating the CR restores theEnvoyProxybut can't clear aDeletionTimestamp, so the cluster stays wrong until someone clears the finalizer by hand.The gateway pod carries on serving and tigerastatus stays green, so this shows up later as a pod that won't start after a restart or drain.
This watches
GatewayClass, clears the finalizers on a terminating one, and reports the repair on tigerastatus. The next reconcile re-creates the class and Envoy Gateway resolvesparametersRefnormally.Clearing another controller's finalizer is deliberate. The wedge only exists because our ownerRef marked a class that Envoy Gateway won't release, and there's no other way out of it. Gateways lose their managed class for a few seconds while the class is re-created, so the proxy deployment is torn down and re-provisioned.
Test plan
go build,go vet,gofmtandgo test ./pkg/controller/gatewayapi/...all pass.New unit test covers a class with
DeletionTimestampset, and asserts both the release and the tigerastatus message. It fails when the repair is short-circuited.Checked by hand on a GCP kubeadm cluster running Enterprise master. Waiting, restarting envoy-gateway, annotating the Gateway and deleting the deployment all left it wedged. Clearing the finalizer and reconciling fixed it in 6 seconds.
Not covered: no e2e, and the upstream skip in EG is unchanged since v0.5.0 and still on EG main, so this only handles our side of it.
Release Note
For PR author
make gen-filesmake gen-versions