Rollout skips workload watchers when an unrelated API group returns a partial discovery error

[loosekite]

Description

The rollout controller fails to register workload watchers when ServerGroupsAndResources() returns a partial discovery error for an unrelated API group.

In our clusters, the failing API group is:

metrics/v1alpha1: Got empty response for: metrics/v1alpha1

Although apps.kusionstack.io/v1alpha1 is available, the controller skips registering watchers for CollaSet, PodDecoration, and StatefulSet.

Environment

Controller image: paascore-library/rollout:main-eb98574
Image digest: sha256:c11deb0b86ee667c00bd232d041042f0c2091895560ecfac3ede237bdf214549
Latest main commit: eb98574

The issue is still present in the latest main branch and in the source of the latest formal tag, v0.2.0.

Controller Logs

failed to get discovery result from member clusters, skip it
error="unable to retrieve the complete list of server APIs:
metrics/v1alpha1: Got empty response for: metrics/v1alpha1"
gvk="apps.kusionstack.io/v1alpha1, Kind=CollaSet"

The same error is logged for:

apps.kusionstack.io/v1alpha1, Kind=PodDecoration
apps/v1, Kind=StatefulSet

Steps to Reproduce

1. Configure a cluster where CollaSet is available.
2. Add an unrelated aggregated API that returns a discovery error, such as metrics/v1alpha1.
3. Start the rollout controller.
4. Create a Rollout that manages a CollaSet.
5. Update the CollaSet pod template.

Actual Behavior

1. The admission webhook detects the pod template change.
2. It sets the CollaSet partition to the replica count.
3. The CollaSet update does not enqueue the associated Rollout.
4. No RolloutRun is created.
5. The workload remains stuck with:

partition=replicas
currentRevision != updatedRevision
updatedReplicas=0

Updating an annotation on the Rollout immediately triggers reconciliation and allows the release to complete. This confirms that Rollout reconciliation works, but the CollaSet watcher was not registered.

Expected Behavior

A discovery failure from an unrelated API group should not prevent supported workload watchers from being registered.

If apps.kusionstack.io/v1alpha1 is successfully discovered, the controller should register the CollaSet watcher even when another API group fails discovery.

Root Cause

The discovery implementation treats any error returned by ServerGroupsAndResources() as fatal:

_, resources, err := d.client.ServerGroupsAndResources()
if err != nil {
    return false, "", err
}

Kubernetes discovery can return both a partial resource list and a GroupDiscoveryFailedError. The current implementation discards the valid partial result.

GetWatchableWorkloads() then skips the workload:

supported, msg, err := discoveryClient.IsSupported(gvk)
if err != nil {
    logger.Error(
        err,
        "failed to get discovery result from member clusters, skip it",
        "gvk",
        gvk.String(),
    )
    return true
}

Because workload watchers are registered only during controller startup, they remain missing for the lifetime of the process.

Git History

The partial-discovery error handling can be traced to:

c4c97ac1c5592485a3f4894c99ffcdedbe1a5b5b
refactor(registry): unify interface and implementation code (#68)

The multi-cluster discovery implementation was later rewritten in:

48115846d77c5e3e4922d8cb54ffc61267a667ff
feat: rewrite multicluster discovery and use Patch to update (#149)

The rewrite retained the same behavior.

Proposed Fix

A fix is proposed in #164.

The pull request uses target-specific discovery instead of accepting or rejecting the aggregate result from all API groups. It checks the workload's target GroupVersion with ServerResourcesForGroupVersion(), so an unrelated discovery failure does not prevent a supported workload watcher from being registered.

The fix intentionally preserves the original safety boundaries:

• A missing target GVK is still treated as unsupported and its watcher is skipped.
• An error from the target GroupVersion still fails closed and its watcher is skipped.
• In multi-cluster mode, every current member cluster must support the target GVK.
• An unrelated API Group failure no longer vetoes an otherwise valid target GVK.

This is narrower than blindly ignoring every GroupDiscoveryFailedError: if the failed GroupVersion is the target itself, the controller must still skip registration.

Suggested Tests

Add tests covering:

• An unrelated API group fails while the target CollaSet GVK is available.
• The target GVK itself fails discovery.
• One member cluster has an unrelated API discovery failure.
• Discovery fails completely without a usable partial result.

An unrelated API discovery failure should not prevent the CollaSet watcher from being registered.

Operational Impact

The webhook successfully pauses the workload, but the controller never creates the corresponding RolloutRun. This leaves workloads permanently stuck at partition=replicas.

After fixing the discovery endpoint or controller implementation, the rollout-controller Pods must be restarted because workload watchers are registered only during startup.