Bug description
Restoring a sharded cluster deadlocks at provisioning: the restore cluster's sharding account secret is created and then deletion-stamped ~2 seconds later while still holding the cluster finalizer, leaving it permanently Terminating. The PVC reconciler then loops forever on the terminating secret, restore pods/PVCs stay Pending, and the restore Cluster never leaves Creating.
Observed on a control plane where BOTH kubeblocks (manager) and kubeblocks-dataprotection are built from the same commit 34e474cf62a8cc3b8a497eadb4f2bb6562eb6865 (PR #10568 head; contains #10360/#10469-era main base).
Key object evidence (secret vlk-cluster-37384-restore-shard-default):
creationTimestamp=2026-07-07T03:55:56ZdeletionTimestamp=2026-07-07T03:55:58Z (2s after creation), deletionGracePeriodSeconds=0finalizers=["cluster.kubeblocks.io/finalizer"] — never removed, so the secret hangs Terminating- ownerReference:
apps.kubeblocks.io/v1/Cluster/vlk-cluster-37384-restore (controller=true, blockOwnerDeletion=true)
- labels include
apps.kubeblocks.io/sharding-name=shard
Controller loop evidence (first conflict 2026-07-07T03:55:58.241Z, then continuous):
persistentvolumeclaim controller: object is being deleted: secrets "vlk-cluster-37384-restore-shard-default" already exists- All three restore shard PVCs (
data-...-shard-rmf-0 / -4dg-0 / -twp-0) report ReconcileFailed with the same error; pods/PVCs Pending indefinitely (>13min at capture).
Impact beyond the hang itself: the restore workload's prepareData stage never runs, so addon-side restore logic (including designed fail-fast refusal paths) is unreachable — the restore neither completes, nor fails, nor can be rejected.
To Reproduce
- Sharded cluster (valkey cluster topology,
shardings[0].name: shard, 3×2) with a completed datafile Backup.
- Create a restore cluster from that backup (same shape).
- Watch the restore cluster: sharding account secret
<restore-cluster>-shard-default gets a deletionTimestamp ~2s after creation while the cluster finalizer is still on it; PVC reconcile loops on the terminating secret; cluster stays Creating forever.
Expected behavior
Whichever side owns the delete (duplicate account reconcile, restore-path cleanup, or sharding-component teardown racing provisioning) must not leave the secret in held-finalizer Terminating during provisioning; and the create path should tolerate/wait out a terminating object instead of hard-looping already exists.
Environment
- manager + dataprotection images:
pr10568-34e474cf6-stella (single-commit control plane; no mixed images)
- Namespace preserved for live inspection:
valkey-ct-pr583-r10-pr10568-dp, cluster vlk-cluster-37384
- Evidence bundle SHA256SUMS:
49314649f382e100cf535df04cd27f89c387e35d246fe9a31b33f29d65f17e4a (secret yaml redacted-metadata, DP/PVC log windows, events, restore object states)
Bug description
Restoring a sharded cluster deadlocks at provisioning: the restore cluster's sharding account secret is created and then deletion-stamped ~2 seconds later while still holding the cluster finalizer, leaving it permanently Terminating. The PVC reconciler then loops forever on the terminating secret, restore pods/PVCs stay Pending, and the restore Cluster never leaves
Creating.Observed on a control plane where BOTH
kubeblocks(manager) andkubeblocks-dataprotectionare built from the same commit34e474cf62a8cc3b8a497eadb4f2bb6562eb6865(PR #10568 head; contains #10360/#10469-era main base).Key object evidence (secret
vlk-cluster-37384-restore-shard-default):creationTimestamp=2026-07-07T03:55:56ZdeletionTimestamp=2026-07-07T03:55:58Z(2s after creation),deletionGracePeriodSeconds=0finalizers=["cluster.kubeblocks.io/finalizer"]— never removed, so the secret hangs Terminatingapps.kubeblocks.io/v1/Cluster/vlk-cluster-37384-restore(controller=true,blockOwnerDeletion=true)apps.kubeblocks.io/sharding-name=shardController loop evidence (first conflict
2026-07-07T03:55:58.241Z, then continuous):persistentvolumeclaimcontroller:object is being deleted: secrets "vlk-cluster-37384-restore-shard-default" already existsdata-...-shard-rmf-0/-4dg-0/-twp-0) reportReconcileFailedwith the same error; pods/PVCs Pending indefinitely (>13min at capture).Impact beyond the hang itself: the restore workload's prepareData stage never runs, so addon-side restore logic (including designed fail-fast refusal paths) is unreachable — the restore neither completes, nor fails, nor can be rejected.
To Reproduce
shardings[0].name: shard, 3×2) with a completed datafile Backup.<restore-cluster>-shard-defaultgets a deletionTimestamp ~2s after creation while the cluster finalizer is still on it; PVC reconcile loops on the terminating secret; cluster staysCreatingforever.Expected behavior
Whichever side owns the delete (duplicate account reconcile, restore-path cleanup, or sharding-component teardown racing provisioning) must not leave the secret in held-finalizer Terminating during provisioning; and the create path should tolerate/wait out a terminating object instead of hard-looping
already exists.Environment
pr10568-34e474cf6-stella(single-commit control plane; no mixed images)valkey-ct-pr583-r10-pr10568-dp, clustervlk-cluster-3738449314649f382e100cf535df04cd27f89c387e35d246fe9a31b33f29d65f17e4a(secret yaml redacted-metadata, DP/PVC log windows, events, restore object states)