[CI] VKE deploy intermittently times out during Helm create / API access

[khalildai]

Problem

The VKE release deploy workflow is intermittently failing against the Kubernetes API while the same workflow and repository head can succeed on adjacent releases. The failures occur after GitHub-hosted runner setup and chart-repository update, so this is not a self-hosted runner problem.

Recent evidence:

• v2.2.80 run 30082329563: helm upgrade --install failed after about 64 seconds with create: failed to create: Timeout: request did not complete within requested timeout - context deadline exceeded.
• v2.2.77 run 29898787812: Helm create failed with server was unable to return a response in the time allotted, but may still be processing the request (post secrets).
• v2.2.74 run 29723466547: Kubernetes /version request timed out while connecting to the VKE endpoint.
• v2.2.78 run 30057246254 succeeded, showing the path is intermittent rather than permanently misconfigured. v2.2.79 was cancelled and is not evidence of success or failure.

The v2.2.80 Cloud Installer/Upgrade Test succeeded separately; that does not establish that the VKE environment received the release.

Why this needs a durable fix

Blindly rerunning helm upgrade --install after an ambiguous create timeout is unsafe: the API server explicitly says it may still be processing the request. The workflow currently reports a red deploy but does not publish enough reconciliation evidence to distinguish “nothing applied” from “release partially/fully applied but response lost.”

Acceptance criteria

• Add a bounded preflight for VKE API reachability before mutation, with an explicit diagnostic for DNS/TCP/TLS/API timeout stage.
• On an ambiguous Helm timeout, query helm status/history and the relevant release Secret/objects before deciding whether a retry is safe.
• Never retry a mutating Helm operation until the prior attempt is proven absent or reconciled.
• Publish a concise receipt containing target environment, chart/app version, Helm revision/state, and the final safe decision (succeeded, confirmed absent, or manual reconciliation required) without exposing kubeconfig or credentials.
• Demonstrate at least two consecutive fresh VKE deploy successes; include one run from a cold GitHub-hosted runner.

No rerun, deploy, kubeconfig, secret, or cluster mutation was performed while filing this issue.

[khalildai]

2026-07-27 13:45 SGT patrol adds a fresh recurrence on v2.2.81:

• run: https://github.com/apecloud/apecloud-cd/actions/runs/30237219600
• deploy job: https://github.com/apecloud/apecloud-cd/actions/runs/30237219600/job/89887207517
• GitHub-hosted runner setup, VKE context configuration, private chart-repository add/update, and chart resolution all completed.
• helm upgrade --install kb-cloud ... --version v2.2.81 --timeout 10m failed after about 78 seconds with UPGRADE FAILED: create: failed to create: Timeout: request did not complete within requested timeout - context deadline exceeded.
• The parallel upgrade-addon job continued for about 20 minutes and was then cancelled; it does not reconcile the ambiguous kb-cloud Helm create state.

This exactly matches the canonical issue: a mutating create request may still have been processed even though the response timed out. Therefore no rerun or second deploy was issued. The next action remains read-only/manual reconciliation (helm status/history plus release Secret/objects) before any mutating retry; the current workflow still emits no receipt proving confirmed absent or succeeded.

[khalildai]

2026-07-28 15:49 SGT patrol adds a fresh recurrence on v2.2.84:

• run: https://github.com/apecloud/apecloud-cd/actions/runs/30335872463
• deploy job: https://github.com/apecloud/apecloud-cd/actions/runs/30335872463/job/90200502314
• GitHub-hosted runner setup, VKE context configuration, chart repository access, and chart resolution completed before the mutating step.
• helm upgrade --install kb-cloud kubeblocks-enterprise/kubeblocks-cloud --version v2.2.84 ... --timeout 10m failed after about 62 seconds with UPGRADE FAILED: create: failed to create: Timeout: request did not complete within requested timeout - context deadline exceeded.

This is the same ambiguous Helm create signature as the v2.2.81 recurrence: the API response timed out, but the create may still have been processed. Therefore this patrol did not rerun or replay the deployment and did not access the cluster. The required next action remains read-only reconciliation (helm status/history plus release Secret/owned objects) before any further mutation; the workflow still lacks a receipt that distinguishes confirmed absence from successful creation.

[khalildai]

2026-07-29 09:51:55–11:52:25 UTC patrol adds a fresh recurrence on v2.2.86, with two attempts on the same run:

• run 30443787547, exact e60c2253d9111e047e58faa4c0e4456c098ead70
• attempt 1 failed after chart repo access succeeded: helm upgrade --install ... --version v2.2.86 ... --timeout 10m returned create: failed to create: Timeout: request did not complete within requested timeout - context deadline exceeded after about 66 seconds
• an external actor reran it; attempt 2 reached the same Helm command and failed with the same ambiguous create timeout after about 83 seconds

The repeated attempt does not establish safe absence or rollback; either API request may have been processed despite the response timeout. Patrol did not issue either attempt, access the cluster, or replay deployment. The required next action remains read-only helm status/history plus release Secret/owned-object reconciliation before any further mutation, and the workflow still needs a receipt that distinguishes confirmed absence from successful creation.

[khalildai]

2026-07-31 05:59:03–07:59:19 UTC patrol adds a fresh recurrence on v2.2.89:

• run 30613317984, exact ab5a160ac1f18f873ed7520a3a6118bbe6beef39
• deploy job 91100802846
• GitHub-hosted runner setup, VKE context, private chart repository access, and chart resolution succeeded
• helm upgrade --install kb-cloud ... --version v2.2.89 ... --timeout 10m then failed after about 78 seconds with create: failed to create: Timeout: request did not complete within requested timeout - context deadline exceeded

This is the same ambiguous mutating-create signature: the API response timed out without proving whether the create was processed. Patrol did not rerun, redeploy, or access the cluster. Read-only helm status/history plus release Secret/owned-object reconciliation is still required before any further mutation.