deploy: capture cloud state on manager ssh fail - #2929
Merged
Conversation
ideaship
marked this pull request as ready for review
July 15, 2026 07:59
ideaship
marked this pull request as draft
July 15, 2026 08:02
Add durable, read-only cloud diagnostics to the manager bring-up rescue
block so that manager bring-up failures are diagnosable from the job log
alone. CI credentials are not otherwise accessible, but the build itself
holds the cloud credentials (OS_CLOUD), so the rescue block queries and
prints the relevant cloud-side state on failure:
- "floating ip show <manager address>": its port_id shows whether the
floating IP is associated to the expected manager port, distinguishing
a control-plane mis-association from a correct association that is
mis-routed at L2 (stale ARP/conntrack).
- "server/port/floating ip list": surfaces leftover or unexpected
resources in the shared project (e.g. leaked managers).
- a live ssh-keyscan fingerprint of the manager address: records which
host actually answered, for comparison with the captured serial
console host keys.
All queries run from terraform_path (as the console-log query does) so
the client finds terraform/clouds.yaml, and each display shows both
stdout and stderr so an auth or API error is visible rather than an
empty result.
The immediate motivation is a recurring failure where the ssh probe
("Wait until ssh public key authentication to the manager works") aborts
with REMOTE HOST IDENTIFICATION HAS CHANGED: the manager booted once
with a stable key and the floating IP reached it correctly early in the
probe, yet the same floating IP reached a different host by the time the
probe gave up. Neither a stale known_hosts entry nor the purge in
PR #2928 explains that; the fault is in floating-IP routing, which these
captures are meant to characterise. The tasks are general enough to keep
long-term: they run only in the rescue path, add no overhead on success,
and cannot affect a passing deploy.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Roger Luethi <luethi@osism.tech>
ideaship
force-pushed
the
deploy-manager-fip-diagnostics
branch
from
July 15, 2026 09:08
36dda18 to
2f55a67
Compare
ideaship
marked this pull request as ready for review
July 15, 2026 09:16
ideaship
requested review from
berendt and
Copilot
and removed request for
Copilot
July 15, 2026 09:17
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds read-only cloud-side diagnostics to the manager bring-up rescue
block in
playbooks/deploy.yml. When manager bring-up fails, the rescue pathnow captures and prints, using the build's own
OS_CLOUDcredentials:openstack floating ip show <manager address>— itsport_idshowswhether the floating IP is associated to the expected manager port
(control-plane) vs a correct association that is mis-routed at L2 (stale
ARP/conntrack).
openstack server/port/floating ip list— leftover or unexpectedresources in the shared project (e.g. leaked managers).
ssh-keyscan … | ssh-keygen -lffingerprint of the manager address— which host actually answered, to compare against the captured
serial-console host keys.
All tasks are read-only and best-effort (
changed_when/failed_when: false),run only in the rescue path (no overhead on a passing deploy), and need no
CI credential access — the evidence lands in the job log.
Why (motivation, not scope)
A recurring failure: the ssh probe "Wait until ssh public key authentication
to the manager works" aborts with
REMOTE HOST IDENTIFICATION HAS CHANGED.Log analysis of two occurrences shows the manager booted once with a
stable key and the floating IP reached it correctly early in the probe, yet
the same floating IP reached a different host by the time the probe
gave up — the FIP is re-routed mid-build to a host that is not the manager
this build booted. Neither a stale
known_hostsentry nor the purge in theearlier fix explains that:
We would rather collect more decisive data before attempting another fix.
The earlier attempt addressed the wrong layer, and guessing again risks doing
the same. The single most decisive datum is the floating IP's
port_idatfailure time (control-plane mis-association vs L2 mis-routing); these
diagnostics exist to capture it — and related state — from the next
occurrence, so any follow-up fix targets the layer that is actually at fault.
They are general enough to keep long-term.
Occurrences (Zuul)
testbed-deploy-next-in-a-nutshell-with-tempest-ubuntu-24.04:https://zuul.services.osism.tech/t/osism/build/af7b9a8f7bf54b7582e4274744a781c4
testbed-update-stable-current-ubuntu-24.04:https://zuul.services.osism.tech/t/osism/build/2e71da7b41e045b4840a261450ba030c
Both ran on orchestrator
zuul-testbed04.🤖 Generated with Claude Code