Skip to content

Fix: Diagnose merge failures from live state, not the rejection - #485

Merged
tykeal merged 6 commits into
lfreleng-actions:mainfrom
modeseven-lfreleng-actions:fix/merge-failure-diagnosis
Sep 9, 2026
Merged

tykeal merged 6 commits into
lfreleng-actions:mainfrom
modeseven-lfreleng-actions:fix/merge-failure-diagnosis

Conversation

@ModeSevenIndustrialSolutions

@ModeSevenIndustrialSolutions ModeSevenIndustrialSolutions commented Sep 4, 2026 •

Copy link
Copy Markdown
Contributor

Closes #482.

A 157-PR owner-wide run reported 17 failures. Sampling four showed the stated cause was wrong in every case, and two were fully mergeable by the time the run printed them as failed.

A merge rejection is a snapshot. GitHub states what was unsatisfied at the instant the merge was attempted, which routinely names required checks that had merely not finished — and nothing re-read that before the summary printed.

PR Reported as Actually
test-python-project#396 Required workflows failed: Semantic Pull Request 🛠️ clean — nothing wrong with it
docker-workflows#70 Required workflows failed: Semantic Pull Request 🛠️ clean — judged before its rebase landed
python-workflows#84 3 required workflows not satisfied All three pass; pre-commit.ci - pr was the blocker
workflows-template#55 AI Slop Scan 🧹, Zizmor Scan 🌈 pre-commit.ci was the blocker

All four are fixtures in tests/test_merge_failure_diagnosis.py.


What changes

1. A status check no longer hides behind a workflow

_format_failure_reason returned on the first group it found, so a rejection naming both showed only the workflows. Each kind is now listed, with its own verb read from its own clause — workflows that have not finished routinely accompany a context that has already failed.

2. Unfinished is not failed

A PR GitHub would merge now is recorded as UNSETTLED and counted apart from FAILED: one needs another run, the other needs a human. The confirmation step already re-read every reported failure, and the payload it fetches carries mergeable_state, so the reconciliation itself costs no extra request.

Three guards, each earned during review:

  • Only a state-based refusal may be withdrawn. The run also reports FAILED for its own troubles — an exception, a rebase that did not complete, a 502, a missing token scope. Decided from the HTTP status by allowlist (405, 409), so an unrecognised status, or an exception carrying none, keeps its message rather than losing it.
  • Only affirmative mergeability withdraws, matching _state_is_waitable. A closed PR never qualifies.
  • Re-judged after every task finishes, not only when its own does. In an owner-wide run a PR refused at minute two can go green at minute ten; the earlier check alone could not see that. This was the issue's acceptance criterion.

3. Name what is blocking, without over-claiming

Read against the live head — a dependabot rebase moves it, and this tool requests those rebases. Three sources, weighted by what they prove:

Source Carries Proven blocker when…
commit status contexts context name the branch requires it
Actions workflow runs workflow name: GitHub quoted it as required
check runs job name never — reported as failing, not blamed

codeql.yml declares the workflow CodeQL and the job Audit Repository, so matching a rejection against check-run names compares two vocabularies that need not agree.

All three paginate; each reports whether it answered; a proven blocker stands on its own evidence, an unproven one requires every probe to have answered. Everything failing is named exactly once.

4. Retry a run that never reached a verdict

error is pre-commit.ci failing to complete, failure is the hooks reporting a real problem — only the first is retried. The repair gate covers unknown as well as blocked, though a missing status still needs a settled PR. Suppression is scoped to the incident by timestamp, and the poll ignores the pre-trigger reading — otherwise the first poll returns the very error being retried.


Corrections to the issue

Recorded in-thread:

  • Cause 1 is smaller than described — the reconciliation hook already existed; it asked the wrong question, and ran at the wrong time.
  • Cause 3's second half names the wrong state — blocked covers pending required checks; unknown was the gap.
  • Criterion 6 needed no change — _pending_age already uses the status's own timestamp.
  • Cause 4 has a naming trap — MergeStatus.PENDING is the initial non-terminal state.

Scope

Fifteen Copilot rounds produced 36 findings. Many were genuine defects in code this branch added while fixing an earlier one — the stale head, the over-broad refusal classification, the unpaginated status read (that endpoint defaults to 30), and the end-of-run reconciulation. Those are all here.

A cluster of hardening was implemented and then deliberately removed, about 680 lines. It addressed inputs that need a rule name containing our delimiters, a workflow name containing a comma, two workflow files sharing a name:, an app-pinned requirement, or more than a thousand status contexts. The machinery and its tests cost more than the failures they prevent are worth, and none of it reaches the reported cause — which a blocked PR now reads from live API state rather than from the rejection string. rule_violations.py is consequently left close to its original shape (+59 lines, against +171 at the peak).

Also declined, with reasons: rulesets are not enumerated to resolve required workflow names (a ruleset names a workflow by file path while a run carries its name:, and GET /orgs/{org}/rulesets returned 403 throughout the 503-PR run in docs/BULK_RUN_PERFORMANCE_AUDIT.md); stale is not treated as failing (GitHub applies it to a run a later push made irrelevant).

One review finding is correct but pre-existing and out of scope — required checks from rulesets and branch protection are cumulative, and only one is consulted. Filed as #494 rather than folded in: it touches four callers, costs a request per repo/branch, and tightens reliable for all of them.

Validation

Check Result
pytest 2438 passed, 20 skipped (baseline 2317)
basedpyright --warnings src tests 0 errors, 0 warnings
mypy src tests Success, 283 files
ruff check / format --check All checks passed
reuse lint Compliant
aislop 100 / 100, 0 errors, 0 warnings — unchanged against main
prek All hooks passed

Every fix was reverted in isolation and its tests shown to fail. _precommit_ci.py crossed the 400-line gate during review and was split rather than suppressed, following #475's precedent: _precommit_status.py (what pre-commit.ci said) and _precommit_wait.py (reading it, and waiting).

One existing test asserted that a pre-commit.ci error posts no comment, and now asserts the opposite, with the distinction it turns on named in the class docstring.

Also here

#429 closed as completed — verified that shorthand targets and local-checkout defaulting both landed with #475.

A ruleset rejection can name a required workflow and a required status
check at once. The renderer returned on the first group it found, so
the status checks were unreachable whenever a workflow was named too,
and the summary listed conditions that pass while omitting the one
that blocks.

Observed on python-workflows#84: all three reported workflows had
passed, and the sole blocker was the status context
"pre-commit.ci - pr", which never appeared.

Each kind now carries its own verb. A single verb is read from the
outcome wording nearest the quoted names, so on a combined message the
workflows borrowed the status check's "failing" and were reported as
failed when they had merely not finished. Those two call for opposite
responses: one needs a human, the other only needs waiting.

Bullets are labelled only when both kinds appear, so every existing
single-kind report renders exactly as before.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Matthew Watkins <mwatkins@linuxfoundation.org>
A merge rejection states what was unsatisfied at the instant the merge
was attempted, which routinely names required checks that had merely
not finished. Nothing re-read that before the summary printed, so a
pull request that went green moments later was still reported as
failed, under a cause that had stopped being true.

Two of the four failures sampled from a 157-PR run were fully
mergeable by the time the run printed them: test-python-project#396
and docker-workflows#70 were both clean, and the second had been
judged before the rebase it was sent had completed.

The confirmation step already re-reads every about-to-be-reported
failure, and the payload it fetches carries mergeable_state, so
settling this costs no further request. A PR GitHub would merge now is
recorded as UNSETTLED and counted apart from FAILED: one needs another
run, the other needs a human, and reporting both the same way sent
operators looking for causes that did not exist.

An unknown mergeable_state stays a failure. GitHub computes
mergeability in the background, so unknown is an absence of evidence
rather than evidence of a block, but it is equally not evidence the PR
would merge, and only that would justify withdrawing a failure.

UNSETTLED is deliberately neither PENDING, the initial non-terminal
state every result starts in, nor AUTO_MERGE_PENDING, where GitHub
finishes the merge server-side without another run.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Matthew Watkins <mwatkins@linuxfoundation.org>
A rejection message is preferred over state-based inference because it
usually carries the actionable cause. On a ruleset rejection it
frequently does not: it lists the rules GitHub evaluated, including
ones that had merely not finished, and it can omit the condition
actually holding the merge.

python-workflows#84 was reported against three required workflows.
All three had passed. The sole blocker was the status context
"pre-commit.ci - pr", which the message never mentioned.

A blocked PR is now re-read and the blocking set derived from live
state. Two sources are needed, because GitHub reports through two
mechanisms that do not overlap: Actions workflows report as check
runs, while pre-commit.ci, DCO and similar integrations report as
commit status contexts. A view built from check runs alone cannot see
that context at all, which is how the wrong answer was reached.

Status contexts are filtered to those the base branch requires, so an
advisory integration cannot be presented as the reason a merge was
refused. Check runs are deliberately not filtered that way: a required
workflow declared by a ruleset never appears among the required status
contexts, so filtering on that list would discard the names a ruleset
rejection is about.

Only blocked PRs are re-examined. A conflicted, behind or draft PR is
already described accurately by its state, so reading its checks would
spend requests without adding anything.

An empty reading is not an all-clear. When nothing can be established
the recorded reason stands, so a token that cannot read rulesets
leaves the operator no worse off than before.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Matthew Watkins <mwatkins@linuxfoundation.org>
Two gaps left a stalled pre-commit.ci in place, and the context is a
required status check, so the PR stayed blocked either way.

The stuck test accepted only pending. Any other state read as a
reported result, which lumped error in with success and failure. The
commit status API separates "the check failed" from "the checker
failed", and pre-commit.ci uses that separation: failure is the hooks
reporting a genuine problem with the change, error is a run that never
completed -- an upload that 5xx'd, a container that died. The first
reports the same result however often it is re-run. The second
routinely clears on the next attempt, and until it does nobody has
reached a verdict at all.

Age is not consulted for an error. A terminal state does not become
more stuck with time, and there is nothing left to await. A genuine
failure is still left alone, so re-running one cannot cost a comment
and a five-minute wait to be told the same thing.

The repair gate ran only for blocked. That state does cover pending
required checks, so the gap was narrower than it first appears: it was
unknown that was skipped, which is what GitHub reports while it works
mergeability out in the background -- the window a freshly pushed PR
sits in, and exactly the PRs a stalled check repair exists for. Acting
there is safe because the repair's own preconditions never consult the
mergeable state.

Per-PR timing needed no change: the pending age already comes from the
status's own updated_at, not from a run-wide clock.

The existing parametrised test asserted that error posts no comment,
and now asserts the opposite, with the distinction it turns on named
in the class docstring.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Matthew Watkins <mwatkins@linuxfoundation.org>
The run now reports an outcome the README did not describe, and the
right response to it does not follow from the word alone: unsettled
means re-run, where every neighbouring category means either nothing
or investigate.

Lists all seven together rather than documenting the new one in
isolation, since the value lies in the contrast. Records why unsettled
exists at all, so a reader meeting a non-zero count knows it is not a
softened failure, and notes that a failure now names the condition
blocking the merge.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Matthew Watkins <mwatkins@linuxfoundation.org>
@ModeSevenIndustrialSolutions
ModeSevenIndustrialSolutions requested review from a team and a balanced review from Copilot September 4, 2026 10:26
@github-actions github-actions Bot added the bug Something isn't working label Sep 4, 2026

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

Fifteen review rounds; the substantive corrections are grouped by what
they change rather than by which round found them. Several were defects
in code this branch added while fixing an earlier finding.

Reading the wrong commit. The probes took head_sha and base_branch from
the snapshot captured before the merge attempt. A dependabot rebase
moves the head and this tool requests those rebases, so the PR most
likely to reach the diagnosis with a stale head is one the run itself
moved -- docker-workflows#70, an issue fixture, among them. Both now
come from the refreshed payload.

Reconciling too early. The per-PR confirmation fires as each task
completes, which in an owner-wide run is long before anything prints, so
a pull request refused at minute two and clean by minute ten was still
reported failed. That is the issue's own acceptance criterion. Every
failure now gets one last look after all tasks finish, costing one
request per PR still reported as failed, with the tracker tally
corrected alongside so the live counts and the summary agree.

Withdrawing failures that were never verdicts. The run reports FAILED
for its own troubles too: an unhandled exception, a rebase that did not
complete, a 502, a missing token scope. None says anything about
mergeability, so a later clean read must not rewrite them as unsettled
and bury the message explaining what to fix. The classification travels
with the reason, decided from the HTTP status by allowlist -- 405 and
409 -- so an unrecognised status, or an exception carrying none, keeps
its message. Withdrawal also requires affirmative mergeability, matching
_state_is_waitable, and a closed PR never qualifies.

Absence inferred from partial reads. Check runs asked for one page;
the combined-status endpoint defaults to thirty and was asked for no
page size at all. Each feeds a message that replaces the recorded
reason, and each reasons about absence, so a short page read as an
all-clear for whatever sat on page two. Both paginate, as do the issue
comments the duplicate-nudge check reads. A failed probe was likewise
indistinguishable from an empty answer -- the 503-PR audit records
GET /orgs/{org}/rulesets returning 403 -- so each probe now reports
whether it answered, and the required-check lookup exposes the
reliability signal it was previously using only to decide caching. A
proven blocker still stands on its own evidence; an unproven one
requires every probe to have answered.

Blaming the wrong thing. Names are matched case-sensitively, as GitHub
matches them. Required workflows are matched against Actions workflow
*run* names rather than check-run names, which are two namespaces:
codeql.yml declares the workflow CodeQL and the job Audit Repository.
Workflow runs collapse to the latest per name, so a re-run does not
leave a superseded failure to be blamed. FAILING_CONCLUSIONS gains
action_required and startup_failure, both terminal and both blocking;
stale stays out, since GitHub applies it to a run a later push made
irrelevant. Everything failing is now named exactly once, either as a
blocker or as also-failing.

Retriggering that did not work. An error status sits on the commit
until pre-commit.ci replaces it, so the first poll after a nudge
returned the very failure being retried and ended the wait. The reading
that prompted the trigger is carried into polling. A missing status is
stuck only once GitHub has settled the PR, since on an unsettled one it
cannot be told from one that has not propagated. Suppression is scoped
to the incident by timestamp.

Reporting. get_results_summary exposes the new outcome, in both its
populated and empty shapes, so the counts add up for a programmatic
consumer. The note holding GitHub's rejection is written only when
nothing is noted yet, since the end-of-run pass would otherwise replace
it with our own live reading.

Deliberately not included. Several findings concerned inputs that need
a rule name containing our delimiters, a workflow name containing a
comma, two workflow files sharing a name, or an app-pinned requirement.
Each was implemented and then removed: the hardening cost more in
machinery and tests than the failures it prevented are worth, and none
of them reaches the reported cause, which a blocked PR now reads from
live API state rather than from the rejection string. rule_violations
is left close to its original shape for the same reason. Also declined:
rulesets are not enumerated to resolve required workflow names, since a
ruleset names a workflow by file path while a run carries its name:,
and that endpoint returned 403 throughout the audited run.

Splitting the pre-commit.ci status readers and the poll into their own
modules keeps the file under the size gate. Split rather than
suppressed, along a seam the code already had.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Matthew Watkins <mwatkins@linuxfoundation.org>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Live blocker attribution can be wrong for app-pinned checks and can degrade during final reconciliation.

Review details

Suppressed comments (2)

src/dependamerge/merge_manager/_not_mergeable.py:246

  • The end-of-run pass no longer has the original rejection in result.error: the first diagnosis replaced it with blocked by required workflow: … and preserved the rejection in warning. Parsing the rewritten text yields no required workflow names, so a still-failing workflow is downgraded on the second pass to merely failing checks. Reuse the preserved rejection when present so required-workflow attribution remains stable across reconciliation.
        rejection = result.error or ""

src/dependamerge/merge_manager/_live_blockers.py:190

  • This drops integration_id, so an app-pinned requirement is treated as required by context name alone. A same-named failing status from another GitHub App does not satisfy or block that requirement, but this code reports it as the proven blocker. Since the failing-status probe carries no source identity, exclude app-pinned entries from promotion (as the PR description requires) or propagate app identity through the probe.
        return {
            str(entry.get("context", "")).strip()
            for entry in required
            if isinstance(entry, dict) and entry.get("context")
        }, True
  • Files reviewed: 30/30 changed files
  • Comments generated: 0 new
  • Review effort level: Balanced

@ModeSevenIndustrialSolutions

Copy link
Copy Markdown
Contributor Author

Review follow-ups captured

Fifteen review rounds produced 36 findings. The substantive ones are fixed on this branch; everything declined or deliberately removed now has an issue, so nothing rests only in a resolved thread.

Issue What it holds Why not here
#494 Required checks from rulesets and branch protection are not combined Pre-existing on main; four callers, costs a request per repo/branch, tightens reliable for all of them
#495 Seven low-probability diagnosis imprecisions Implemented then removed — ~680 lines for inputs that essentially do not occur
#496 Recreate dispatch cannot tell a refusal from a failed attempt Needs a change to the recreate contract; this PR is scoped to diagnosis

The removed work in #495 is archived with its tests at archive/merge-diagnosis-hardening (4606bd4), so anything worth reinstating can be lifted rather than rewritten.

Declined outright, not deferred

Three findings are decisions rather than debts, and are recorded in the commit body so they are not re-litigated by a future reviewer:

  • Resolving required workflow names from rulesets — a ruleset names a workflow by file path while a run carries its name:, so joining them means fetching and parsing each file; and GET /orgs/{org}/rulesets returned 403 throughout the 503-PR run in docs/BULK_RUN_PERFORMANCE_AUDIT.md. The rejection message is both cheaper and more reliable.
  • Treating stale as failing — GitHub applies it to a run a later push made irrelevant, so reading it as a failure would let a superseded run block a commit it never examined.
  • Disambiguating an unbalanced quote inside a rule name — recorded as item 1d of Blocker diagnosis is imprecise for pathological names and configurations #495. GitHub escapes neither delimiter, so the input is not reliably parseable by any rule.

A note for whoever reviews next

If Copilot is asked for another pass, it will re-raise most of #495 — those findings were valid, and we judged the machinery not worth it rather than judging them wrong. Worth reading as a recorded trade rather than as regression.

This branch was successfully deployed

1 active deployment
integration — e2543f0f Deployed Sep 8, 2026 by ModeSevenIndustrialSolutions via Live dry-run integration tests #280
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fix: Merge failures are misdiagnosed and transient states reported as failed

3 participants