Add a modified-only dispatch input to the PR review evaluation

[Wenjie Fan (gggdttt)]

Summary

Adds an opt-in modified-only dispatch input to pr-review-evaluation.yml and wires it to the
get-entries reusable workflow, which already implements --modified-only.

This is the entry-selection half of the closed PR #849, resubmitted on its own. The other half of
#849 — an engine-ref input that overrode the pinned engine at run time — is deliberately not
included and should stay out: a run-time override lets the recorded pin and the executed pin
diverge, which is the exact property the BC-Bench -> BC-ALAgents -> BCQuality chain exists to
protect, and which PR #851 is hardening by exporting the pinned identity. Moving the engine pin
remains a pull request that edits install-agent-harnesses/action.yml.

Entry selection carries no such hazard. bcbench dataset list --modified-only already exists
(src/bcbench/commands/dataset.py L31-52) and get-entries.yml already exposes the flag
(L8, L49, L63-64); only pr-review-evaluation.yml never passed it through, so the code-review
category had no cheap way to evaluate just the entries a pull request touches.

Motivation: the offline eval gate for self-improvement pull requests (ADO 648887, FP3/FP4) needs to
score the handful of gold answers a candidate adds, not all 144 entries.

Changes

• modified-only dispatch input, default: false. Existing dispatches are byte-for-byte unaffected.
• Passed through to get-entries, where it takes precedence over test-run (existing behaviour of
  that workflow, unchanged).
• Added to the requeue workflow-inputs JSON. Without this, a modified-only run with
  repeat > 1 would silently re-run the full corpus on every requeued pass, because requeue
  rebuilds the dispatch from that JSON and would have dropped the flag.
• Concurrency group gains a modified bucket. A gate run sets test-run: false, so it would
  otherwise land in the full group behind a full-corpus run with cancel-in-progress: false —
  a gate that queues behind a 144-entry run is not usable as a gate. Runs that do not set
  modified-only keep their previous group exactly (test or full).

Notes

• No version bump. The input is opt-in and default-false, so no existing run's result changes.
  This matches the convention for workflow-only changes (Add PR review performance telemetry and dashboards #807, Update model selection for nl2al/bcal #826, Add data-query category: AL query-generation benchmark #740, Add extensibility request advisor evaluation #814, Pin and persist LLM judge models #788, Add skill for agent harness version bump #789);
  CONTRIBUTING.md reserves a minor bump for tooling updates that may affect results.
• No conflict with Add advanced code review metrics view #851. That PR edits the evaluate-with-pr-review outputs: block and the
  summarize-results with: block; this PR edits the dispatch inputs, the concurrency group, the
  get-entries with: block, and the requeue JSON. The hunks are disjoint.

Validation

• uv run ruff format / uv run ruff check on the touched test file — clean.
• uv run ty check . --ignore=unresolved-import --exclude "notebooks/" — 1 diagnostic, pre-existing
  on main (src/bcbench/redteam.py:150, unused ty: ignore); unrelated to this change.
• uv run pytest -q -m "not e2e" — 875 passed, 2 skipped, 1 deselected.
• yaml.safe_load of the workflow asserts the file still parses and confirms the resolved input
  list, concurrency expression, and get-entries inputs.
• New test test_pr_review_workflow_propagates_modified_only pins both wiring points, including the
  requeue JSON, so the full-corpus regression above cannot silently return.
• git diff --stat — 2 files, 18 insertions, 3 deletions; no formatter noise.

[Sun Haoran (haoranpb)]

Search for all places where we use input.test-run, because a modified only run is probably also a test job.

pin-commit should also consume inputs.modified-only, otherwise it will leave behind an ephemeral tag

Similarly, summarize-results

[Wenjie Fan (gggdttt)]

Thanks — the sweep found a worse problem than the leftover tag, and you were right that a modified-only run is essentially a test job.

summarize-results (the serious one)

mock gates three things in summarize-results.yml, not just artifact retention:

• --use-capi ${{ !inputs.mock && '--storage braintrust --storage kusto' || '' }}
• Update leaderboard in a new branch, if: ${{ !inputs.mock && !inputs.skip-leaderboard }}
• the evaluation-summary retention days

With mock: ${{ inputs.test-run }} a modified-only run had mock=false, so a two-entry run would have been published to Braintrust and Kusto and committed to docs/_data/code-review.json as a benchmark result for that model. That is corruption of the record, not just noise. Now:

mock: ${{ inputs.test-run || inputs.modified-only }}

The evaluation-summary artifact is still uploaded under mock, so consumers that read the summary in-run are unaffected.

pin-commit and requeue

Rather than only suppressing the tag, I disabled the requeue path outright — per your other comment, one run is the point of a modified-only dispatch:

requeue:
  if: ${{ !cancelled() && !failure() && !inputs.test-run && !inputs.modified-only }}

pin-commit:
  test-run: ${{ inputs.test-run || inputs.modified-only }}

These two go together: with requeue skipped, nothing would ever reach cleanup-ephemeral-tag, so pin-commit must not create the tag in the first place. That also removes the exit 1 ("No ephemeral tag is available for requeue") that suppressing the tag alone would have caused on repeat > 1.

Rest of the sweep

The remaining inputs.test-run sites in this workflow:

• retention-days on the evaluation results artifact — changed, a partial run is short-lived.
• mock — changed, above.
• get-entries test-run: — left alone; get-entries.yml already gives modified-only precedence.
• concurrency group — already a separate modified group in the first commit, so a gate run does not queue behind a full-corpus run.

dataset-validation.yml is the existing precedent and needs nothing: it uses modified-only with no summarize or publish step at all, which is the same "verify, don't publish" shape this now has.

The input description records the semantics, and test_pr_review_workflow_treats_modified_only_as_a_partial_run pins all four wiring points.

120d082a, 631f3d9d. 876 passed, 2 skipped.

[Sun Haoran (haoranpb)]

Good addition!

[Sun Haoran (haoranpb)]

[nit] this looks a bit wired when description is long