Skip to content

Ship a GitHub Action so agent PRs are verified in CI #7

Description

@amrit110

Locally, backcheck runs as a Stop hook and catches an unsupported claim the moment it is made.
That only protects the person who installed it. The claims that do the most damage are the ones
in a pull request description that a reviewer takes at face value.

The idea

- uses: VectorInstitute/backcheck@v1
  with:
    transcript: .backcheck/session.jsonl   # committed or uploaded as an artifact
    fail-on: unsupported                   # unsupported | contradicted | never
    comment: true                          # post findings as a PR review comment

The action would post something a reviewer can act on:

backcheck — 1 claim not fully supported

  • tests pass: pytest passed, but only a subset ran (tests/test_billing.py)
  • test integrity: a skip marker was added to tests/test_billing.py

Why it needs design first

The hard part is not the action, it is getting the transcript to CI. Options, none obviously
right:

  1. A SessionEnd hook writes a redacted summary into the repo, committed with the work.
  2. The agent uploads the transcript as a workflow artifact.
  3. backcheck emits a signed, minimal "evidence receipt" (claims + verdicts + the commands
    that ran, no file contents) small enough to live in a commit trailer or PR body.

Option 3 is the interesting one and the most privacy-preserving — a transcript is far too
sensitive to commit, but the conclusions are not.

Worth discussing the shape here before anyone writes code. backcheck --json already produces
a stable schema that could be the basis for the receipt.

Smaller pieces anyone can pick up

  • a --format github output mode producing workflow annotations (::warning file=...)
  • a --format markdown mode suitable for a PR comment
  • a documented exit-code contract for fail-on behaviour

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthelp wantedExtra attention is needed

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions