chore(e2e): added an e2e testing setup#208
Open
derekmisler wants to merge 4 commits into
Open
Conversation
derekmisler
force-pushed
the
feat/e2e-reviewer-testing
branch
from
022fe66 to
fbc73f9
Compare
Signed-off-by: Derek Misler <derek.misler@docker.com>
derekmisler
force-pushed
the
feat/e2e-reviewer-testing
branch
from
fbc73f9 to
e306e79
Compare
derekmisler
changed the title
aheritier
previously approved these changes
May 12, 2026
Contributor
docker-agent
left a comment
docker-agent
left a comment
There was a problem hiding this comment.
Assessment: 🔴 CRITICAL
This PR adds comprehensive E2E testing infrastructure for the mention-reply pipeline. The structure is sound and the approach is correct, but there are two high-severity bugs that break the tests' correctness guarantees, plus a medium-severity design issue that causes every automated run to post to the wrong PR.
Fix 1 & 2: Replace invalid 'gh api --jq --argjson' with pipe to jq - In test-mention-reply-inline 'Verify inline reply' step: pipe gh api output to 'jq --argjson' instead of using invalid '--jq --argjson' flag combo - Same fix in 'Cleanup anchor comment and thread replies' step Fix 3: Replace hardcoded PR docker#207 with dynamic TEST_PR_NUMBER - Add job-level 'if: github.event_name == pull_request' guard to both test-mention-reply-toplevel and test-mention-reply-inline jobs (prevents running on push-to-main where no PR number exists) - Add job-level 'env: TEST_PR_NUMBER: ${{ github.event.pull_request.number }}' to both jobs - Update all synthetic event JSON, gh api endpoint URLs, and assertions to use $TEST_PR_NUMBER - Remove hardcoded default '207' from test-e2e-reviewer.yml workflow_dispatch input Fix 4: Revert self-review-pr.yml local ref back to pinned SHA - Restore 'uses: docker/cagent-action@f208610 # v1.5.3' - Local './.github/workflows/review-pr.yml' ref is resolved from default branch (main), not the PR branch, so it doesn't achieve dogfooding
Contributor
docker-agent
left a comment
docker-agent
left a comment
There was a problem hiding this comment.
Assessment: 🟡 NEEDS ATTENTION
3 medium-severity issues found in the new E2E workflow additions: one missing cleanup step (orphan PR comments on every manual run) and two missing GitHub token permission declarations.
- Add `issues: write` to test-mention-reply-toplevel job (test-e2e.yml) - Add `pull-requests: write` to test-mention-reply-inline job (test-e2e.yml) - Add cleanup step to inline-mention job (test-e2e-reviewer.yml) to delete the anchor PR review comment and its thread replies on every manual run, preventing orphan comments from accumulating
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds comprehensive E2E test coverage for both top-level and inline mention scenarios.
Changes
test-e2e.ymlthat validate the mention-reply handler correctly extracts and outputs metadata for bothissue_comment(top-level) andpull_request_review_comment(inline) eventstest-e2e-reviewer.ymlfor ad-hoc testing of full review, top-level mention, and inline mention scenarios against any PR./.github/workflows/review-pr.yml)Test plan
is-inline=falseandin-reply-to-id=""is-inline=trueandin-reply-to-id=<comment-id>