Skip to content

docs: add a pre-PR consistency audit to AGENTS.md - #58

Open
aknuds1 wants to merge 3 commits into
prometheus:mainfrom
aknuds1:docs/agents-consistency-audit
Open

aknuds1 wants to merge 3 commits into
prometheus:mainfrom
aknuds1:docs/agents-consistency-audit

Conversation

@aknuds1

@aknuds1 aknuds1 commented Nov 14, 2026 •

Copy link
Copy Markdown

Add a check to AGENTS.md to, when opening or updating a PR, not claim Go parity for behaviour one did not verify. Also clarify the instruction about closing any issue, because not every PR will close one.

The revisions stem from reviewing #52. AFAICT, its claim of fixing #33 isn't correct, it only solves part of it. I propose these revisions, to hopefully steer future agents to avoid falling into the same trap.

Go parity claim

#52 says there are “no intentional differences from Go,” and its commit message broadly describes the timestamp behavior as matching Go. However; AFAICT, for an exemplar without a timestamp, honor_timestamps=true, a sample timestamp of 10 seconds, and a configured fallback of 20 seconds, Go gives the exemplar 10 seconds because it uses the resolved sample time. Rust gives it 20 seconds because it uses the configured fallback.  This seems to be a difference predating #52 though.

@aknuds1
aknuds1 marked this pull request as ready for review November 14, 2026 17:42

@bwplotka bwplotka left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

I don't want to block it but I never had any of those problems with my Google models (Gemini 4 argon, carbon hydro).

Anyone has those problems? @nicolastakashi @saswatamcode

Do we have evidences contributors did this mistake at least once in this repo?

@aknuds1
aknuds1 requested a review from bwplotka November 18, 2026 08:23
@aknuds1

aknuds1 commented Nov 18, 2026

Copy link
Copy Markdown
Author

@bwplotka I took a critical look at this PR, and revised it. Are the points clearer now?

@aknuds1

aknuds1 commented Nov 18, 2026 •

Copy link
Copy Markdown
Author

Do we have evidences contributors did this mistake at least once in this repo?

The added/revised points came out of reviewing PRs to this repo, and concrete issues found. I can provide specific evidence if that's desired.

@bwplotka

Copy link
Copy Markdown
Member

Yes please I am curious! What model, what's setup? Your agents?

@aknuds1

aknuds1 commented Nov 18, 2026

Copy link
Copy Markdown
Author

@bwplotka added to the PR description. Agent-wise, I used DeepSeek v4.1 Flash 1017 in Pi for the PR review, but after your initial review, cross-checked my AGENTS.md changes with Codex/Claude for non-biased takes. The latter helped me par down (remove subjectively weak benchmark advice) and clarify.

@bwplotka bwplotka left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, then can you add those rules to your coding rules? They are not specific to the rust repo are those?

I feel AGENTs.md should have repo specific rule not generic coding rules, wdyt?

@aknuds1

aknuds1 commented Nov 19, 2026 •

Copy link
Copy Markdown
Author

Ok, then can you add those rules to your coding rules? They are not specific to the rust repo are those?

@bwplotka I think for that to work, every potential contributor would have to have them. The point is to steer agents better to avoid the pitfalls made in a PR I was reviewing. The idea is to save reviewers' energy. I don't need them in my own AGENTS.md :)

I also think that the current advice of "Link the GitHub issue the PR closes" ought to change at the very least, because not every PR closes an issue. Even if it relates to an issue, it may not actually close it.

Comment thread AGENTS.md Outdated
- Sign off every commit with `git commit -s` (DCO).
- PR titles use `<area>: <summary>`, for example `parse: add OpenMetrics 1.0 parser`.
- Link the GitHub issue the PR closes.
- Link the relevant GitHub issue, if any.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- Link the relevant GitHub issue, if any.
- Link the relevant GitHub issue, if any. Use `Fixes #N` only when the PR meets all of the issue's acceptance criteria, otherwise use `Refs #N`.

@nicolastakashi

Copy link
Copy Markdown
Contributor

I'd drop the tree audit and the generic part of the claims audit, since neither is specific to this repo. The tree audit also wouldn't have changed #52, because its author already found the scrape-manager TODO and listed it as out of scope.

Regarding the Go parity part is specific to this repo, so I'd keep it as one line in "Go is the reference", something like "Do not claim Go parity for behavior you did not verify", wdyt?

@nicolastakashi

Copy link
Copy Markdown
Contributor

Also if we feel we need more determinism from the agents I'd suggest we invest in hooks for the most famous agents e.g. claude, codex, pi, antigravity, opencode so we can make sure that all behave the same way and we can ensure more determinist behavior.

Encode the checks that catch a change which is correct in its diff but wrong in its
claims, its issue links, or its blast radius: a tree-wide grep for the identifiers it
touches, a mapping from every PR-body claim to the line or command that proves it, and a
match between `Fixes #N` and the issue's acceptance criteria.

Add one benchmark rule as well: do not publish a delta whose spread is wider than the
effect being claimed.

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Run the tree, claims, and issue audits before opening or updating a PR. Search old and new identifiers and relevant issue references, verify claims in PR descriptions, commits, and specs, and link issues accurately for both complete and partial work.

Remove the proposed benchmark rule.

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
@aknuds1
aknuds1 force-pushed the docs/agents-consistency-audit branch from 4aeb48b to e27793c Compare November 21, 2026 09:52
@aknuds1

aknuds1 commented Nov 21, 2026

Copy link
Copy Markdown
Author

@nicolastakashi applied your feedback. PTAL.

Also if we feel we need more determinism from the agents I'd suggest we invest in hooks for the most famous agents e.g. claude, codex, pi, antigravity, opencode so we can make sure that all behave the same way and we can ensure more determinist behavior.

I'm not familiar with these hooks. Would you propose one or more of them?

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
@aknuds1
aknuds1 force-pushed the docs/agents-consistency-audit branch from e27793c to bb775a7 Compare November 21, 2026 09:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants