Repository navigation
Conversation
There was a problem hiding this comment.
Thanks!
I don't want to block it but I never had any of those problems with my Google models (Gemini 4 argon, carbon hydro).
Anyone has those problems? @nicolastakashi @saswatamcode
Do we have evidences contributors did this mistake at least once in this repo?
|
@bwplotka I took a critical look at this PR, and revised it. Are the points clearer now? |
The added/revised points came out of reviewing PRs to this repo, and concrete issues found. I can provide specific evidence if that's desired. |
|
Yes please I am curious! What model, what's setup? Your agents? |
|
@bwplotka added to the PR description. Agent-wise, I used DeepSeek v4.1 Flash 1017 in Pi for the PR review, but after your initial review, cross-checked my AGENTS.md changes with Codex/Claude for non-biased takes. The latter helped me par down (remove subjectively weak benchmark advice) and clarify. |
bwplotka
left a comment
There was a problem hiding this comment.
Ok, then can you add those rules to your coding rules? They are not specific to the rust repo are those?
I feel AGENTs.md should have repo specific rule not generic coding rules, wdyt?
@bwplotka I think for that to work, every potential contributor would have to have them. The point is to steer agents better to avoid the pitfalls made in a PR I was reviewing. The idea is to save reviewers' energy. I don't need them in my own AGENTS.md :) I also think that the current advice of "Link the GitHub issue the PR closes" ought to change at the very least, because not every PR closes an issue. Even if it relates to an issue, it may not actually close it. |
| - Sign off every commit with `git commit -s` (DCO). | ||
| - PR titles use `<area>: <summary>`, for example `parse: add OpenMetrics 1.0 parser`. | ||
| - Link the GitHub issue the PR closes. | ||
| - Link the relevant GitHub issue, if any. |
There was a problem hiding this comment.
| - Link the relevant GitHub issue, if any. | |
| - Link the relevant GitHub issue, if any. Use `Fixes #N` only when the PR meets all of the issue's acceptance criteria, otherwise use `Refs #N`. |
|
I'd drop the tree audit and the generic part of the claims audit, since neither is specific to this repo. The tree audit also wouldn't have changed #52, because its author already found the scrape-manager Regarding the Go parity part is specific to this repo, so I'd keep it as one line in "Go is the reference", something like "Do not claim Go parity for behavior you did not verify", wdyt? |
|
Also if we feel we need more determinism from the agents I'd suggest we invest in hooks for the most famous agents e.g. claude, codex, pi, antigravity, opencode so we can make sure that all behave the same way and we can ensure more determinist behavior. |
Encode the checks that catch a change which is correct in its diff but wrong in its claims, its issue links, or its blast radius: a tree-wide grep for the identifiers it touches, a mapping from every PR-body claim to the line or command that proves it, and a match between `Fixes #N` and the issue's acceptance criteria. Add one benchmark rule as well: do not publish a delta whose spread is wider than the effect being claimed. Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Run the tree, claims, and issue audits before opening or updating a PR. Search old and new identifiers and relevant issue references, verify claims in PR descriptions, commits, and specs, and link issues accurately for both complete and partial work. Remove the proposed benchmark rule. Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
4aeb48b to
e27793c
Compare
|
@nicolastakashi applied your feedback. PTAL.
I'm not familiar with these hooks. Would you propose one or more of them? |
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
e27793c to
bb775a7
Compare
Add a check to
AGENTS.mdto, when opening or updating a PR, not claim Go parity for behaviour one did not verify. Also clarify the instruction about closing any issue, because not every PR will close one.The revisions stem from reviewing #52. AFAICT, its claim of fixing #33 isn't correct, it only solves part of it. I propose these revisions, to hopefully steer future agents to avoid falling into the same trap.
Go parity claim
#52 says there are “no intentional differences from Go,” and its commit message broadly describes the timestamp behavior as matching Go. However; AFAICT, for an exemplar without a timestamp,
honor_timestamps=true, a sample timestamp of 10 seconds, and a configured fallback of 20 seconds, Go gives the exemplar 10 seconds because it uses the resolved sample time. Rust gives it 20 seconds because it uses the configured fallback. This seems to be a difference predating #52 though.