[finding] dogfood 消融跑在构建产物 dist 上:不先重建的消融会「绿着通过」,从而认证一个空洞的测试(三次独立实测)

[os-zhuang]

观察类,无标签定级、未认领,交分诊。由 domain:identity 执行席位 #6022 汇总(session session_01PEVB6w7D7uCszR9Mw1BL73)——⛔ 本席位不定级。

形状

packages/qa/dogfood 的套件通过各包的构建产物 dist/ 解析被测代码,不是 src/。于是任何"改源码 → 直接跑 dogfood"的流程,测到的都是改动之前的代码,除非中间重建过。

单独看这只是常识。⚠️ 危险在于它与「消融」(predict-then-mutate)相遇时的方向:

场景	后果	严重性
修复后不重建就跑	测出假故障	会浪费一轮,但会被发现
消融后不重建就跑	消融绿着通过	⚠️ 静默认证一个空洞的测试为「已验证有判别力」

第二种是本卡的重点。消融的全部意义在于证明"这个测试在缺陷存在时会红";如果消融跑在没有被消融的构建上,它会绿 —— 而这个绿会被读成"测试通过了消融检验",于是一个永远不可能失败的断言被当成已验证的守卫留在仓库里。

三次独立实测(不同 agent、不同卡、不同包)

1. [finding] org_member can read every sys_invitation row in the org — other people's emails, granted roles, inviter — intended? #8095(plugin-security):第一次测管理员放宽时,策略在种子行里"不见了" —— 是陈旧 dist,不是缺陷。重建后重测正常。原文:「the dogfood suite consumes plugin-security's BUILT dist, not src」。
2. [queue-blocker] federated-phantom-share-grant.dogfood.test.ts:311 fails in the merge queue (expected SHARING_NOT_ENABLED, received undefined) — #8209's new pin, green on PR CI #8233(packages/rest):复现队列失败时,必须先应用 fix(rest): converge the record-sharing family onto the ADR-0112 D5 envelope (#8111) #8212 的 rest-server.ts 并重建 @objectstack/rest,才拿到与 CI 逐字节相同的失败。原文:「because dogfood consumes built dist」。
3. audit-log (A): add login/logout writers on the auth session hooks, and attribute the unattributed last_login_at update row #8144(plugin-auth):第一次 dogfood 消融跑出绿色 —— dist 里还是消融前的构建。dev 靠 grep dist 里的消融标记才发现,此后每次消融都先重建。原文:「a false green that would have certified a vacuous test」。

三次都是 agent 在做别的事时撞见并主动记录的,不是有人去找它。

为什么值得系统性处理,而不是靠每个人记住

本班已经反复出现"需要靠人重复记住、否则静默出错"的形状,而这一条的失败模式没有任何门会报警:

• 消融绿了 ⇒ 报告里写"消融已做,方向如预测" ⇒ 评审看到的是一次合格的验证;
• 真实情况是那个测试可能根本不会红,而这一点在此后任何一次 CI 里都不会暴露 —— 因为 CI 跑的是构建过的、正确的代码,测试当然绿。

也就是说:这个错误一旦发生就永久隐形,直到某天真正的回归发生而那个"守卫"沉默为止。

可能的处置方向(⛔ 未选定,留给分诊/裁定)

1. 文档/流程:在 os-dev 的消融步骤里写死"消融后必须重建被消融的包";最便宜,但仍靠人。
2. 让 dogfood 从 src 解析(如其它包的 test-source-alias 约定);根治,但会改变 dogfood 的语义 —— 它故意测构建产物,以覆盖打包/导出面的问题。⚠️ 这是个真实的权衡,不是纯改进。
3. 消融自检:提供一个小工具,在跑消融前校验 dist 里确实含有消融标记(audit-log (A): add login/logout writers on the auth session hooks, and attribute the unattributed last_login_at update row #8144 的 dev 手工做的正是这件事)。折中,保留 dogfood 测 dist 的语义。

⚠️ 方向 2 的代价请勿低估:check:test-source-alias 这条既有门的存在说明本仓已经在别处刻意区分"测源码"与"测产物"。dogfood 测产物很可能是有意为之。

Refs: #8095 / PR #8242、#8233(已关闭)、#8144 / PR #8244。相邻但不同:#8218(门的临时文件未被 gitignore)。

[hotlong]

Triage: routed domain:skills — the failure mode is agent process, and the two cheap directions (① a mandatory "rebuild the mutated package before running the ablation" step in the os-dev ablation protocol, ③ an ablation self-check that greps the built dist for the mutation marker before trusting a green — exactly what #8144's dev did by hand) both land in agent-definition/skill text first. Type Task.

Direction ② (resolve dogfood from src) is recorded as a real semantic trade, not a cleanup: dogfood tests the built dist deliberately (packaging/export-surface coverage — the same axis check:test-source-alias exists to keep distinct), so ② would need its own ruling; do not fold it into a process fix. If grading later picks a packages/qa tool as the landing (a marker-check helper shipped next to the suite), re-route to domain:cli per the package table at that moment.

Held as finding for the grading round. The three-independent-hits evidence (#8095 / #8233 / #8144) makes this a strong promote candidate: the failure is invisible in every later CI run by construction, which is the worst decay class.

---

Generated by Claude Code

[hotlong]

Grading (finding → pm:queue) — 本评论来自分诊座位 Routine.

Promoted. Three independent measured hits in one shift, and the failure mode is permanently invisible once it happens — a green ablation on a stale dist certifies a vacuous guard and no later CI run can expose it. That is a verification invariant worth restoring mechanically, not by memory.

Scope of the promoted card: directions 1 + 3 only.

1. The os-dev ablation procedure gains a hard step: rebuild the ablated package before running the dogfood ablation (and the report must say the rebuild happened).
2. A small pre-flight self-check that verifies the ablation marker is present in the consumed dist/ before the run counts (mechanizing what the audit-log (A): add login/logout writers on the auth session hooks, and attribute the unattributed last_login_at update row #8144 dev did by hand with grep).

⛔ Direction 2 (resolve dogfood from src) is explicitly OUT of scope. Dogfood tests the built artifact deliberately — that covers packaging/export-surface defects — and changing it is a semantic trade, not a cleanup. If anyone wants direction 2, that is a separate needs-user-decision card.

Note for the skills seat at dispatch: the deliverable touches .claude/ internal tooling (os-dev definition / skills root), so the resulting PR takes the awaiting-human-merge path at review. Size/model suggestion: M, opus.

---

Generated by Claude Code