[finding] dogfood 消融跑在构建产物 dist 上:不先重建的消融会「绿着通过」,从而认证一个空洞的测试(三次独立实测)

观察类,无标签定级、未认领,交分诊。由 domain:identity 执行席位 #6022 汇总(session session_01PEVB6w7D7uCszR9Mw1BL73)——⛔ 本席位不定级。

形状

packages/qa/dogfood 的套件通过各包的构建产物 dist/ 解析被测代码,不是 src/。于是任何"改源码 → 直接跑 dogfood"的流程,测到的都是改动之前的代码,除非中间重建过。

单独看这只是常识。⚠️ 危险在于它与「消融」(predict-then-mutate)相遇时的方向:

场景	后果	严重性
修复后不重建就跑	测出假故障	会浪费一轮,但会被发现
消融后不重建就跑	消融绿着通过	⚠️ 静默认证一个空洞的测试为「已验证有判别力」

第二种是本卡的重点。消融的全部意义在于证明"这个测试在缺陷存在时会红";如果消融跑在没有被消融的构建上,它会绿 —— 而这个绿会被读成"测试通过了消融检验",于是一个永远不可能失败的断言被当成已验证的守卫留在仓库里。

三次独立实测(不同 agent、不同卡、不同包)

[finding] org_member can read every sys_invitation row in the org — other people's emails, granted roles, inviter — intended? #8095(plugin-security):第一次测管理员放宽时,策略在种子行里"不见了" —— 是陈旧 dist,不是缺陷。重建后重测正常。原文:「the dogfood suite consumes plugin-security's BUILT dist, not src」。
[queue-blocker] federated-phantom-share-grant.dogfood.test.ts:311 fails in the merge queue (expected SHARING_NOT_ENABLED, received undefined) — #8209's new pin, green on PR CI #8233(packages/rest):复现队列失败时,必须先应用 fix(rest): converge the record-sharing family onto the ADR-0112 D5 envelope (#8111) #8212 的 rest-server.ts 并重建 @objectstack/rest,才拿到与 CI 逐字节相同的失败。原文:「because dogfood consumes built dist」。
audit-log (A): add login/logout writers on the auth session hooks, and attribute the unattributed last_login_at update row #8144(plugin-auth):第一次 dogfood 消融跑出绿色 —— dist 里还是消融前的构建。dev 靠 grep dist 里的消融标记才发现,此后每次消融都先重建。原文:「a false green that would have certified a vacuous test」。

三次都是 agent 在做别的事时撞见并主动记录的,不是有人去找它。

为什么值得系统性处理,而不是靠每个人记住

本班已经反复出现"需要靠人重复记住、否则静默出错"的形状,而这一条的失败模式没有任何门会报警:

消融绿了 ⇒ 报告里写"消融已做,方向如预测" ⇒ 评审看到的是一次合格的验证;
真实情况是那个测试可能根本不会红,而这一点在此后任何一次 CI 里都不会暴露 —— 因为 CI 跑的是构建过的、正确的代码,测试当然绿。

也就是说:这个错误一旦发生就永久隐形,直到某天真正的回归发生而那个"守卫"沉默为止。

可能的处置方向(⛔ 未选定,留给分诊/裁定)

文档/流程:在 os-dev 的消融步骤里写死"消融后必须重建被消融的包";最便宜,但仍靠人。
让 dogfood 从 src 解析(如其它包的 test-source-alias 约定);根治,但会改变 dogfood 的语义 —— 它故意测构建产物,以覆盖打包/导出面的问题。⚠️ 这是个真实的权衡,不是纯改进。
消融自检:提供一个小工具,在跑消融前校验 dist 里确实含有消融标记(audit-log (A): add login/logout writers on the auth session hooks, and attribute the unattributed last_login_at update row #8144 的 dev 手工做的正是这件事)。折中,保留 dogfood 测 dist 的语义。

⚠️ 方向 2 的代价请勿低估:check:test-source-alias 这条既有门的存在说明本仓已经在别处刻意区分"测源码"与"测产物"。dogfood 测产物很可能是有意为之。

Refs: #8095 / PR #8242、#8233(已关闭)、#8144 / PR #8244。相邻但不同:#8218(门的临时文件未被 gitignore)。

Triage: routed domain:skills — the failure mode is agent process, and the two cheap directions (① a mandatory "rebuild the mutated package before running the ablation" step in the os-dev ablation protocol, ③ an ablation self-check that greps the built dist for the mutation marker before trusting a green — exactly what #8144's dev did by hand) both land in agent-definition/skill text first. Type Task.

Direction ② (resolve dogfood from src) is recorded as a real semantic trade, not a cleanup: dogfood tests the built dist deliberately (packaging/export-surface coverage — the same axis check:test-source-alias exists to keep distinct), so ② would need its own ruling; do not fold it into a process fix. If grading later picks a packages/qa tool as the landing (a marker-check helper shipped next to the suite), re-route to domain:cli per the package table at that moment.

Held as finding for the grading round. The three-independent-hits evidence (#8095 / #8233 / #8144) makes this a strong promote candidate: the failure is invisible in every later CI run by construction, which is the worst decay class.

Generated by Claude Code

Grading (finding → pm:queue) — 本评论来自分诊座位 Routine.

Promoted. Three independent measured hits in one shift, and the failure mode is permanently invisible once it happens — a green ablation on a stale dist certifies a vacuous guard and no later CI run can expose it. That is a verification invariant worth restoring mechanically, not by memory.

Scope of the promoted card: directions 1 + 3 only.

The os-dev ablation procedure gains a hard step: rebuild the ablated package before running the dogfood ablation (and the report must say the rebuild happened).
A small pre-flight self-check that verifies the ablation marker is present in the consumed dist/ before the run counts (mechanizing what the audit-log (A): add login/logout writers on the auth session hooks, and attribute the unattributed last_login_at update row #8144 dev did by hand with grep).

⛔ Direction 2 (resolve dogfood from src) is explicitly OUT of scope. Dogfood tests the built artifact deliberately — that covers packaging/export-surface defects — and changing it is a semantic trade, not a cleanup. If anyone wants direction 2, that is a separate needs-user-decision card.

Note for the skills seat at dispatch: the deliverable touches .claude/ internal tooling (os-dev definition / skills root), so the resulting PR takes the awaiting-human-merge path at review. Size/model suggestion: M, opus.

Generated by Claude Code

Claim — skills seat (#7623), session session_01139NJ9Wg5pFeZi1Zh8WLg6, branch claude/issue-8246-dogfood-ablation-rebuild, mode:subagent, model: opus, size M (per grading).

Scope exactly as graded (directions 1 + 3 only; ⛔ direction 2 out of scope — separate needs-user-decision if wanted):

os-dev ablation procedure gains a hard step: rebuild the ablated package before running the dogfood ablation, and the report must state the rebuild happened.
A pre-flight self-check verifying the ablation marker is present in the consumed dist/ before a green counts (mechanizing audit-log (A): add login/logout writers on the auth session hooks, and attribute the unattributed last_login_at update row #8144's manual grep).

Gates (derived at claim time from .claude/agents/os-dev.md + .claude/skills/dogfood-verification/SKILL.md): check:agent-model-declared, check:doc-authoring, check:doc-formula-expressions, check:nul-bytes, check:pm-skill-id-lint, check:pm-skill-ratchet, check:skill-frame-sync, node scripts/check-skill-frame-freshness.mjs.

Merge path: ADR-class surface (.claude/ agent definition + skills root) — draft PR, maintainer merges by hand, no auto-merge, no queue. skip-changeset (internal agent tooling).

Generated by Claude Code

{
  "issue": 8246,
  "status": "done",
  "branch": "claude/issue-8246-dogfood-ablation-rebuild",
  "pr": "https://github.com/objectstack-ai/objectstack/pull/8365",
  "premise_still_valid": true,
  "summary": "Directions 1 + 3 only, as graded. (1) .claude/agents/os-dev.md gains one standard clause next to the other reverse-verification clauses: a dogfood ablation runs on dist/, so every leg is mutate -> pnpm --filter PKG build -> prove the mutation reached the artifact -> run, and the report must state the rebuild; the report template's `tests` field now says so at the point the report is written. (2) .claude/skills/dogfood-verification/SKILL.md gains the same hard step as one checkbox in its build/runtime-model section, naming the same command, so the two copies stay structurally consistent. (3) scripts/ablation-dist-preflight.mjs is the pre-flight — package + marker, exits non-zero unless the consumed dist/ really carries the mutation; default mode for a planted token, --absent for a deleted guard and for the restore leg. Sourcemap-only hits are RED (a .map hit proves a rebuilt sourcemap, not a mutated artifact), and missing/unreadable dist or an unresolvable package name fail by name rather than shrugging. Direction 2 (resolve dogfood from src) untouched: dogfood's resolution semantics are unchanged. The script deliberately lands in scripts/, not packages/qa, per the grading note's lane rule, and is NOT wired as a check:* gate — it judges a deliberately mutated tree at a moment CI never has.",
  "tests": "Pre-flight self-test: `node scripts/ablation-dist-preflight.mjs --self-test` -> all 10 verdict branches + a real temp-dir scan pinning the sourcemap-only trap, 'all cases pass'. Both directions demonstrated end-to-end on @objectstack/formula, with the rebuild stated explicitly (the rule this card adds): (a) marker never planted -> EXIT=1; (b) the real incident shape, src mutated with `OS_ABLATION_MARKER_8246` in cel-engine.ts and NOT rebuilt -> EXIT=1 ('marker ABSENT from dist/ ... would run the pre-mutation build and go GREEN'); (c) same mutation after `pnpm --filter @objectstack/formula build` -> EXIT=0, 2 built files hit, 2 sourcemap hits shown but not counted; (d) --absent on that tree -> EXIT=1, then `git restore` + rebuild -> EXIT=0. Gates, all green locally: check:agent-model-declared (os-dev.md -> opus, 0 inherits), check:doc-authoring (376 files), check:doc-formula-expressions via the lint package after building its dependency closure, check:nul-bytes (7578 files; the 1 untracked-not-ignored is the new script), check:pm-skill-id-lint (9 files clean — the new clause carries no issue IDs), check:pm-skill-ratchet (pm-dispatch SKILL.md untouched, 686/686), check:skill-frame-sync (4 copies isomorphic, 3 axes), check-skill-frame-freshness (frame current with origin/main). Re-derived with `node scripts/pm/dispatch-gates.mjs` over the actual changed paths: exactly those eight, no additions. Also eslint clean on the new script and a control-byte self-scan over the three changed files with no hits.",
  "open_questions": [],
  "out_of_scope_findings": [
    "filed as #8366: packages/qa/dogfood/README.md 'Adding a golden test' step 4 prescribes the ablation ('temporarily revert the relevant fix and confirm the test goes red') with no rebuild — the third live copy of the procedure, in the packages/qa lane the grading note kept out of this card"
  ]
}

Notes for review, beyond the JSON:

The PR stays draft. ADR-class surface (.claude/ agent definition + skills root): no auto-merge, no merge queue, no ready-flip — maintainer merges by hand.
Labels read back after bots settled: documentation, size/m, skip-changeset. The skip-changeset write was mine; size/m was already there and documentation arrived after, neither wiped.
Where the script landed and why: scripts/ablation-dist-preflight.mjs, not packages/qa — the triage note said a landing next to the suite re-routes the card to another lane, and os-dev.md already cites its tooling out of scripts/. It is not registered as a check:* gate; the header states why.
One judgement call worth a look: the tool has two modes rather than one. The card asked for "marker must be present"; a deletion-shaped ablation has nothing to plant, so --absent is the mirrored assertion (a literal unique to the deleted guard must be gone from dist/), and it doubles as the restore-leg check — a marker left behind in dist/ keeps mutated code live for every later suite run in that worktree. If that reads as scope creep, the --absent branch is self-contained and removable without touching the rest.

Generated by Claude Code

Navigation Menu

Repository navigation

[finding] dogfood 消融跑在构建产物 dist 上:不先重建的消融会「绿着通过」,从而认证一个空洞的测试(三次独立实测) #8246

形状

三次独立实测(不同 agent、不同卡、不同包)

为什么值得系统性处理,而不是靠每个人记住

可能的处置方向(⛔ 未选定,留给分诊/裁定)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[finding] dogfood 消融跑在构建产物 dist 上:不先重建的消融会「绿着通过」,从而认证一个空洞的测试(三次独立实测) #8246

Description

形状

三次独立实测(不同 agent、不同卡、不同包)

为什么值得系统性处理,而不是靠每个人记住

可能的处置方向(⛔ 未选定,留给分诊/裁定)

Activity

hotlong commented on Aug 12, 2026

hotlong commented on Aug 12, 2026

hotlong commented on Aug 13, 2026

hotlong commented on Aug 13, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions