Calibrate implementation rigor before creating process artifacts

[netsky-prod]

Summary

• add preflight reflection before code, specs, plans, tests, or review workflows are created
• separate design shape (spike/bounded/architectural) from process weight
• require each retained artifact or gate to serve a named claim, risk, or material decision
• make reflection select evidence rather than substitute for it
• keep focused review for explicit security, containment, protocol, irreversible, and externally visible risks
• prevent runtime/test machinery from being created solely to validate prompts, skills, prose, or static templates

Motivation

The current three-path router makes architectural work automatically flow through a written spec and implementation plan. That can cause the process to create its own prerequisites: a spec justifies a plan, the plan justifies task decomposition, and task decomposition justifies additional tests and reviews. Once that machinery exists, agents tend to harden it instead of asking whether it should have existed.

This change moves that judgment before the first artifact. The agent identifies the requested deliverable, supported claims, concrete risks, cheapest reliable evidence, retained process, and explicit omissions. The path label then controls design exploration depth only.

The intent is not less evidence. It is no process artifact without an epistemic job. Prompt-only and reversible work should stay lightweight; explicit filesystem, untrusted-input, secret-handling, network, or protocol promises still retain focused validation and review.

Scope

This PR changes only skills/brainstorming/SKILL.md. The public fork contains the downstream applicability changes; this PR isolates the smallest upstream-compatible entry point.

Behavioral comparison

We ran 20 autonomous Codex builds across 10 approved project briefs: one run with upstream Superpowers and one with the reflective fork for each brief. Prompts, model (gpt-5.6-sol, high reasoning), repository seeds, and 30-minute cap were held constant. The prompt described only the product and did not ask agents to reduce ceremony.

After all builds completed, but before any candidate implementation was inspected, an isolated evaluator derived five binary product requirements from each brief. The rubric was therefore masked before scoring but was not preregistered before the experiment. Candidate copies excluded .git, .superpowers, docs/superpowers, node_modules, pycache, .validator-deps, and .verification. Two fresh-context evaluators inspected or ran both candidates against the frozen checklist. Product tests remained available because the evidence procedures could execute them, so masking could still leak through test volume or source structure; this was model-assisted masking, not guaranteed human blinding. Process artifacts, test count, commits, and implementation style earned no score.

Product completeness:

• fork: 48/50 required behaviors; 8/10 projects passed all five
• upstream: 47/50 required behaviors; 8/10 projects passed all five
• unverified judgments: 0

The five-item binary rubric detects gross omissions at the brief threshold. It does not establish quality equivalence above that threshold.

Operational results:

• completed within cap: fork 10/10, upstream 7/10
• faster arm: fork 9/10
• observed sample median: fork 14m21s, upstream 24m54s
• capped total runtime: fork 2h25m38s, upstream at least 3h56m35s
• observed reduction: at least 1h30m57s (38.4% of the capped upstream total)
• process files created: fork 5, upstream 35

Three upstream runs were terminated at the 30-minute cap, so their true completion times are unknown: the total runtime and time reduction reported above are lower bounds. The sample median is still identified because its fifth and sixth order statistics were both completed, uncensored runs.

Verification-modality trace audit

A post-hoc, unblinded trace audit coded five performed verification modalities: automated behavior suite, runtime or real-browser smoke, package validator, adversarial boundary probe, and independent reviewer.

• fork had an arm-only modality in 1/10 pairs: a real headless-browser interaction pass for Decision Journal
• upstream had an arm-only modality in 8/10 pairs: independent review, sometimes followed by additional adversarial regressions
• 1/10 pairs used the same modality set

An accepted review correction is not automatically a product improvement. A paired impact audit traced each correction back to the literal brief and checked whether the fork already delivered the same behavior without review. Of the eight upstream-only review pairs, three produced a unique brief-bound protection missing from the fork: the absolute no-human-approval rule in the incident plugin, startup rejection of HTTP header values the server cannot emit, and race-safe enforcement of the no-symlink-outside-root promise. Two review pairs found risks the fork had already handled without a reviewer (webhook replay and documentation-root containment). The remaining three mainly hardened implementation-specific edges or reviewer-created contracts not required by the brief.

The paired impact audit was targeted at the eight upstream-only review pairs; it was not an exhaustive re-audit of all 95 PASS judgments. It found one false positive incidentally: the incident-plugin criterion passed even though the fork allowed an approval claim when supplied in source material, while the brief prohibited any such claim. The raw 48/50 and 47/50 pass counts are therefore upper bounds whose unknown overstatement may differ by arm. They must not be read as a fork quality advantage or as proof of quality equivalence; at most, the masked rubric found no obvious broad completeness collapse.

The benchmarked fork removed review more consistently than it selected it. The current PR head incorporates the resulting refinement by explicitly retaining one focused boundary or final review for named containment, untrusted-input, secret-handling, network, or protocol promises. That refinement was derived from the traces and has not been validated by the 20-run comparison.

The experiment used one model (gpt-5.6-sol at high reasoning) and exactly one build per policy per brief, with no within-condition repetitions. It therefore does not estimate run-to-run variance and cannot separate a rule effect from stochastic variation within any individual pair; the aggregate is descriptive, not a causal effect estimate.

The broader fork at commit 37b8155 was used for the comparison, so the numbers validate that intervention stack and expose its review-selection gap rather than attributing the entire effect to this single-file PR. The current upstream-scoped head adds the preflight decision and the boundary-review refinement derived from the traces.