Review: WebHarbor reset and smoke verification script (#47, original by @Lxr-max)

[jackjin1997]

Reviewer-owned continuation of #47 by @Lxr-max. The original commit 777fcdf is preserved
as the first commit on this branch; the rest are the reviewer's.

What the original change got right

scripts/check_reset_smoke.py discovers the site registry from websyn_start.sh and
control_server.py, drives POST /reset/<site>, POST /reset-all and GET /health,
smoke-checks each homepage, and degrades cleanly when the control plane is absent.
Ambiguous DB pairs and unknown --site slugs were already structured results rather than
crashes. All of that is kept.

What this change fixes

The DB parity check read a filesystem the reset never touches. It hashed
<repo>/sites/<site>/{instance,instance_seed}, but control_server.reset_db() operates on
/opt/WebSyn/<site>/instance and the Dockerfile lays sites out with COPY sites/ /opt/WebSyn/.
Under the workflow the README documents — script on the host, environment in docker run —
those are never the same files. Reproduced against a live control plane with two real sites:

setup	before	after
environment stopped, local pair self-consistent	md5=PASS "runtime DB matches seed DB"	md5=SKIP "no successful reset to verify"
deployment verified clean, stale row in the checkout's DB	md5=FAIL "differs from seed DB after reset", exit 1	md5=SKIP [none], exit 0; --docker-container reports PASS
clean clone, healthy environment	MD5 0/0/24 — the advertised check never ran	SKIP naming where the real DBs live and how to point at them

instance/ is gitignored and created inside the container, so on a fresh checkout the check
was skipped for every site; four sites (compass, walmart_careers, osu,
rotten_tomatoes) regenerate instance_seed during the image build, so even a fully
asset-fetched checkout holds a different seed than the deployment.

• --docker-container NAME hashes under /opt/WebSyn/<site> inside the running deployment;
  --db-root PATH covers a host deployment. There is no default — without one of them the
  check reports SKIP with source none rather than quietly comparing the checkout.
• every site result reports md5_source (human output and --json), so a PASS always
  names the DBs it read.
• no parity verdict is reported when no reset succeeded, and "after reset" wording is
  reserved for a source that is the reset target.

Registry faults escaped as tracebacks. Registry drift, a missing websyn_start.sh, an
unparseable SITES and a BASE_PORT mismatch all reached the user as unhandled exceptions —
and registry drift is exactly what a reset/smoke checker should report cleanly. They are now
RegistryError rendered as structured findings, with BASE_PORT mismatches named
specifically instead of being reported as list drift.

--strict failed a correct environment. Absent local instance/ emitted a warning and
--strict promoted it to failure, so the README's own --strict example failed on the
documented Docker workflow. That warning is gone; --strict still escalates genuine warnings
such as an undecidable DB pair.

A failed --reset-all was counted once per registered site on top of the endpoint error,
and discarded the HTTP detail. Now recorded once, with the detail.

Verification

• python3 -m pytest scripts/ -q → 20 passed, from a cold clone of this branch into an
  empty directory. 13 tests added; the pre-existing md5 tests now run against a live control
  plane, because a parity verdict needs a reset for "after" to mean anything.
• 20 scenarios executed against a real control plane: this branch's own
  control_server.py and site_runner.py with two real sites (merriam_webster,
  cambridge_dictionary, assets 65c479f8) copied into /opt/WebSyn/<site>, matching the
  Dockerfile layout. Hashes reported by --docker-container were cross-checked against
  docker exec md5sum inside the container and matched exactly.
• Every fix was demonstrated against an injected real defect before and after the change.
• The frozen runs went through two independent review passes. The first returned five
  failures that resolved to one real defect in my own first fix — a flagless run still
  compared the checkout whenever it happened to have an instance/ directory, which failed
  a healthy deployment on a stale local DB and produced a green --json parity result
  without reading /opt/WebSyn. Fixed in fda2266 with regression tests written first. The
  second pass over the re-frozen runs returned 20/20, and additionally caught that
  --db-root's help text still advertised the removed default — fixed in f3a799d.
• Re-verified after merging upstream main (FedEx, Review: Add FedEx WebHarbor mirror (original by @Lxr-max) #82), which changed both registries this
  script parses: 25 sites discovered, fedex at 40024, no traceback.

Not covered

Site UI fidelity, upstream source fidelity, agent task runs and Hugging Face assets are not
applicable to this change; it touches no site, seed, asset or Dockerfile. The scenario
environment ran two real sites rather than all 25 — the 25-site path is covered by registry
discovery against the real registries plus the unit tests, not by live resets of every site.
--docker-container shells out to the docker CLI; podman and remote daemons are untested.
A --db-root verdict is a statement about that root only: the utility reports the source it
read rather than trying to prove that root is what the control plane reset.

[jackjin1997]

Update — an independent review of the frozen runs found a real defect in my own first pass, fixed in fda2266.

The SKIP-with-source-none path was only reached when the checkout had no sites/<site>/instance directory. Any checkout that happened to carry one — anyone who has run a site on the host, or extracted assets and booted locally — silently fell back to comparing the checkout and issued a parity verdict labelled local:, contradicting the documented behaviour that a flagless run reports SKIP rather than comparing this checkout's files. Two consequences, both reproduced against the live control plane:

• a stale local DB failed a healthy deployment: reset=PASS, home=PASS, the deployment's own instance and instance_seed byte-identical, yet exit 1 on "local runtime DB differs from local seed DB";
• a flagless --json run reported md5_status: PASS for every site without ever reading /opt/WebSyn — the same false confidence the first commit set out to remove, just narrowed.

A flagless run now always reports SKIP with source none. Under --db-root, a root that does not hold the site's DBs is an error (the requested check cannot run), while an undecidable DB pair stays a warning so --strict still has a genuine warning to escalate. --reset-all failures now carry the HTTP detail instead of discarding it.

Two regression tests were written first and confirmed failing before the fix; the tests that exercise the local source now pass --db-root explicitly. 20 pass. The full scenario set was re-executed against the same control plane, plus a scenario covering exactly the case that was wrong (healthy deployment, checkout carrying matching instance/ directories, no flag → SKIP [none]).

Still draft: the re-frozen runs are going back for an independent second pass before this is marked ready.