Reverse sweep: detect and repair holes in the driver repo (full behind meta) #192

Problem

The ShardReconciler (#139/#142) uses full as its driver and repairs the other shards against it — so nanopubs missing from full itself are its documented blind spot (ShardReconciler.java). This week produced three real cases of exactly that class, all on rdf4j 6.0.0:

1. petapico, 2026-08-21 morning: 1 nanopub acked during an MDB_READERS_FULL window, permanently absent from the store. Loader counter matched the fleet; only count(distinct ?np) vs meta revealed it.
2. kpxl, 2026-08-20 14:31–14:48Z: 7 nanopubs published in the minutes before a crash-looping restart — present in meta and text, absent from full, load-number sequences diverged by 7.
3. kpxl, 2026-08-21: 68 nanopubs lost when a recovery restart dropped rdf4j's acked-but-unmerged changesets (the changeset-overlay pile-up incident). Again: meta/text complete, full silently short.

The invariant that makes detection easy: the loader writes meta last, only after every other repo acked — so meta containing a nanopub that full lacks is proof of an acked-then-lost commit, never of in-flight loading.

Proposed fix

Extend the reconciliation sweep with a reverse pass:

• Verify full membership against meta's load stamps (a bounded recent window per tick is enough — all observed holes were near the head, created by restart/overload events).
• Repair by reconstructing content from the per-pubkey shard (fetchNanopubAllStatementsFromPubkeyRepo, the same source ShardReconciler.reloadNanopub already uses) and calling loadNanopubToRepo(np, content, "full") — the count/checksum chain converges because the XOR checksum is order-independent (proven by this week's manual repairs).

Manual workarounds used meanwhile (for the record)

• Counter rewind (bounded recent holes, <1 min): stop the query container, set npa:hasRegistryLoadCounter in the admin repo to just below the hole and npa:hasStatus to READY, start plain — the update loop re-processes only that window with the per-repo isLoaded skips.
• Full wipe + re-ingest (heavyweight), or FORCE_RESYNC — but see the companion issue: the whole-registry sweep is not viable on a live instance.

🤖 Generated with Claude Code