Skip to content

[DO NOT MERGE] Assets: integration branch for #16358 + #16393 - #16459

Draft
synap5e-bot[bot] wants to merge 114 commits into
masterfrom
synap5e/test/merge-16358-16393
Draft

synap5e-bot[bot] wants to merge 114 commits into
masterfrom
synap5e/test/merge-16358-16393

Conversation

@synap5e-bot

@synap5e-bot synap5e-bot Bot commented Sep 21, 2026

Copy link
Copy Markdown

What this is

Do not merge. This branch is #16358 and #16393 combined on top of current master, so the two can be exercised together before either one lands. It contains no work of its own beyond the conflict resolution described below, and nothing should be based on it — it goes stale the moment either PR head moves.

Contents: #16358 at 9d71c9b7 + #16393 at ca8aba7a + master at b0f4b7b2.

Why it exists

Both PRs are individually mergeable against master, so the conflict only appears in the combination: 15 files, 5 source and 10 test. They overlap because they change the same functions for different reasons — #16358 changes where work happens relative to the SQLite write lock (stats and hashes hoisted out, chunked bounded transactions), and #16393 changes what happens when that work fails. In seed_asset_specs and tick_watch_list those are two answers to the same question, so the combination needs a decision rather than a merge strategy.

How the conflicts were resolved

The principle throughout: #16358's transaction discipline and #16393's failure-path hardening both survive. Three places were genuine design interactions rather than textual conflicts.

seed_asset_specs — opposed failure policies. #16358 wraps the spec loop so any fault rolls the whole batch back, discards orphaned content and re-raises; #16393 replaced that with per-spec absorption into a returned first_error so one bad file cannot discard the batch. Resolved by layering: each spec keeps its own savepoint, faults are absorbed and returned, and the outer compensation is kept for what still escapes.

insert_asset_specs — a problem neither PR has alone. #16358 moved the commit into run_write_txn, which retries its callback on lock contention; #16393 needs the batch fault available after a failed commit. The fault is therefore captured out of band and reassigned on every attempt, so one recorded by an attempt whose work was discarded cannot outlive it.

tick_watch_list — drop the entry or keep it. The two PRs carry directly contradictory assertions here. Resolved by separating an absorbed fault (bad stat, bad spec, returned seed error → drop the entry with telemetry, which is what non-wedging was about) from an exception escaping the write transaction (→ put the entry back, because a failed write is the database's fault and not the file's).

Two smaller notes: mark_missing_outside_prefixes_safely's two return contracts turned out compatible rather than conflicting — None only when the prune failed having committed nothing, a partial count otherwise — and both PRs' existing tests pass unchanged. And #16393's guard for a file vanishing during the recovery hash is structurally absorbed by #16358 hashing before the transaction opens.

What a reviewer should look at

Three things are unresolved by design, and the trade is stated rather than defended.

  1. Under the layering above, the only fault that still escapes seed_asset_specs is MemoryError, so for the ordinary case the orphan compensation existed for — create_record failing after create_content inserted — that compensation no longer runs and savepoint rollback is the whole guarantee. That holds on the WAL writer engine this combination gets from fix(assets): prevent SQLite concurrent writer failures #16358. _discard_unreferenced_content's own docstring says it does not hold on a no-WAL engine and that the binding is not knowable from the call site, which makes this a question about fix(assets): harden the asset catalogue's failure paths #16393 standing alone rather than about the combination. Closing it properly means per-spec compensation, which neither PR writes, so it was flagged rather than invented here.
  2. The tick_watch_list split above is a judgment call. It was resolved toward fix(assets): harden the asset catalogue's failure paths #16393, and fix(assets): prevent SQLite concurrent writer failures #16358's queue-drain characterization test — which documented the pre-hardening behaviour of an unreadable watched file propagating out of the tick — was rewritten to document the new behaviour rather than deleted.
  3. Four changed signatures or return types rippled into call sites outside the conflict hunks, where the merge silently kept whichever spelling it picked. Those are reconciled in a separate commit (1adfd4b5) so they read apart from the resolution itself.

Tests

python -m pytest tests-unit/assets_test tests-unit/seeder_test tests-unit/app_test tests-unit/test_assets_event_log_static.py tests-unit/feature_flags_test.py -q \
  --ignore=tests-unit/app_test/model_manager_test.py \
  --ignore=tests-unit/app_test/test_db_init_locking.py \
  --ignore=tests-unit/app_test/test_prompt_worker.py \
  --deselect tests-unit/assets_test/test_event_log_sites.py::test_noassets_emits_no_enabled_event

863 passed, 72 skipped, 1 deselected, 91 warnings in 45.67s

ruff check is clean on app/assets, app/database and the touched test trees.

The four excluded targets are a gap in local verification, not a claim about the branch: they fail identically on master in the same environment, which is what pins it on the environment rather than on this merge. CI on this PR is the real check for them.

🤖 Generated with Claude Code

…eout, immediate write transactions, bounded retry)
init_db() runs for every install, so forcing journal_mode=WAL there made a
default startup promote journal mode, leave -wal/-shm sidecars, and raise on
any filesystem that rejects WAL - none of which a user who never enabled
assets asked for. Assets is the database's only writer, so gate the WAL pragma
and the writer engine on the flag and keep upstream's single-engine wiring
otherwise.

NoAssets.startup() now cleans the temp directory before recording hash-mode
transition intent, so an unusable database costs the transition record rather
than also costing temp cleanup.
Commit 1dbcdcd and the comment-cleanup pass 8205022 reduced this to "All
database reads and writes, including the legacy import, run under the lock",
dropping the part that did the work: upstream master locks after migrating and
justifies it with "Alembic uses its own connection, so we must wait until it's
done before locking -- otherwise our own lock blocks the migration". That is
false, the lock is on a separate <db>.lock file, and the surviving sentence
said nothing to stop a contributor "fixing" the ordering back.

Restored and adapted rather than pasted: the legacy copy and the db_exists
probe now happen inside the lock, which the original text predates, so both
are named in the list of things the ordering makes mutually exclusive.
MemoryError is an Exception, so the per-spec and per-batch handlers stored it alongside ordinary faults and carried on - allocating for every remaining spec and then every remaining batch while the process was already out of memory. Both handlers now let it through, and the scan records a failure and stops.
The route gained a 500 PRUNE_FAILED branch and the seeder method gained a None return, both so a prune that did not run cannot be reported as a clean one. Neither contract was written down.
…ning)

Temp branch for combined testing only; not for upstream. Base is current
origin/master. #16358 merged clean; #16393 conflicted in 5 source files and
10 test files.

Resolution principle: #16358's transaction discipline (stat/hash taken before
the write lock, bounded chunked write transactions, interrupt_check) and
#16393's failure-path hardening (per-spec fault isolation, truthful telemetry,
non-wedging watch list) both survive.
The conflict resolution changed four signatures/returns that call sites
outside the conflict hunks still used the old spelling of. Auto-merge kept
whichever side a given hunk came from, so these are merge fallout, not new
behaviour:

- seed_asset_specs now takes #16358's pre-lock stats/prepared-recovery
  arguments AND returns #16393's (created, batch_fault). Test call sites and
  the seed_with_recovery helper widened to match.
- insert_asset_specs returns (created, batch_fault) and commits through
  run_write_txn, so tests that bound the fixture engine at scanner's
  create_session bind it at app.database.db.WriteSession instead.
- enrich_assets_batch returns #16393's third element (consumed).
- tick_watch_list takes #16358's interrupt_check instead of a caller session.

Two behavioural picks where the PRs genuinely disagree, both resolved toward
#16393 (the later, explicitly hardening change), with the superseded #16358
tests updated rather than deleted:

- An unreadable watched file is dropped with scanner.watch_stat_failed instead
  of propagating out of the tick. #16358's queue-drain *characterization* test
  documented the old behaviour; it now documents the new one.
- tick_watch_list pops each entry before anything that can fail, so a stat,
  spec-build or seed fault cannot wedge the list — but an exception escaping
  insert_asset_specs puts the entry back, because a failed write transaction
  is the database's fault and not the file's. That keeps #16358's
  terminal-commit-failure test passing; #16393's escaping-fault test asserted
  the dropped entry and now asserts it is retained.

tests-unit/{assets_test,seeder_test,app_test}: 862 passed, 72 skipped.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant