[DO NOT MERGE] Assets: integration branch for #16358 + #16393 - #16459
Draft
synap5e-bot[bot] wants to merge 114 commits into
Draft
synap5e-bot[bot] wants to merge 114 commits into
synap5e-bot[bot] wants to merge 114 commits into
Conversation
…eout, immediate write transactions, bounded retry)
…d lock-hold bounded
…-transaction rule
init_db() runs for every install, so forcing journal_mode=WAL there made a default startup promote journal mode, leave -wal/-shm sidecars, and raise on any filesystem that rejects WAL - none of which a user who never enabled assets asked for. Assets is the database's only writer, so gate the WAL pragma and the writer engine on the flag and keep upstream's single-engine wiring otherwise. NoAssets.startup() now cleans the temp directory before recording hash-mode transition intent, so an unusable database costs the transition record rather than also costing temp cleanup.
Commit 1dbcdcd and the comment-cleanup pass 8205022 reduced this to "All database reads and writes, including the legacy import, run under the lock", dropping the part that did the work: upstream master locks after migrating and justifies it with "Alembic uses its own connection, so we must wait until it's done before locking -- otherwise our own lock blocks the migration". That is false, the lock is on a separate <db>.lock file, and the surviving sentence said nothing to stop a contributor "fixing" the ordering back. Restored and adapted rather than pasted: the legacy copy and the db_exists probe now happen inside the lock, which the original text predates, so both are named in the list of things the ordering makes mutually exclusive.
MemoryError is an Exception, so the per-spec and per-batch handlers stored it alongside ordinary faults and carried on - allocating for every remaining spec and then every remaining batch while the process was already out of memory. Both handlers now let it through, and the scan records a failure and stops.
The route gained a 500 PRUNE_FAILED branch and the seeder method gained a None return, both so a prune that did not run cannot be reported as a clean one. Neither contract was written down.
…txn' into synap5e/test/merge-16358-16393
…ning) Temp branch for combined testing only; not for upstream. Base is current origin/master. #16358 merged clean; #16393 conflicted in 5 source files and 10 test files. Resolution principle: #16358's transaction discipline (stat/hash taken before the write lock, bounded chunked write transactions, interrupt_check) and #16393's failure-path hardening (per-spec fault isolation, truthful telemetry, non-wedging watch list) both survive.
The conflict resolution changed four signatures/returns that call sites outside the conflict hunks still used the old spelling of. Auto-merge kept whichever side a given hunk came from, so these are merge fallout, not new behaviour: - seed_asset_specs now takes #16358's pre-lock stats/prepared-recovery arguments AND returns #16393's (created, batch_fault). Test call sites and the seed_with_recovery helper widened to match. - insert_asset_specs returns (created, batch_fault) and commits through run_write_txn, so tests that bound the fixture engine at scanner's create_session bind it at app.database.db.WriteSession instead. - enrich_assets_batch returns #16393's third element (consumed). - tick_watch_list takes #16358's interrupt_check instead of a caller session. Two behavioural picks where the PRs genuinely disagree, both resolved toward #16393 (the later, explicitly hardening change), with the superseded #16358 tests updated rather than deleted: - An unreadable watched file is dropped with scanner.watch_stat_failed instead of propagating out of the tick. #16358's queue-drain *characterization* test documented the old behaviour; it now documents the new one. - tick_watch_list pops each entry before anything that can fail, so a stat, spec-build or seed fault cannot wedge the list — but an exception escaping insert_asset_specs puts the entry back, because a failed write transaction is the database's fault and not the file's. That keeps #16358's terminal-commit-failure test passing; #16393's escaping-fault test asserted the dropped entry and now asserts it is retained. tests-unit/{assets_test,seeder_test,app_test}: 862 passed, 72 skipped.
…txn' into synap5e/test/merge-16358-16393
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
Do not merge. This branch is #16358 and #16393 combined on top of current
master, so the two can be exercised together before either one lands. It contains no work of its own beyond the conflict resolution described below, and nothing should be based on it — it goes stale the moment either PR head moves.Contents: #16358 at
9d71c9b7+ #16393 atca8aba7a+masteratb0f4b7b2.Why it exists
Both PRs are individually mergeable against
master, so the conflict only appears in the combination: 15 files, 5 source and 10 test. They overlap because they change the same functions for different reasons — #16358 changes where work happens relative to the SQLite write lock (stats and hashes hoisted out, chunked bounded transactions), and #16393 changes what happens when that work fails. Inseed_asset_specsandtick_watch_listthose are two answers to the same question, so the combination needs a decision rather than a merge strategy.How the conflicts were resolved
The principle throughout: #16358's transaction discipline and #16393's failure-path hardening both survive. Three places were genuine design interactions rather than textual conflicts.
seed_asset_specs— opposed failure policies. #16358 wraps the spec loop so any fault rolls the whole batch back, discards orphaned content and re-raises; #16393 replaced that with per-spec absorption into a returnedfirst_errorso one bad file cannot discard the batch. Resolved by layering: each spec keeps its own savepoint, faults are absorbed and returned, and the outer compensation is kept for what still escapes.insert_asset_specs— a problem neither PR has alone. #16358 moved the commit intorun_write_txn, which retries its callback on lock contention; #16393 needs the batch fault available after a failed commit. The fault is therefore captured out of band and reassigned on every attempt, so one recorded by an attempt whose work was discarded cannot outlive it.tick_watch_list— drop the entry or keep it. The two PRs carry directly contradictory assertions here. Resolved by separating an absorbed fault (bad stat, bad spec, returned seed error → drop the entry with telemetry, which is what non-wedging was about) from an exception escaping the write transaction (→ put the entry back, because a failed write is the database's fault and not the file's).Two smaller notes:
mark_missing_outside_prefixes_safely's two return contracts turned out compatible rather than conflicting —Noneonly when the prune failed having committed nothing, a partial count otherwise — and both PRs' existing tests pass unchanged. And #16393's guard for a file vanishing during the recovery hash is structurally absorbed by #16358 hashing before the transaction opens.What a reviewer should look at
Three things are unresolved by design, and the trade is stated rather than defended.
seed_asset_specsisMemoryError, so for the ordinary case the orphan compensation existed for —create_recordfailing aftercreate_contentinserted — that compensation no longer runs and savepoint rollback is the whole guarantee. That holds on the WAL writer engine this combination gets from fix(assets): prevent SQLite concurrent writer failures #16358._discard_unreferenced_content's own docstring says it does not hold on a no-WAL engine and that the binding is not knowable from the call site, which makes this a question about fix(assets): harden the asset catalogue's failure paths #16393 standing alone rather than about the combination. Closing it properly means per-spec compensation, which neither PR writes, so it was flagged rather than invented here.tick_watch_listsplit above is a judgment call. It was resolved toward fix(assets): harden the asset catalogue's failure paths #16393, and fix(assets): prevent SQLite concurrent writer failures #16358's queue-drain characterization test — which documented the pre-hardening behaviour of an unreadable watched file propagating out of the tick — was rewritten to document the new behaviour rather than deleted.1adfd4b5) so they read apart from the resolution itself.Tests
ruff checkis clean onapp/assets,app/databaseand the touched test trees.The four excluded targets are a gap in local verification, not a claim about the branch: they fail identically on
masterin the same environment, which is what pins it on the environment rather than on this merge. CI on this PR is the real check for them.🤖 Generated with Claude Code