feat(seidb): add exact-version state store snapshots - #3919
Open
blindchaser wants to merge 10 commits into
Open
Conversation
Add opt-in Pebble checkpoints for State Store with ordered queue barriers, crash-safe publication, retention, metrics, and SC-aligned cadence. Preserve honest per-database version bounds across snapshot and prune races, including split EVM stores. Co-authored-by: Cursor <cursoragent@cursor.com>
|
The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #3919 +/- ##
==========================================
- Coverage 59.48% 58.61% -0.88%
==========================================
Files 2325 2242 -83
Lines 198660 190263 -8397
==========================================
- Hits 118180 111519 -6661
+ Misses 69240 68140 -1100
+ Partials 11240 10604 -636
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2a2ca0794e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
![seidroid[bot]](https://avatars.githubusercontent.com/in/2745135?s=60&v=4){kind=small}
There was a problem hiding this comment.
A large but carefully built, default-off feature: ordered in-queue checkpoint barriers give exact snapshot labels, publication is staged/renamed/symlinked with startup recovery, and the config plumbing follows the repo's characterization conventions. No blocking correctness or security problems found; the notes below are documentation, test-quality, and edge-case robustness items.
Findings: 0 blocking | 9 non-blocking | 5 posted inline
Blockers
- None at the file/PR level.
Non-blocking
CosmosStateStore.WaitForPendingWritesandEVMStateStore.WaitForPendingWritesare new exported methods with no production caller — they exist solely forsnapshot_test.go'ssettlehelper. Anexport_test.goshim or an unexported hook would keep the public store surface free of test-only plumbing.sei-cosmos/storev2/rootmultinow importssei-db/state_db/ss/compositeonly to holdvar _ stateStoreSnapshotScheduler = (*sscomposite.CompositeStateStore)(nil), coupling the Cosmos store layer to a concrete SS implementation. The same build-time protection could live in packagess(or as a method onseidbtypes.StateStore), keeping the dependency one-directional.- The trigger sits in
flush(), which is reached first fromGetWorkingHash()— i.e. beforeCommit. A boundary snapshot can therefore be published for a height the node has not yet committed (SS is already ahead in that window, so the image is self-consistent). Worth confirming this is acceptable for the rollback restore model, since exact labelling is the point of the barrier. - 5 suggestion(s)/nit(s) flagged inline on specific lines.
- 1 non-blocking pre-existing issue(s) listed below under pre-existing issues.
Pre-existing issues
- [suggestion]
sei-cosmos/storev2/rootmulti/store.goempty-block path (~line 276) callsssStore.SetLatestVersion(currentVersion)synchronously while older changesets may still sit in the async apply queue.ApplyChangesetSyncpersists the latest-version marker inside its own batch (NewBatch(..., version, ...)), so a later-applied older batch can overwrite the on-disk marker and regress the SS watermark across a crash/restart.
Reject historical SS queries below the highest member floor before they can return partial routed state. Keep the snapshot referenced by current during retention so a failed link update cannot leave a dangling target. Also detach the snapshot design overview from exported const godoc and make recovery mismatch tests assert the tolerated highest-floor contract. Co-authored-by: Cursor <cursoragent@cursor.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
The composite read gate turned a prune that races an in-flight historical query into a process crash: the cosmos KVStore wrapper over a StateStore panics on any read error, and a query store keeps reading at its fixed version after the floor moved. Enforce the floor only where an error can be returned to the caller (query-store construction, Query, VersionExists) and let a pruned member report absence, which is what its engine already does. Co-authored-by: Cursor <cursoragent@cursor.com>
Cosmos SS and EVM SS now own their snapshot roots, staging, current links, and retention, while the composite layer only coordinates cadence and best-effort joint publication. This prepares future GC wiring by adding ExternalPruning stand-down without implementing gc.PrunableStore before rollback can report an honest floor. Co-authored-by: Cursor <cursoragent@cursor.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
There are 3 total unresolved issues (including 2 from previous reviews).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 6839692. Configure here.
PruneSnapshots is the entry point an external collector prunes an SS through, and gc.PrunableStore only calls it when ExternalPruning reports true. Standing it down in that mode made it a silent no-op, so snapshots would grow without bound once the collector owned retention. Only internal count-based retention stands down now, which is what ExternalPruning is for. Co-authored-by: Cursor <cursoragent@cursor.com>
A member that could not be staged left barriers queued for the members before it, while the coordinator released the version for another attempt. The retry then cleared the staging directory the queued barrier was writing into, and published whatever the interrupted checkpoint had reached under a label claiming to hold every version up to it. Staging is now split: every member reserves its directory before any barrier is queued, so a request either queues all of them or none. Preparing also refuses a staging directory that already exists rather than clearing it, since a checkpoint may still be writing there; Open clears the ones a crash left behind. A failed checkpoint now aborts the staging directories of its peers instead of leaving them until the next startup. Co-authored-by: Cursor <cursoragent@cursor.com>
Each member counts only its own snapshot directories, so an unpaired newer height took the keep slot the newest shared height occupied and retention deleted it, in Open and after every publish. A restore reads the newest height every member holds, so that left nothing to restore from until the next fully paired publication. The coordinator now hands the members a shared floor — the newest height they all hold — which retention keeps regardless of the keep window, and republishes it after every publication. It is resolved before the members open, because opening one runs its retention. Also: the hardlink preflight probe takes a fixed name and is cleared on both sides before use, so a crash mid-probe leaves one known path per directory instead of accumulating; and pebbledb reports consecutive prune failures, since every pass raises the earliest-version marker before it deletes and repeated failures narrow the served range while nothing leaves disk. Co-authored-by: Cursor <cursoragent@cursor.com>
Each of these was written twice by the snapshot work, so a later change had two places to keep in agreement: - the fan-in over parallel checkpoints, now management.FanIn - the checkpoint capability test, now management.SupportsCheckpoint - the retention delete loop, now Manager.removeSnapshots - the prune marker prologue, now beginPrunePass/endPrunePass, which also holds the skip-baseline reasoning that four comments carried - the "-snapshots" sibling root, now one path helper - the pending-write capability, now types.PendingWriteWaiter PruneSnapshots takes publishMu, because publication renames a directory in and swaps current under it, so a cut line resolved a moment earlier could otherwise delete what current now names. Retention attempts every candidate and joins the errors, which is what the count-based path already did. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add opt-in, exact-version Pebble checkpoints for State Store. This PR contains snapshot creation and retention only; SS rollback remains separate and will use these checkpoints as state-WAL replay inputs.
currentsymlink, with startup recovery and retention.Snapshots require Pebble and hardlinks. All enabled Cosmos and EVM SS databases must share a filesystem with the snapshot root. SS snapshots are internal rollback restore points, not state-sync or archive inputs.
Test plan
go test -race -count=1 ./sei-db/db_engine/pebbledb/... ./sei-db/config/... ./sei-db/state_db/ss/... ./sei-cosmos/storev2/rootmulti/... ./app/... ./sei-cosmos/server/config/...golangci-lint run ./sei-db/... ./app/... ./sei-cosmos/storev2/... ./sei-cosmos/server/config/...Made with Cursor