Snapshot Engine + flatKV integration: phase 1

[cody-littley]

Describe your changes and provide context

Integrate flatKV and the new snapshot engine.

As part of the pipelining refactor, our goal is to move three things off of the main execution thread:

• hashing
• pebble checkpointing (i.e. on-disk snapshots)
• flushing data down into pebble

In order to limit the size of this PR, this PR only moves flushing of data off the main thread, although it does lay the groundwork for moving hashing and checkpointing off-thread.

[cursor]

PR Summary

High Risk
Touches commit-store flush semantics and removes a DB caching layer while changing snapshot sealing/finalization and iterator concurrency—core state-commit paths where mistakes can corrupt or fork chain state.

Overview
Moves async flush plumbing toward the snapshot engine and tightens rootmulti so each block height hands changesets to the commit store at most once, matching baseapp’s multiple working-hash / flush calls per height. Late changesets after the working hash was taken now fail fast instead of advancing the commit store to a phantom block.

Removes the Pebble dbcache layer (OpenWithCache, sharded LRU, metrics, cached batches). Pebble opens directly; read-through caching lives in the snapshot engine instead. Pebble tests drop the cached/uncached dual mode.

Snapshot engine API/lifecycle refactor: SetHash / AwaitHash / InitialHash become Finalize(writes) (metadata under a ReservedPrefix), iterators take IterOptions, return dbm.Iterator, and no longer block writes (fixed view at creation with pinned DB reads). Adds Name(), EscapeHatchUnderlyingDB(), and documents that Close owns the underlying DB. Shard comments drop iterator-based write refusal.

ABCI tests for migration batch size split: one test drives applyMigrationBatchSize directly for “param mid-block vs next BeginBlock”; another uses FinalizeBlock + Commit when the param is already in state at block start.

^{Reviewed by Cursor Bugbot for commit c44b7dd. Bugbot is set up for automated code reviews on this repo. Configure here.}

[github-actions]

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
✅ passed	✅ passed	✅ passed	✅ passed	Aug 12, 2026, 6:19 PM

[yzang2019]

What does out of service mean for a shard? When would a shard become out of service?

[cody-littley]

Two scenarios qualify as "out of service":

• The shard is shutdown. Only reachable if you call shutdown concurrently with an operation that touches a shard (which is illegal).
• The shard has crashed. Only possible when the DB experiences a crash. DB failures are fatal, so we should never try to recover them.

Updated the doc to spell this out.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit c44b7dd. Configure here.}

[blindchaser]

is looks RootHash commits the pending block. then later composite commit increments FlatKV again? is there a double commits on flatkv?

[cody-littley]

This is intentional on my part. The core problem is that comsos wants the hash before the commit, but the proper shape of the storage engine is to provide a hash only after the block is finalized+committed.

I've set it up so that if we call Commit() multiple times for the same block number, later commits become no-ops. If you try to write changes to the block after the commit, you get an error. Previously it was technically possible to write changes after we get the hash but before the commit, but this would either mean the hash is actually an invalid hash for that block.

Worth a discussion in parking lot.

[blindchaser]

Config.Fsync is not passed to the five snapshot-engine FlushSync fields. Since FlushSync defaults to false, programmatic callers that set Fsync=true still get unsynchronized normal flushes. can we propagate this value to all five store configurations and add a test.

[cody-littley]

good point, fixed

[yzang2019]

Shall we just remove the metadata DB?