Remove metadata DB - #3928
Conversation
PR SummaryHigh Risk Overview Composite / read paths: Tooling & layout: Snapshot/working layout is account/code/storage/misc only; flat-layout migration and metadata-related config/tests are removed. Reviewed by Cursor Bugbot for commit 7649c18. Bugbot is set up for automated code reviews on this repo. Configure here. |
|
The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).
|
![seidroid[bot]](https://avatars.githubusercontent.com/in/2745135?s=60&v=4){kind=small}
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #3928 +/- ##
==========================================
- Coverage 59.48% 58.79% -0.70%
==========================================
Files 2325 2254 -71
Lines 198660 191385 -7275
==========================================
- Hits 118180 112522 -5658
+ Misses 69240 68152 -1088
+ Partials 11240 10711 -529
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
| miscDir := filepath.Join(dir, workingDirName, miscDBDir) | ||
| if _, err := os.Stat(miscDir); err != nil { |
There was a problem hiding this comment.
why do we only look at miscDB for getting LatestVersion? Shall we look at the minimum of four?
| if meta.CommittedVersion != s.committedVersion { | ||
| return fmt.Errorf( | ||
| "flatkv: %s is at version %d but the store is at %d; this store holds a block "+ | ||
| "its write-ahead log lost, which no replay can reconcile (restore from a snapshot)", |
There was a problem hiding this comment.
SetInitialVersion writes the same watermark into four separate databases with four separate commits, and the new requireAlignedDataDBs guard refuses to open a store whose four watermarks disagree — so a crash in the middle of that loop produces exactly the state the guard rejects, and the code that would repair it lives after the load that now fails.
| } | ||
|
|
||
| syncOpt := types.WriteOptions{Sync: s.config.Fsync} | ||
| for _, ndb := range s.namedDataDBs() { |
There was a problem hiding this comment.
SetInitialVersion writes each data DB's watermark in its own separate durable batch — four independent Pebble commits, no atomicity across them. What if the node crash in between?
| func (s *CommitStore) replayIntoMutableStore(targetVersion int64) (err error) { | ||
| // It runs at startup (open/openTo) and during Rollback, never concurrently with live commits. | ||
| func (s *CommitStore) replayIntoMutableStore(targetVersion int64) error { | ||
| if err := s.catchUpFromWAL(targetVersion); err != nil { |
There was a problem hiding this comment.
First, replay bypasses the contiguity check that normally protects the watermark. Commit refuses to go backwards — it enforces version != s.committedVersion+1. But replay doesn't go through Commit; it calls commitBatches directly.
Second, prepareBatch only skips a DB when it has both no writes and is already at or past the version. The && is what matters here — if the replayed block touches this DB at all, the batch is built even though the DB is ahead.
Third, the batch stamps the replayed version unconditionally. There is no "only move forward"
So a DB sitting at version 3 that gets block 2 replayed into it is written back down to version 2.
The check has to happen before catchup, and it has to compare against the WAL rather than against the other DBs. Comparing DBs to each other pre-catchup would be wrong, because a plain torn commit legitimately leaves them disagreeing and that case is fully recoverable.
2 participants
Describe your changes and provide context
Remove the metadata DB.