fix: rotate stalled active gossip sync peers#1553
Open
jjyr wants to merge 2 commits into
Open
Conversation
Abort an active gossip sync after repeated pending dependency results so network maintenance can assign another peer. Reset the pending counter whenever the cursor advances and cover the missing-dependency case with a regression test.
jjyr
marked this pull request as ready for review
July 13, 2026 11:25
Require active-sync cursors to advance monotonically before resetting the stall budget. Count pending results, non-advancing validated cursors, and request timeouts through the same bounded peer-rotation path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Why
An active sync peer can keep a limited sync slot indefinitely by returning messages whose dependencies are unavailable, replaying already-known messages that do not advance the cursor, or never answering requests. The receiver previously retried the same peer without a bounded no-progress policy.
Implementation
Only a cursor strictly greater than the current cursor is accepted as progress. Production builds allow 20 consecutive no-progress events; tests use a smaller threshold while exercising the same state transition. Once the threshold is reached, the syncing actor emits the existing
ActiveSyncingAbortedevent and stops, allowing normal maintenance to select another peer without banning it.The production request timeout remains 20 seconds, so a completely silent peer is bounded at approximately 400 seconds before rotation.
Testing
RUST_LOG=off cargo nextest run -p fnn --features rocksdb active_sync(10 passed)cargo fmt --all -- --checkcargo clippy --all-targets --features rocksdb -p fnn -- -D warnings