fix(rpc-client): retry transient 'upstream not synced' RPC errors - #563
Open
SQD-Trevor-Agent wants to merge 1 commit into
Open
fix(rpc-client): retry transient 'upstream not synced' RPC errors#563SQD-Trevor-Agent wants to merge 1 commit into
SQD-Trevor-Agent wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cause (proven)
dump-avalanche-mainnet-0anddump-binance-testnet-0(imagesubsquid/evm-dump:ac90a36d, namespaceevm-archive) are crash-looping — 1051 restarts each, exit code 1. The last line before every exit is a single fatal:Both dumps talk to the shared
erpcproxy. erpc momentarily reports a finalized head (94655304) ahead of the upstream node's own latest (94655301). The dumper reads that finalized head, requests receipts for a block within it, and erpc answers with a JSON-RPC-32503/-32603"upstream not synced" error inside a 200 response.Unlike a
429(which arrives as anHttpErrorand is already retried), this error is a plainRpcError, soRpcClient.isConnectionErrorreturnsfalse, the error is not retried (despiteretryAttemptsbeing effectively infinite), it propagates out of the ingest loop, andDumper.run's handler logs itfataland the process exits. The block becomes available within a couple of seconds — this is a transient condition that should be retried, not a fatal crash. The fatality is the bug (a single upstream hiccup should not crash-loop the dump).Fix
Classify the transient "upstream not synced" / "does not have the requested block yet"
RpcErroras retryable inRpcClient.isConnectionError, mirroring the existingRetryErrorpaths (e.g. Hyperliquid "invalid block height"). Retry machinery already backs off, so the dump waits out the brief head/tip skew instead of crashing. The classification logic is extracted into an exported pureisRetryableError()so it is unit-testable.Tested
util/rpc-clientvitest suite green (12/12). Newclient.test.tsasserts the real erpc error shape (1 upstream not synced, plural, anddoes not have the requested block yet) is retryable, and that a genuineexecution revertedis not.rush build --to @subsquid/rpc-clientand--to @subsquid/evm-dumpboth compile clean.Falsification
If
dump-avalanche-mainnet-0/dump-binance-testnet-0keep exiting withexitCode 1on... : N upstream(s) not syncedafter this image ships, the classification did not catch the wording — capture the exactRpcError.messageand widen the matcher. Separately, the underlying erpc/Alchemy anomaly (finalized head ahead of upstream latest) should be raised with whoever manages the erpc config; this change stops it from crash-looping the dump but does not fix the proxy inconsistency.