Skip to content

fix(rpc-client): retry transient 'upstream not synced' RPC errors - #563

Open
SQD-Trevor-Agent wants to merge 1 commit into
masterfrom
alert-fix/jnodxt-dump-not-synced-retry
Open

fix(rpc-client): retry transient 'upstream not synced' RPC errors#563
SQD-Trevor-Agent wants to merge 1 commit into
masterfrom
alert-fix/jnodxt-dump-not-synced-retry

Conversation

@SQD-Trevor-Agent

Copy link
Copy Markdown

Cause (proven)

dump-avalanche-mainnet-0 and dump-binance-testnet-0 (image subsquid/evm-dump:ac90a36d, namespace evm-archive) are crash-looping — 1051 restarts each, exit code 1. The last line before every exit is a single fatal:

sqd:evm-dump  RpcError: gave up retrying on network-level after 1.48s: 1 upstream not synced
  at validateError (evm-rpc/lib/rpc.js:280)
  method=eth_getBlockReceipts params=[0x5a45347]  (block 94655303)
  ErrUpstreamBlockUnavailable: upstream does not have the requested block yet
  blockNumber=94655303 latestBlock=94655301 finalizedBlock=94655304

Both dumps talk to the shared erpc proxy. erpc momentarily reports a finalized head (94655304) ahead of the upstream node's own latest (94655301). The dumper reads that finalized head, requests receipts for a block within it, and erpc answers with a JSON-RPC -32503/-32603 "upstream not synced" error inside a 200 response.

Unlike a 429 (which arrives as an HttpError and is already retried), this error is a plain RpcError, so RpcClient.isConnectionError returns false, the error is not retried (despite retryAttempts being effectively infinite), it propagates out of the ingest loop, and Dumper.run's handler logs it fatal and the process exits. The block becomes available within a couple of seconds — this is a transient condition that should be retried, not a fatal crash. The fatality is the bug (a single upstream hiccup should not crash-loop the dump).

Fix

Classify the transient "upstream not synced" / "does not have the requested block yet" RpcError as retryable in RpcClient.isConnectionError, mirroring the existing RetryError paths (e.g. Hyperliquid "invalid block height"). Retry machinery already backs off, so the dump waits out the brief head/tip skew instead of crashing. The classification logic is extracted into an exported pure isRetryableError() so it is unit-testable.

Tested

  • util/rpc-client vitest suite green (12/12). New client.test.ts asserts the real erpc error shape (1 upstream not synced, plural, and does not have the requested block yet) is retryable, and that a genuine execution reverted is not.
  • Regression test is a true red→green: with the new matcher removed, 3 tests fail; with it, all pass.
  • rush build --to @subsquid/rpc-client and --to @subsquid/evm-dump both compile clean.

Falsification

If dump-avalanche-mainnet-0 / dump-binance-testnet-0 keep exiting with exitCode 1 on ... : N upstream(s) not synced after this image ships, the classification did not catch the wording — capture the exact RpcError.message and widen the matcher. Separately, the underlying erpc/Alchemy anomaly (finalized head ahead of upstream latest) should be raised with whoever manages the erpc config; this change stops it from crash-looping the dump but does not fix the proxy inconsistency.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant