fix(evm-rpc): retry eRPC upstreams-exhausted errors instead of crashing the dumper

[elina-chertova]

Symptom

Alert linea-mainnet_No_Dumper_Data (evm-archive, kind=dump). dump-linea-mainnet-0 is in CrashLoopBackOff (140 restarts / 27h); no raw data written.

Root cause (proven from pod logs)

The linea-mainnet dumper was routed through eRPC on 2026-07-22 (infra commit 248d45930 "Roll out eRPC to almost every archive ingester"; the dump pod's 27h age matches). eRPC's only primary upstream for linea is Alchemy, which is currently degraded — live probes return HTTP 503 -32001 "Unable to complete request at this time" and 429 "compute units per second exceeded" for linea.

eRPC does not fail over to the healthy dwellir/uniblock fallbacks, and returns the request as an HTTP 200 body carrying a JSON-RPC error whose code is normalized to -32601 ("method eth_getBlockByNumber does not exist/is not available") with data.code: ErrFailsafeRetryExceeded → cause.code: ErrUpstreamsExhausted.

EvmRpcClient.isConnectionError did not recognize this shape, so the error was fatal (dump log level 5) and the process exited:

RpcError: The method eth_getBlockByNumber does not exist/is not available
  at EvmRpcClient.receiveResult (/squid/util/rpc-client/lib/client.js:376)

Before the eRPC switch the same underlying Alchemy 503 arrived as a raw HTTP 503, which the client already retries — so eRPC repackaging it as a fatal JSON-RPC error is what turned a recoverable degradation into a crash-loop.

Fix

Recognize eRPC's stable data.code (ErrUpstreamsExhausted / ErrFailsafeRetryExceeded) as a retryable connection error in EvmRpcClient.isConnectionError. This restores parity with a direct provider 5xx: the dumper backs off and retries (retryAttempts=MAX) instead of crash-looping, so a single provider hiccup can no longer take the dump down. A genuine -32601 with no eRPC data stays fatal.

Test

evm/evm-rpc/test/rpc-client.test.ts — asserts the production error shape is retryable and a real method-not-found is not. Red on pre-fix (2 fail), green after.

Falsification

If the dump still crash-loops after deploying an evm-dump built from this change with the same -32601/ErrUpstreamsExhausted fatal line, the classifier isn't matching the real payload shape.

Note (operator action, not in this PR)

This stops the crash-loop but does not by itself restore linea data flow: Alchemy is down for linea while eRPC isn't failing over. An operator should route linea to a healthy provider (dwellir + uniblock both return 200 for linea now) or fix the eRPC failover, and raise the Alchemy linea 503/429 with the provider (https://status.alchemy.com). Provider swaps are an operator mitigation, so they're intentionally not included here.

[elina-chertova]

Related open PRs harden EvmRpcClient/rpc-client against other transient eRPC/aggregator signatures that crash-loop the dumper: #501 (-32503), #505 (-32020 response too large), #520 (Cloudflare 521-524), #513 (invalid block height on finalized-head probe). This one covers a distinct signature — eRPC's data.code ErrUpstreamsExhausted/ErrFailsafeRetryExceeded, normalized to a misleading -32601. They could be consolidated into one error-classification pass if preferred.

[SQD-Trevor-Agent]

Bumping this: the exact crash this PR fixes is currently firing fleet-wide — a correlated cluster of ~34 alerts across 19 Alchemy-primary EVM archive networks (plasma-testnet/mainnet, soneium, avalanche-mainnet, sonic, mantle, opbnb-*, sei, base-sepolia, optimism-sepolia, robinhood-*, binance-testnet, tempo-moderato, hyperliquid, …), as *_Dumper_Pod_Restarts / *_No_Dumper_Data / *_Ingest_Behind_Head. Trigger: the ongoing Alchemy major incident "Elevated API Errors Across Multiple Networks" (https://status.alchemy.com, started 2026-09-03 15:41 UTC). eRPC excludes the throttled Alchemy primary and the fatal-looking error crash-loops the dumpers (e.g. dump-plasma-testnet-0: 197 restarts / 24h, reason=Error).

Confirmed against live pod logs that this PR's data.code classifier covers the observed shapes:

• plasma-testnet fatal: RpcError: method not found code:-32601, data.code:"ErrFailsafeRetryExceeded", cause.code:"ErrUpstreamsExhausted" ("1 upstream unsupported method", from failover to a non-archival uniblock upstream lacking eth_getBlockReceipts). ✅ matched by isErpcUpstreamsExhaustedError.
• avalanche-mainnet fatal: RpcError: gave up retrying on network-level after 1.14s: 1 upstream not synced (ErrFailsafeRetryExceeded). ✅ matched.

One sibling variant I did not see a data.code on — worth a look before merge:

• soneium-mainnet fatal: RpcError: cannot parse json-rpc response: "Syntax error no sources available, the input json is empty" (from rpc.js:120). This looks like an empty/parse response when eRPC has no usable upstream; if it arrives without data.code it would still be fatal after this fix. Could be handled here (or via RpcProtocolError retry) so the whole eRPC-degradation family is covered.

Given the fleet-wide impact, could we get this reviewed/merged and rolled to the archive dumpers? Happy to add the soneium parse-variant if you'd like it in the same PR.