rpc: route -32601 (method not found) to a capable endpoint instead of crashing - #81
Open
SQD-Trevor-Agent wants to merge 1 commit into
Open
rpc: route -32601 (method not found) to a capable endpoint instead of crashing#81SQD-Trevor-Agent wants to merge 1 commit into
SQD-Trevor-Agent wants to merge 1 commit into
Conversation
When an endpoint returns JSON-RPC error -32601 (method not found / not supported) for a request, and another endpoint in the pool can serve that method, mark the method unsupported on the failing connection and retry it there instead of propagating the error to the top-level coroutine, which crashes the whole ingest process. Previously a single endpoint that gated a method (e.g. a provider putting debug_traceBlockByHash behind a paid plan) would crash-loop the ingester even though a capable endpoint was configured.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
An eth-ingest process for a network with
traces: trueand two configured endpoints crash-looped in production (metis-mainnet). One endpoint (nodies) gates tracing methods behind a paid plan and returns-32601fordebug_traceBlockByHash; the other (Alchemy) serves it. When a trace request was scheduled onto nodies, theRpcErrorpropagated all the way up:-32601is not in_is_retryable_error's list, so_send's callback took theelsebranch and calleditem.future.set_exception(ex), which surfaced in the ingest coroutine and killed the whole process — even though a fully capable endpoint (Alchemy) was in the pool. The container then crash-looped every ~5 min and was never scraped.Change
In
RpcClient._send's done-callback, when the error is-32601and at least one other connection can handle the request, mark the method unsupported on the failing connection (newRpcConnection.unsupported_methods, consulted by_can_handlealongside the staticmissing_methods) and re-queue the item so it lands on a capable endpoint. If no other endpoint can serve it, behaviour is unchanged (the future still fails), so a genuinely-unsupported-everywhere method still surfaces loudly rather than spinning.This is the runtime, auto-detected complement to the existing operator-configured
--endpoint-missing-method/missing_methodsmechanism.Test
tests/test_rpc_client.py(new): two endpoints viahttpx.MockTransport, the first returns-32601fordebug_traceBlockByHash, the second a valid result. Asserts the call resolves via the capable endpoint and that the method is recorded in the failing connection'sunsupported_methods.await client.call('debug_traceBlockByHash', ...)raisesRpcError(-32601)→ FAILED (reproduces the crash).Run:
python -m unittest tests.test_rpc_client.Falsification
Wrong if re-queuing could loop: it can't — the failing connection is excluded via
unsupported_methodsbefore re-push, and re-push only happens when another connection already satisfies_can_handle(so_reg_in_queue's handler assertion holds). If every endpoint returns-32601for a required method, the future still fails as before.