Remote: stop retrying oversized gRPC messages

[blico-v2]

Summary

• Detect grpc-go, grpc-java, and gRPC C++ message-size rejections in remote gRPC calls.
• Stop retrying only when the error matches an exact, anchored gRPC message-size format.
• Preserve retries for quota, overload, and all other RESOURCE_EXHAUSTED failures.
• Preserve the original gRPC failure and exclude the deterministic rejection from remote-cache circuit-breaker failures.

Motivation

When a gRPC message exceeds the receiver's configured limit, the receiving gRPC implementation may reject it before the application handler runs. Retrying the unchanged message cannot succeed: neither the message bytes nor the receiver's fixed limit changes between attempts. The retries therefore add backoff delay without doing useful work.

gRPC deliberately standardized framework message-size failures as RESOURCE_EXHAUSTED in grpc/grpc#10612, the same status applications use for potentially transient failures such as quota exhaustion. The protocol provides no structured subtype that distinguishes these cases; this ambiguity and the lack of a typed alternative were noted in the original discussion.

This keeps the existing general treatment of RESOURCE_EXHAUSTED as transient. It recognizes only exact, anchored grpc-go, grpc-java, and gRPC C++ message-size formats, avoiding confusion with quota or descriptive server errors. Rust Tonic reports the corresponding declared-size failure as OUT_OF_RANGE, which Bazel already treats as non-retryable.

The classification is centralized in RemoteRetrier and preserves the original gRPC status, description, and trailers.

The circuit-breaker treatment follows the precedent discussed in #18583 and implemented in #18662: deterministic request-size failures do not indicate an unhealthy remote-cache service.

This change does not make the RPC succeed; the receiver's limit must be raised or the message reduced. It only avoids retrying a message that cannot succeed under the current limit.

Tests

• grpc-go, grpc-java, and gRPC C++ message-size formats fail after one attempt.
• malformed and descriptive near-matches are not recognized.
• ordinary RESOURCE_EXHAUSTED remains retryable.
• deterministic cache request failures do not count against the circuit breaker.

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[bazel-docs-pr-commenter]

✅ Bazel docs preview is ready!

Preview URL: https://bazel-pr-30716.mintlify.app/

Updated for 403281e2d49ff8a978fc39bf02ce405e6d5e9d38

[dzbarsky]

It seems weird to special case go and java implementations. Why not rust or c++? It's also odd that we have typed statuses to decide what to do but are instead sniffing for strings. Perhaps grpc should distinguish between a framework resource exceeded (message size) or an application one (replenishing quota)?

[blico-v2]

Perhaps grpc should distinguish between a framework resource exceeded (message size) or an application one (replenishing quota)?

grpc/grpc#10612

Ultimately, I think this is an unfortunate consequence of having a fixed set of response codes, especially given that the same set of codes is returned by both gRPC and the application. I would not recommend that applications depend on the human-readable text strings, because there is no guarantee that those won't change over time. But I don't have a better solution to recommend.

:(

It seems weird to special case go and java implementations. Why not rust or c++?

Added implementation for C++. Rust is not included because its receive-size enforcement incidentally returns OUT_OF_RANGE (grpc/grpc-rust#1274)

[dzbarsky]

Perhaps grpc should distinguish between a framework resource exceeded (message size) or an application one (replenishing quota)?

grpc/grpc#10612

Ultimately, I think this is an unfortunate consequence of having a fixed set of response codes, especially given that the same set of codes is returned by both gRPC and the application. I would not recommend that applications depend on the human-readable text strings, because there is no guarantee that those won't change over time. But I don't have a better solution to recommend.

:(

It seems weird to special case go and java implementations. Why not rust or c++?

Added implementation for C++. Rust is not included because its receive-size enforcement incidentally returns OUT_OF_RANGE (grpc/grpc-rust#1274)

I've never seen someone use grpc-rust, tonic is what the OSS world mostly uses AFAIK. Which is exactly my point, it doesn't really make sense to enumerate the quirks of a fixed set of implementations :)

[blico-v2]

I've never seen someone use grpc-rust, tonic is what the OSS world mostly uses AFAIK. Which is exactly my point, it doesn't really make sense to enumerate the quirks of a fixed set of implementations :)

grpc-rust is tonic’s current repository (formerly hyperium/tonic); #1274 changes Tonic’s decoder directly.

That being said, I agree this workaround is not ideal. It is the most conservative approach I found, but I’m open to alternatives.