Repository navigation
feat(Spanner.V1): Add operation metrics on unary RPCs for built in metrics - #15807
robertvoinescu-work merged 2 commits into
Conversation
51dee39 to
6892696
Compare
There was a problem hiding this comment.
Code Review
This pull request updates the Renovate configuration to include the Google Cloud CommerceProducer V1Beta API, bumps dependency versions for SharpCompress and LibGit2Sharp, and updates package descriptions in the README. Additionally, it introduces a large number of auto-generated code snippets for various services under the Google Ads AdManager V1 API, including AdRule, AdSpot, CdnConfig, ContentBundle, CreativeSet, CustomTargetingValue, LiveStream, and Order service clients. There are no review comments to address, and I have no feedback to provide.
c660e31 to
0ecadfd
Compare
0ecadfd to
f26d752
Compare
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces a BuiltInMetricsWrapper for SpannerClient to intercept API calls and record built-in metrics, along with corresponding unit tests and builder integration. The review feedback highlights several important improvements: overriding ClientIdentity in the wrapper to delegate to the inner client, addressing metric inaccuracies for lazy-loaded paged operations (ListSessions), ensuring builder interceptors do not overwrite existing client/task instances, and asserting on exception messages in unit tests to ensure correct propagation.
| public override PagedEnumerable<ListSessionsResponse, Session> ListSessions(ListSessionsRequest request, CallSettings callSettings = null) | ||
| => ExecuteWithInstrumentation(nameof(ListSessions), request, () => _inner.ListSessions(request, callSettings)); | ||
| public override PagedAsyncEnumerable<ListSessionsResponse, Session> ListSessionsAsync(ListSessionsRequest request, CallSettings callSettings = null) | ||
| => ExecuteWithInstrumentation(nameof(ListSessionsAsync), request, () => _inner.ListSessionsAsync(request, callSettings)); |
There was a problem hiding this comment.
The ListSessions and ListSessionsAsync methods return lazy PagedEnumerable and PagedAsyncEnumerable instances. The actual gRPC calls to the Spanner service are only executed when these enumerables are iterated.
Because ExecuteWithInstrumentation only wraps the initial call that constructs the enumerable, the recorded metrics (latency and status) will only reflect the negligible time taken to instantiate the lazy enumerable (typically ~0ms with OK status), rather than the actual network calls made during iteration.
Consider documenting this limitation, or if accurate metrics are required for paged operations, wrapping the returned enumerables to intercept the actual page-fetching RPCs.
There was a problem hiding this comment.
This is to spec leaving as is.
There was a problem hiding this comment.
Then do we need to wrap them here? There's no Spanner latency or anything like that we'd be recording. Nor are we encompassing the whole pagination operation.
There's some guidance on how to do this for golden signals I think, maybe we should follow that here as well.
Happy to punt these paginated methods for a later PR while we figure it out.
There was a problem hiding this comment.
I will revisit with spanner, but looking more at other languages I see they do not emit operation metrics for these methods. I've made these methods purely pass through for now.
f26d752 to
e6d2907
Compare
efevans
left a comment
There was a problem hiding this comment.
Looks good, had one question and a few code suggestions
e6d2907 to
51e2899
Compare
amanda-tarafa
left a comment
There was a problem hiding this comment.
Overall good with a few change requests. When we get to GAPIC observability I'd really like to look into improving this implementation.
| partial void InterceptBuild(ref SpannerClient client) | ||
| { | ||
| client ??= MaybeCreateEmulatorClientBuilder()?.Build(); | ||
| client ??= Build(); |
There was a problem hiding this comment.
Please call BuildImpl here and wrap the result on your client. That way, if we change the implementation of BuildImpl you don't have to make changes to this code. For instance, right now you need to add a call to Validate on the wrapping method. If in a month we add a "log client creation" call on BuildImp for all GAPICs, then you'll have to come here and change this code to add that.
There was a problem hiding this comment.
Done, but there's a cascading effect to this change as there's no way to share the client Id between the interceptor and client except through a side channel - this was making me lean towards this initially, but let me know what you think about have an async local build context instead - I've added a comment to highlight it.
| private SpannerClient BuildWithMetrics(CallInvoker callInvoker) | ||
| { | ||
| Validate(); | ||
| var identity = Labeler.GenerateIdentity(); |
There was a problem hiding this comment.
This I don't understand.
There was a problem hiding this comment.
Ah, is it because the attempt interceptor needs it? So, it's the attempt interceptor and the client wrapper that need it. can you just remove it from SpannerClient?
There was a problem hiding this comment.
Correct, removed.
| public override SpannerSettings Settings => _inner.Settings; | ||
|
|
||
| ///<inheritdoc/> | ||
| internal override ClientIdentity ClientIdentity => _clientIdentity; |
There was a problem hiding this comment.
I think you just can remove ClientIdentity from SpannerClient.
| /// A delegating wrapper around <see cref="SpannerClient"/> that intercepts | ||
| /// execution of all methods to record metrics. | ||
| /// </summary> | ||
| internal class BuiltInMetricsWrapper : SpannerClient |
There was a problem hiding this comment.
Because we cannot extend SpannerClientImp what will happen when Spanner adds a new method is not that we won't instrument it, it's that it will be unusable because the client we are returning will throw Unimplemented. We need to write a unit test for this that via reflection checks that all Unimplemented methods in SpannerClient are implemented in BuiltInMetricWrapper.
There was a problem hiding this comment.
I've added a unit test that explicitly states which methods need to be implemented.
| private static StatusCode GetStatus(Exception ex) => ex switch | ||
| { | ||
| RpcException rpcEx => rpcEx.StatusCode, | ||
| OperationCanceledException => StatusCode.Cancelled, |
There was a problem hiding this comment.
Hmmm. I think we are not doing this in the attempt, and I'm not sure we should do it here either. We probably care about RPC errors only?
There was a problem hiding this comment.
Removed the operation cancelled exception casting. And moved this method into the base class so it can be used here and in the interceptor.
| private static readonly Exception s_rpcException = new Grpc.Core.RpcException(s_timeoutStatus); | ||
|
|
||
| [Fact] | ||
| public void Properties_DelegateToInner() |
There was a problem hiding this comment.
We need to this via reflection and for methods as well.
There was a problem hiding this comment.
Good point. I added a test to make sure that the methods are also wired up correctly - checking for eventual delegation to the inner client..
1045e36 to
e9b4d21
Compare
b76c42d to
593c136
Compare
amanda-tarafa
left a comment
There was a problem hiding this comment.
Looking good. I have a suggestion for new tests that may come in a later PR, and a couple small things. Thanks!
| private BuiltInMetricsWrapper(SpannerClient inner, ClientIdentity? clientIdentity = null) | ||
| { | ||
| _inner = GaxPreconditions.CheckNotNull(inner, nameof(inner)); | ||
| _clientIdentity = clientIdentity ?? Labeler.GenerateIdentity(); |
There was a problem hiding this comment.
Let's instead check that the identity is not null?
There was a problem hiding this comment.
Done, I mostly had this for convenience in the tests, but it's not really saving much lines of code there.
| partial void InterceptBuild(ref SpannerClient client) => client = MaybeCreateEmulatorClientBuilder()?.Build(); | ||
| partial void InterceptBuild(ref SpannerClient client) | ||
| { | ||
| client = MaybeCreateEmulatorClientBuilder()?.Build() ?? Build(); |
There was a problem hiding this comment.
I'm sorry, I know I suggested we didn't wrap the emulator client earlier. But thinking about it more, I think that the emulator client call invoker will have the attempt interceptor and there's no easy way to get rid of that. So maybe we should just wrap it here for completeness. I still think it's weird to have these metrics for the emulator, but it'd be weirder to have them partially. And in any case, this is all opt-in.
What do you think?
There was a problem hiding this comment.
I like having as much consistency between emulator and non emulator makes things a bit easier to follow - if it wasn't opt in I might lean the other way. On another note if users want to consume these metrics it might provide an easy onramp for them if they can get these in an emulated environment first. I moved to wrap everything consistently.
| } | ||
|
|
||
| [Fact] | ||
| public async Task Builder_ClientIdentity_MatchesBetweenInterceptorAndWrapper_Sync() |
There was a problem hiding this comment.
Left a comment earlier. This test is good and tests the happy case. But let's add a couple other tests for checking this same thing when the builder is used to build several clients at the same time. We can build enough clients (20. 50?) to make sure that it will at least flake if we are doing something wrong or if we remove the async local.
There was a problem hiding this comment.
This is a good point, I've added more tests to cover all the cases mentioned: multiple clients run a mix of sync and async builds.
593c136 to
aed8b7b
Compare
| builder.InitializeLocalBuildContext(); | ||
| try | ||
| { | ||
| var rawClient = builder.BuildImpl(); |
There was a problem hiding this comment.
NOTE: Both emulated and non-emulated now use BuildImpl whereas before the emulator used Build. This makes the code a little cleaner by keeping things symmetric. It behaves effecitvely the same, but wheras before we would go through the MaybeCreateEmulatorClientBuilder a second time returning null and treating it like a normal client builder from that point on we now just avoid that recursion immediately going to BuildImpl.
| var tasks = new List<Task<BuiltInMetricsWrapper>>(); | ||
| for (int i = 0; i < clientCount; i++) | ||
| { | ||
| if (buildMode == BuildMode.AsyncOnly || (buildMode == BuildMode.Mixed && i % 2 == 0)) |
|
Integration Tests for Spanner.V1 and Spanner.Data pass, merging. |
b/404948213