[WIP] Thin-client endpoint probe drill-test: probe + HPK + QueryPlan + HTTP/2 ping + perf/Kusto tooling#49725
Draft
jeet1995 wants to merge 142 commits into
Draft
[WIP] Thin-client endpoint probe drill-test: probe + HPK + QueryPlan + HTTP/2 ping + perf/Kusto tooling#49725jeet1995 wants to merge 142 commits into
jeet1995 wants to merge 142 commits into
Conversation
… QueryPlan proxy routing Add RNTBD token mappings for x-ms-cosmos-supported-query-features (0x002B) and x-ms-cosmos-query-version (0x002C) so the thin client proxy can read these values from the RNTBD body when processing QueryPlan requests. Previously these headers were only set as HTTP headers by QueryPlanRetriever and were lost when QueryPlan was routed through the proxy path, since ThinClientStoreModel serializes requests as RNTBD (not HTTP headers). IDs match server-side proxy definitions per ADO PR 1982503. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add testThinClientChangeFeedFullRange covering FeedRange.forFullRange() across multiple partition keys, and testThinClientChangeFeedPartitionKey covering FeedRange.forLogicalPartition with exact doc count + PK validation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Documents all 59 thin client E2E tests across query (50), point operations (3), change feed (3), and stored procedures (3) with SQL, query features covered, and known account-side blockers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… QueryPlan proxy routing Add RNTBD token mappings for x-ms-cosmos-supported-query-features (0x00F0) and x-ms-cosmos-query-version (0x00F1) so the thin client proxy can read these values from the RNTBD body when processing QueryPlan requests. IDs are provisional (0x00F0, 0x00F1) — must be coordinated with server-side proxy team. See ADO PR 1982503 for the proxy-side design. Note: The design doc listed 0x002B/0x002C but those are already assigned to PartitionKey/PartitionKeyRangeId in the Java SDK. Using 0x00F0/0x00F1 to avoid ID collision until final server-side IDs are assigned. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…BD instructions - Fix testGetCurrentDateTime: assert ISO 8601 format instead of exact match (gateway and proxy return slightly different timestamps) - Add DefaultAzureCredential support via COSMOS.USE_AAD_AUTH system property for accounts with disableLocalAuth=true - Add RNTBD class reference as .github/instructions/rntbd.instructions.md - Add pom.xml system properties for THINCLIENT_ENABLED, HTTP2_ENABLED, USE_AAD_AUTH - Add beforeSuiteReuse mode for degraded accounts Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Switch baseline from Gateway V1 to Direct TCP to avoid JVM config interference (THINCLIENT_ENABLED/HTTP2_ENABLED affect Gateway V1) - Assert :10250 endpoint only on Gateway V2 results (not baseline) - Rename helpers: assertDirectAndThinClientMatch (was gateway) - Document seedTestData schema in Javadoc - Remove 'Expected to fail' comments (account has vector search enabled) - Clean up class/method Javadoc Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Following the simplification of the prefix EPK fix to reuse the existing StartEpk/EndEpk HTTP headers (a56db9a), the RxDocumentServiceRequest prefixPartitionKeyQuery flag is no longer read anywhere. Remove the field, its accessors, the clone copy, and the caller assignment in ParallelDocumentQueryExecutionContextBase. Test changes: - Retrofit CosmosMultiHashTest into the thinclient TestNG group so the MULTI_HASH prefix over-span coverage also runs against GatewayV2 (proxy :10250) over HTTP/2, not just the emulator gateway. - ThinClientQueryE2ETest: correct the prefix-scope Javadoc to describe the actual header-copy behavior and add a deterministic prefix-partition-key readAllItems regression. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The prefix-query flag on RxDocumentServiceRequest was removed in 03a7c67 after the fix was simplified to reuse the StartEpk/EndEpk HTTP headers. The only functional edit in this file (the setter call) went away with it, leaving behind unnecessary refactor noise. Restore the file to its upstream/main state so the hotfix touches no query execution code. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ort' into thin-client-probe-flow # Conflicts: # sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/CosmosNotFoundTests.java # sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/implementation/ConfigsTests.java # sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/implementation/EndpointProbeClientTests.java # sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/implementation/ReadManyByPartitionKeyQueryPlanRoutingTest.java # sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/implementation/ThinClientProbeWiringTests.java # sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/implementation/ThinClientRoutingGateTests.java # sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/implementation/query/QueryPlanRetrieverSupportedFeaturesTest.java # sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/rx/ThinClientQueryE2ETest.java # sdk/cosmos/azure-cosmos/CHANGELOG.md # sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/Configs.java # sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/EndpointProbeClient.java # sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/GlobalEndpointManager.java # sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/RxDocumentClientImpl.java # sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/query/PartitionedQueryExecutionInfo.java # sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/query/QueryPlanRetriever.java
…stream-activity' into thin-client-probe-flow # Conflicts: # sdk/cosmos/azure-cosmos/CHANGELOG.md # sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/http/ReactorNettyClient.java
…epk-overspan' into thin-client-probe-flow # Conflicts: # sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/OperationPoliciesTest.java # sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/PerPartitionCircuitBreakerE2ETests.java # sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/implementation/ThinClientE2ETest.java # sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/rx/TestSuiteBase.java # sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/rx/ThinClientQueryE2ETest.java # sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/rx/ThinClientTestBase.java # sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/ThinClientStoreModel.java
The merge of AzCosmos_GatewayV2_QueryPlanSupport (Azure#47759) dropped probe's thin-client-probe supporting code while keeping the method bodies that reference it, breaking compilation: - GlobalEndpointManager: restore imports (http.HttpClient, java.util.Set) and the thinClientProbeClient AtomicReference field. - LocationCache: restore getThinClientRegionalEndpoints() and the collectThinClientEndpointsAllOrNothing() helper. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…retry The @BeforeSuite container-readiness probe issues a QueryPlan against a freshly created container. With thin-client default-on, a fresh container in a multi-region account can transiently return 404/subStatus 1003 (OWNER_RESOURCE_NOT_EXISTS) during regional metadata propagation. The classifier did not allowlist 1003, so the probe treated it as non-retryable and cascade-skipped the entire thinclient suite. Add OWNER_RESOURCE_NOT_EXISTS to the NOTFOUND retryable allowlist in isRetryableCollectionReadinessFailure so the probe retries through propagation and the suite runs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…Flavor cold-start knobs Instrument azure-cosmos-benchmark for the cold-start Create-latency matrix driving the thin-client probe (PR Azure#49437): - pom: parameterize azure-cosmos + azure-cosmos-encryption versions (-Dazure-cosmos.version=...) so GA 4.79/4.80/4.81 and the probe build can be benchmarked from one harness. - Common tags (RunId, SdkVersion, ConnectionMode, EndpointFlavor, Phase) applied to every metric so a shared Grafana dashboard can slice runs. - endpointFlavor knob (ComputeGateway|ThinClient) maps to COSMOS.THINCLIENT_ENABLED at client build to drill both the Gateway V1 and Gateway V2 (thin-client) HTTP/2 paths. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Add two opt-in Micrometer meters (gated on COSMOS.HTTP2_STREAM_METRICS_ENABLED): - cosmos_client_http2_connectionsOpened (Counter) - cosmos_client_http2_requestsPerConnection (DistributionSummary) Recorded in ReactorNettyClient via doOnConnected / closeFuture; requests-per- connection uses a deferred-drain queue drained on the CpuMemoryMonitor 5s tick so counts survive per-cycle registry teardown. Used to diagnose the cold-start HTTP/2 connection burst on the thin-client / Gateway-V2 Create path. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Add an opt-in KUSTO reporting destination that streams all benchmark metrics with full dimensions directly into an Azure Data Explorer table via the streaming-ingest REST endpoint (bearer auth, no extra Maven dependency). Run-level common tags (RunId/SdkVersion/ConnectionMode/ EndpointFlavor/Phase) map straight to first-class BenchmarkMetrics columns for downstream Grafana dashboards. - KustoMetricsReporter: streaming-ingest reporter over a SimpleMeterRegistry - KustoReporterConfig: JSON config (cluster/db/table/auth) - ReportingDestination.KUSTO + BenchmarkConfig/Orchestrator wiring - Guard null tenantId in applicationName derivation for multi-cycle runs Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Introduces best-effort eager warm-up of the shared thin-client HTTP/2 connection pool at client-build time so the first data-plane request avoids the cold TLS+HTTP/2 handshake burst. Adds COSMOS.HTTP2_EAGER_CONNECTION_WARMUP_ENABLED (off by default) and a bounded timeout config, an EndpointProbeClient.warmUpConnections method that never fails client init, and wires it into RxDocumentClientImpl/GlobalEndpointManager. Includes unit tests. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Resolve conflicts and align thin-client QueryPlan routing: - Remove the QueryPlan exemption in TestSuiteBase.assertThinClientEndpointUsed; every request (including QueryPlan) must target the thin-client endpoint (:10250) since Configs.isThinClientQueryPlanEnabled() defaults to true and thin-client tests use HTTP/2, so QueryPlan is routed to Gateway V2. - Update delegating comment in ThinClientTestBase accordingly. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…readiness probe for thin-client-default-on - Remove the QueryPlan exemption in assertThinClientEndpointUsed: when ThinClient is opted in with HTTP/2 enabled, QueryPlan calls MUST also route to Gateway V2 (:10250), matching production default DEFAULT_THINCLIENT_QUERY_PLAN_ENABLED=true. - Restore public static visibility on ThinClientTestBase.assertThinClientEndpointUsed / assertGatewayEndpointUsed so cross-package static imports (CosmosMultiHashTest) compile. - Add OWNER_RESOURCE_NOT_EXISTS (subStatus 1003) to the retryable NOTFOUND allowlist in TestSuiteBase.isRetryableCollectionReadinessFailure. With thin-client default-on the beforeSuite readability probe (a QueryPlan) routes through GW V2; a freshly-created container not yet propagated to a secondary region returns 404/1003, which must be treated as a transient readiness failure instead of cascade-skipping the suite. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Validate the COSMOS.THINCLIENT_QUERY_PLAN_ENABLED sysprop/env kill-switch in both directions: - Unit: new case in ReadManyByPartitionKeyQueryPlanRoutingTest asserts QueryPlan is forced to Gateway V1 (useGatewayMode=true) when the flag is disabled, even with thin-client eligible. - E2E: make TestSuiteBase.assertThinClientEndpointUsed flag-aware so QueryPlan endpoint assertions flip based on the live-read flag while data operations continue to route to the thin-client endpoint. Both scenarios verified green (unit 3/3; E2E 112/112, oracle:true for flag ON and flag OFF). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ort' into jeet1995/thin-client-probe-flow # Conflicts: # sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/CosmosMultiHashTest.java # sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/OperationPoliciesTest.java # sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/PerPartitionCircuitBreakerE2ETests.java # sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/rx/TestSuiteBase.java # sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/rx/ThinClientQueryE2ETest.java # sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/rx/ThinClientTestBase.java # sdk/cosmos/azure-cosmos/CHANGELOG.md # sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/ThinClientStoreModel.java
…zure#47759 merge The Azure#47759 merge conflict resolution kept HEAD's short-circuit-wait loop (which calls isAnyRegionShortCircuitedForPartition) but dropped the method definition. Recovered from probe tip 42c2c7d and re-inserted. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Azure#49725) These changes belong in PR Azure#49725, not the endpoint-probe PR Azure#49437. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Wire runThinClientProbeCycleMono() into the account-refresh funnel in GlobalEndpointManager so the connectivity probe runs on every refresh, delta-gated to new/unproven regions in EndpointProbeClient. Previously the probe driver had no production callers, leaving the default-on thin-client routing gate permanently false. Add a GEM delta-refresh unit test proving a subsequent refresh probes only newly-added regions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…obe wiring Wire runThinClientProbeCycleMono() into GlobalEndpointManager.refreshLocationPrivateAsync so the connectivity probe runs on every account refresh: as a prefix when a fresh databaseAccount is present, and after the foreground database-account read (.map -> .flatMap). Add an implicit-path test lane that leaves COSMOS.THINCLIENT_ENABLED unset (thin-client default-on, routing gated on the probe verdict with Gateway V1 fallback) while keeping HTTP2_ENABLED true: - new 'thinclient-endpoint-probe' maven profile (azure-cosmos-tests/pom.xml) - new Cosmos_Live_Test_ThinClient_EndpointProbe CI stage (tests.yml) - live-thinclient-endpoint-probe-platform-matrix.json - thinclient-endpoint-probe-testng.xml suite - register thinclientEndpointProbe group in TestSuiteBase before/after suite - E2E probe tests: point-operation, query, change-feed, stored-procedure + shared base Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Reconcile with Azure/main (Gateway V2 SP + QueryPlan routing, PR 47759). Centralized op-type routing in ThinClientConnectivityConfig covers all operation types upstream added inline (StoredProcedure, QueryPlan, point, Query, Batch, incremental ChangeFeed); probe gate preserved. Kept PR 49437 default-on thin-client + connectivity-probe work, our query-plan PK-def guard, and the thinclient-endpoint-probe test profile. Adopted upstream's ambient JVM-flag approach in ThinClientTestBase. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Reset QueryPlanRetriever, PartitionedQueryExecutionInfo and 9 test files to Azure/main. These carried Gateway V2 QueryPlan work (partitionKeyDefinition guard, CountIf, queryRanges removal) and parity-test adaptations that overlap upstream Azure#47759, which Azure/main already owns. Scoping this PR to the thin-client connectivity-probe flow. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…o route thin-client Collapse all call sites onto the tri-state reader Configs.isThinClientEnabledExplicitly() and remove the redefined isThinClientEnabled() helper, which had flipped the unset default from false to true. Thin-client routing now requires an explicit opt-in or an affirmative connectivity-probe verdict; an unset flag no longer routes to Gateway V2 by default (null/FALSE probe verdict -> Gateway V1). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Rename Configs.isThinClientEnabledExplicitly() to isThinClientEnabled() (nullable Boolean) and remove the hasUserExplicitlyEnabledThinClient() and ThinClientConnectivityConfig.isExplicitThinClientOptIn() helpers. The routing gate shouldUseThinClientStoreModel now takes the tri-state Boolean directly: a non-null value is a hard contract (TRUE routes to Gateway V2, FALSE pins to V1), and null falls back to the connectivity-probe verdict. This eliminates the scattered Boolean.equals opt-in checks. Default remains null (not default-on). Updated ThinClientRoutingGateTests implicit-path cases to pass null for the tri-state arg (probe-gated) instead of false. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…TestNG group Tag the existing ThinClient*E2ETest classes (and ThinClientTestBase) with both the `thinclient` and `thinclientEndpointProbe` groups so the same test bodies run in either CI lane against a different ambient enablement path: - thinclient lane: -DCOSMOS.THINCLIENT_ENABLED=true (explicit opt-in) - thinclient-endpoint-probe lane: flag unset -> endpoint connectivity probe drives routing Delete the 5 duplicate ThinClientEndpointProbe* copies; the probe suite XML discovers by group across com.azure.cosmos.*, so it auto-picks up the retagged classes with zero duplication. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Catches PR Azure#49725 up with PR Azure#49437. Resolved duplicate PPCB diagnostics fix (both branches added it independently; kept probe-flow/Azure#49734 canonical form for the comment and the regression test). Preserved kusto-sink's unique eager HTTP/2 pre-warm and Kusto/ADX benchmark sink work. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
This is the consolidated branch that folds the thin-client endpoint-probe work together with its dependencies plus the perf benchmark tooling used to drill-test cold start. It corresponds to the endpoint-probe line of work (PR #49437) with the following merged in:
Benchmark / perf tooling (on top of the SDK fixes)
test(cosmos-benchmark): parameterize SDK version + add RunId/EndpointFlavor cold-start knobsperf(cosmos): add opt-in HTTP/2 connection instrumentation metricstest(cosmos-benchmark): add Azure Data Explorer (Kusto) metrics sink— streams benchmark metrics (withRunId/SdkVersion/ConnectionMode/EndpointFlavor/Phasedimensions) into ADX for the perf Grafana dashboards.Notes
main(three-dot).