test(perf): establish PendingRequestTable pressure and recovery matrix

.github/workflows/pending-validation.yml

@@ -4,16 +4,21 @@ on:
   pull_request:
     branches: [dev]
     paths:
-      - 'src/SharpLink.Abstractions/SharpLinkTelemetry.cs'
+      - 'src/SharpLink.Abstractions/SharpLinkTelemetry*.cs'
       - 'src/SharpLink.Client/PendingRequestTable.cs'
-      - 'src/SharpLink.Client/SharpLinkClient.Attempts.cs'
+      - 'src/SharpLink.Client/SharpLinkClient*.cs'
+      - 'src/SharpLink.Client/SharpLinkCircuitBreaker*.cs'
       - 'src/SharpLink.Runtime/Codec/CodecHelpers.cs'
       - 'src/SharpLink.Runtime/Codec/DateTimeCodec.cs'
       - 'src/SharpLink.Runtime/Codec/StructCodec.cs'
       - 'test/SharpLink.UnitTests/Validation/**'
+      - 'test/SharpLink.Benchmarks/PendingRequestMatrix*.cs'
+      - 'test/SharpLink.Benchmarks/Program.cs'
       - 'eng/validate-pending-lifecycle.py'
       - 'eng/validate-codec-semantics.py'
       - 'eng/validate-datetimeoffset-performance.py'
+      - 'eng/run-pending-request-matrix.sh'
+      - 'doc/pending-request-matrix.md'
       - '.github/workflows/pending-validation.yml'
   workflow_dispatch:
 
@@ -101,3 +106,31 @@ jobs:
           name: pending-and-codec-evidence
           path: artifacts/validation
           if-no-files-found: warn
+
+  issue-571-pending-request-matrix:
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    env:
+      TESTINGPLATFORM_TELEMETRY_OPTOUT: '1'
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          ref: ${{ github.event.pull_request.head.sha || github.sha }}
+          persist-credentials: false
+      - uses: ./.github/actions/setup-dotnet
+      - name: Record issue 571 evidence provenance
+        run: |
+          mkdir -p artifacts/validation/issue-571
+          git rev-parse HEAD | tee artifacts/validation/issue-571/commit.txt
+          dotnet --info > artifacts/validation/issue-571/dotnet-info.txt
+          uname -a > artifacts/validation/issue-571/uname.txt
+          lscpu > artifacts/validation/issue-571/lscpu.txt
+      - name: Run issue 571 deterministic matrix smoke
+        run: bash eng/run-pending-request-matrix.sh ci artifacts/validation/issue-571/matrix
+      - name: Upload issue 571 matrix evidence
+        if: always()
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: issue-571-pending-request-matrix
+          path: artifacts/validation/issue-571
+          if-no-files-found: warn

doc/pending-request-matrix.md

@@ -0,0 +1,82 @@
+# PendingRequestTable performance and recovery matrix
+
+This document defines the reproducible evidence matrix for issue #571. The matrix is deliberately implemented in `test/SharpLink.Benchmarks`; it does not add benchmark-only branches or counters to the production pending-request hot path.
+
+## Commands
+
+```bash
+# Fast deterministic CI coverage. This is what pending-validation.yml runs on pull requests.
+bash eng/run-pending-request-matrix.sh ci artifacts/perf/pending-request-matrix-ci
+
+# P0 formal matrix.
+bash eng/run-pending-request-matrix.sh p0 artifacts/perf/pending-request-matrix-p0
+
+# P0 plus P1 stress/extensions, including protocol-high sparse capacity,
+# 128 producers, 99%/25%-long mix, feature-heavy production saturation,
+# cancellation/deadline mix, and 20 recovery cycles.
+bash eng/run-pending-request-matrix.sh p1 artifacts/perf/pending-request-matrix-p1
+```
+
+Each command records a single JSON report plus the console log. The report includes the exact GitHub SHA when running in Actions, runtime/OS/architecture, processor count, GC mode, and `Stopwatch.Frequency`. GitHub Actions additionally uploads `dotnet --info`, `uname -a`, and `lscpu` with the report.
+
+Timing values from shared hosted runners are evidence only; they are not CI thresholds. Compare performance only on controlled hardware, preferably by alternating `dev` and the candidate revision on the same machine and retaining raw reports from every run.
+
+## Matrix coverage
+
+### High occupancy and saturation
+
+The runner holds a real `PendingRequestTable` at controlled occupancy and rotates registrations/completions while keeping long-lived entries resident. It reports requested and actual occupancy, average/P95/P99/max sampled occupancy, QPS, CPU ns/op, allocation/op, P50/P95/P99/P99.9 operation latency, request-ID advances, extra probe attempts caused by occupied slots, rejection counts, and per-producer progress.
+
+P0 covers capacities 64, 1K, 16K and 65K with 50/75/90/95/99% occupancy and 1/8/32 producers. Full-capacity cells verify fail-fast `ResourceExhausted` behavior and immediate full-capacity reuse. P1 adds 128-producer 99% cells.
+
+### Sparse deadlines
+
+The deterministic scheduler path uses a controllable `TimeProvider` and invokes the production deadline scan directly. It records scan cost, capacity inspected, active/deadline counts, and single/staggered/clustered expiration patterns. It explicitly verifies that a deadline never completes before its monotonic boundary and that expiration succeeds at the boundary. A separate real-timer cell measures P50/P95/P99/max deadline lateness while only eight calls are active in a 65K table.
+
+P1 additionally exercises the protocol hard maximum capacity (1,048,576) with sparse active/deadline state without materializing one million pending operations.
+
+### Long/short mixed lifetimes
+
+Long calls are created first and held deterministically while producer workers rotate short calls at steady occupancy. Long calls are never modeled with random sleeps. Terminal modes are response, user cancellation, deterministic deadline, and connection-close cleanup. The report includes short-call latency/QPS, actual occupancy, long-call count/share, producer progress, and terminal duration. P1 adds the 99%-occupied, 25%-long, 128-producer case.
+
+### Production-shaped profiles
+
+The matrix includes real loopback TCP RPC profiles using generated contracts and validates every response:
+
+- `plain-control`: TCP, no TLS/compression/metrics/retry/breaker/admission.
+- `typical-production`: TLS, Zstd compression via `SharpLinkZstdCompressionProvider`, normal SharpLink metrics, retry, circuit breaker, admission control, and 0/256/4096-byte payload cells.
+- `feature-heavy` (P1): the typical profile plus full client/server tracing.
+- `feature-heavy-saturation` (P1): the same TLS + Zstd + metrics/tracing + retry + circuit-breaker + admission configuration with a 64-entry client pending table. A deterministic server barrier holds 58 real RPCs open (90.625% occupancy) before a burst of 64 short RPCs is released through the remaining capacity. While the held baseline remains active, two additional calls are explicitly cancelled and two method-timeout calls are allowed to terminate as `DeadlineExceeded`; the remaining held calls are then released through the same barrier. Immediately before the held-open phase, a successful probe records the server-side `SharpLinkCallContext.Current.SessionId`; recovery performs another successful probe and requires the reported session ID to be exactly equal, so a hidden reconnect cannot satisfy the same-session gate.
+
+The ordinary production cells report QPS, process CPU/call, allocation/call, Gen0/1/2 counts, failures, retries, `ResourceExhausted`, pending high-water/after state, and P50/P95/P99/P99.9 latency. The targeted saturation cell additionally reports actual occupancy at the barrier, pending high-water and percentages, short-call success/`ResourceExhausted`/other-error distribution, short-call QPS/CPU/allocation and terminal/success latency distributions, cancellation and deadline terminal counts, retry/resource-exhausted metrics, zero-pending recovery, the measured `sessionBefore`/`sessionAfter` identities, and their `sessionReuse` equality result.
+
+### Overload and recovery
+
+Each recovery cycle is operation-count/barrier driven:
+
+1. low-occupancy sequential baseline probe;
+2. ramp to full capacity;
+3. verify fail-fast overload and place controlled async renters into the waiter path;
+4. release exactly enough capacity, require every waiter to make progress, then simulate disconnect cleanup;
+5. prove pending/waiter counts return to zero, refill the entire table to prove all capacity is reusable, drain it again, and run a post-recovery baseline probe.
+
+CI runs three cycles, P0 seven, and P1 twenty. Full-GC heap samples are retained per cycle; the `RecordingOwner` drops its entire exactly-once request-ID history whenever pending ownership reaches a quiescent zero boundary, so full-GC samples do not retain a monotonically growing harness-only `HashSet`. The deterministic gate rejects gross retained-state growth (more than 64 MiB above the minimum observed full-GC heap) while leaving normal performance comparison to formal evidence runs.
+
+## Correctness gates
+
+Any violation throws and fails the matrix before the report is marked complete. The gates cover:
+
+- active occupancy never exceeds configured capacity;
+- every registered request has one terminal completion;
+- owner/capacity accounting never underflows and returns to zero;
+- stale responses cannot match a newer request lifecycle;
+- deadlines never complete early;
+- full-table waiters are released without lost wakeups;
+- disposal wakes waiters and strands no pending call;
+- disconnect cleanup strands no pending call;
+- after every recovery cycle there are zero pending calls and zero waiters;
+- the complete configured capacity can be reused after recovery;
+- ordinary production-profile RPCs return correct results with zero failures and zero pending requests after the measurement window;
+- feature-heavy saturation reaches at least 90% actual pending occupancy under the real production feature stack, preserves the expected cancellation/deadline terminals, releases every held call, returns pending state to zero, and proves same-session recovery by comparing the server-side session identity before pressure with the identity after recovery.
+
+The permanent pull-request gate is the `issue-571-pending-request-matrix` job in `.github/workflows/pending-validation.yml`. It runs the `ci` tier and uploads the exact evidence used for the gate. Formal P1 review additionally runs the feature-heavy saturation/cancellation/deadline cell.

eng/run-pending-request-matrix.sh

@@ -0,0 +1,69 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+cd "$ROOT"
+
+TIER="${1:-${SHARPLINK_PENDING_MATRIX_TIER:-ci}}"
+OUTPUT_DIR="${2:-${SHARPLINK_PENDING_MATRIX_OUTPUT:-artifacts/perf/pending-request-matrix}}"
+REPORT="$OUTPUT_DIR/report.json"
+mkdir -p "$OUTPUT_DIR"
+
+case "$TIER" in
+  ci|p0|p1) ;;
+  *) echo "pending request matrix tier must be ci, p0, or p1" >&2; exit 2 ;;
+esac
+
+dotnet build test/SharpLink.Benchmarks/SharpLink.Benchmarks.csproj -c Release -v minimal
+
+GITHUB_SHA="$(git rev-parse HEAD)" dotnet run \
+  --project test/SharpLink.Benchmarks/SharpLink.Benchmarks.csproj \
+  -c Release --no-build -- \
+  --pending-request-matrix-evidence \
+  --tier "$TIER" \
+  --output "$REPORT" | tee "$OUTPUT_DIR/run.log"
+
+python3 - "$REPORT" "$TIER" <<'PY'
+import json
+import sys
+from pathlib import Path
+
+report_path = Path(sys.argv[1])
+tier = sys.argv[2]
+report = json.loads(report_path.read_text(encoding="utf-8"))
+if report.get("phase") != "complete" or report.get("invariant") is not True:
+    raise SystemExit("pending request matrix report did not complete its correctness gates")
+
+cells = report.get("cells") or []
+categories = {cell.get("category") for cell in cells}
+required = {
+    "hard-gate",
+    "high-occupancy",
+    "sparse-deadline",
+    "long-short-mix",
+    "overload-recovery",
+    "production-profile",
+}
+missing = sorted(required - categories)
+if missing:
+    raise SystemExit(f"pending request matrix report is missing categories: {missing}")
+
+profiles = {cell.get("profile") for cell in cells if cell.get("category") == "production-profile"}
+for required_profile in ("plain-control", "typical-production"):
+    if required_profile not in profiles:
+        raise SystemExit(f"missing production profile {required_profile}")
+if tier == "p1" and "feature-heavy" not in profiles:
+    raise SystemExit("p1 matrix is missing feature-heavy production profile")
+
+if any(cell.get("invariant") is not True for cell in cells):
+    raise SystemExit("one or more pending request matrix cells failed their invariant")
+
+print(json.dumps({
+    "phase": "validated",
+    "tier": tier,
+    "cellCount": len(cells),
+    "categories": sorted(categories),
+    "profiles": sorted(profile for profile in profiles if profile),
+    "report": str(report_path),
+}, sort_keys=True))
+PY