Fix duplicate-key races in endpoint throughput recording

[rbev]

Problem

LicensingDataStore.RecordEndpointThroughput recorded a day's throughput with a check-then-insert sequence guarded by an application-level retry loop: try ExecuteUpdateAsync first, insert if no row was updated, catch the duplicate-key DbUpdateException, and retry the whole batch up to 5 times before throwing InvalidOperationException.

That race is routine on the monitoring path, not an edge case: the monitoring instance sends one batched throughput message every 5 minutes covering all endpoints for the current day, and the receiving endpoint processes those messages at transport concurrency (default 32). At a day boundary or cold start, up to 32 writers can hit the same endpoint/day row while it still doesn't exist — the exact window where two writers both miss the update, both insert, and one loses with a duplicate key. The loser then re-runs its entire batch in a new scope/transaction, multiplying write traffic precisely under peak contention, and a sustained collision streak surfaces as InvalidOperationException("... after 5 attempts ...") — a panic error for a normal concurrency event. (The receiver swallows and logs it; the day's total re-heals over subsequent reports, so the impact was log noise, load amplification, and fragile code — not corruption.)

What changed

• New IEndpointThroughputDialect (ServiceControl.Persistence.EFCore/Infrastructure/) with one atomic add-or-insert statement per day, in the batch order given:
  • SQL Server (SqlServerEndpointThroughputDialect): MERGE ... WITH (HOLDLOCK) — the WHEN MATCHED arm adds to the day's total, WHEN NOT MATCHED creates it. HOLDLOCK (SERIALIZABLE) takes key-range locks on the target so two recorders of the same day cannot both pass the match test and both insert.
  • PostgreSQL (PostgreSqlEndpointThroughputDialect): INSERT ... ON CONFLICT (normalized_name, throughput_source, date_utc) DO UPDATE SET message_count = <table>.message_count + EXCLUDED.message_count — Postgres's canonical atomic upsert; the speculative-insert machinery means no unique violation can ever surface to the client.
• Both statements run on the context's connection inside the transaction the caller already opened, and reuse the existing Table<T>/Execute dialect helpers — the same shape as the ingestion dialects (IFailedMessageIngestionSqlDialect already uses MERGE ... HOLDLOCK for the identical race).
• LicensingDataStore.RecordEndpointThroughput now orders the batch by date (so concurrent overlapping batches take row locks in a consistent order and cannot deadlock against each other), checks the endpoint exists, and delegates the writes to the dialect. The 5-attempt retry loop and its InvalidOperationException are gone: with one atomic statement per day there is no check-then-insert window left to retry. Transient failures (e.g. deadlock victims) are still retried by the existing retrying execution strategy, which wraps the whole batch-including-transaction.
• Dialects are registered per persistence flavor (SqlServerPersistence, PostgreSqlPersistence); both provider .csproj files gain the Particular.LicensingComponent.Contracts reference for ThroughputSource.

No caller-facing contract changes: same accumulation semantics (recording the same day twice adds to the total), same friendly error when the endpoint doesn't exist, same all-or-nothing batch behavior.

Why not the existing duplicate-key patterns?

• The other IsDuplicateKeyException sites (MessageArchiver, EditFailedMessagesDataStore) are claim/lease acquisitions — a duplicate key means another worker owns the operation and the loser is supposed to back off. Here every writer's contribution must land, so the loser must not surrender.
• The shared UpsertAsync helper has last-writer-wins semantics: its update arm mutates a loaded entity (read-modify-write), so it cannot express message_count + delta computed atomically in the database — it would introduce lost updates under exactly this path's concurrency. Its own remarks point hot paths at dialect-specific upserts instead.

Preconditions & safety

• The upsert targets are exactly the composite primary key (NormalizedName, ThroughputSource, DateUtc) / (normalized_name, throughput_source, date_utc) in both providers' Initial migrations, so the ON CONFLICT target and MERGE ... ON always match an existing unique constraint. (PostgreSQL fails loudly if this ever stops matching; SQL Server MERGE relies on it — the PK must not be recreated non-uniquely.)
• Batch atomicity and the retrying-execution-strategy wrapping are unchanged from the previous implementation.

Testing

No new tests required — the change is covered by the existing provider-parameterized persistence suites, notably LicensingDataStoreEFTests:

• Concurrent_recordings_of_the_same_day_all_count — 10 concurrent writers on the same endpoint/day, asserting every contribution lands (this test exercised the old retry path and now exercises the atomic upserts against real SQL Server / PostgreSQL).
• Recording_the_same_day_twice_adds_to_that_day, Endpoints_are_matched_regardless_of_name_casing.