Skip to content

Add SQL Server (mssql2019) persistence plugin#11063

Open
ashish24142 wants to merge 5 commits into
temporalio:mainfrom
ashish24142:ashish/mssql-persistence-plugin
Open

Add SQL Server (mssql2019) persistence plugin#11063
ashish24142 wants to merge 5 commits into
temporalio:mainfrom
ashish24142:ashish/mssql-persistence-plugin

Conversation

@ashish24142

Copy link
Copy Markdown

What was changed

Adds a Microsoft SQL Server persistence plugin (mssql2019, SQL Server 2019+ / Azure SQL) that can serve both the default persistence store and the visibility store, following the existing mysql/postgresql/sqlite plugin architecture (registered via sql.RegisterPlugin, selected with pluginName: "mssql2019").

  • Plugin (common/persistence/sql/sqlplugin/mssql/): per-table query implementations translated from the postgresql plugin — MERGE ... WITH (HOLDLOCK) upserts (executed per row for slices, since sqlx cannot expand MERGE), UPDLOCK/REPEATABLEREAD locking hints, TOP/OFFSET-FETCH pagination, expanded row-value tuple comparisons, central ?@pN rebinding, passwordCommand support via RefreshingConnector, TLS-to-DSN mapping for go-mssqldb.
  • Schema (schema/mssql/v2019/): all 37 main-store tables; single-table visibility (postgres layout) with PERSISTED computed search-attribute columns, a persisted coalesce_close_time ordering column, an SHA2_256 hash PK for chasm_node_maps (SQL Server's 900-byte index key limit), and full-text search over Text attributes via a row_key IDENTITY surrogate key.
  • Visibility query converters: both the unified and legacy paths (OPENJSON for KeywordList, CONTAINS() for Text). T-SQL has no boolean literals, so a small optional hook (pluginBoolValueConverterLegacy) was added to the legacy converter to rewrite true/false1/0 for BIT columns; the unified path uses the existing ConvertComparisonExpr hook.
  • Azure SQL Database compatibility: admin operations issue single-statement batches (CREATE/DROP DATABASE must be alone in a batch there), and the connection-refresh classifier includes Azure's documented transient fault codes.
  • Integration: blank imports in the server and temporal-sql-tool binaries, schema.PathsByDB, AllowListForValidation, persistence test harness wiring, docker-compose service, install-schema-mssql2019 Makefile target, development-mssql.yaml.
  • New dependency: github.com/microsoft/go-mssqldb v1.10.0 (registered with sqlx as "sqlserver").

Why

SQL Server is the remaining major relational database without Temporal support. Organizations standardized on the Microsoft stack (licensing, DBA tooling, compliance) currently have no supported way to run Temporal on the database they operate, on-prem or on Azure SQL.

How it was tested

All against a real SQL Server 2022 instance (Docker, with mssql-server-fts):

  • Both schemas apply cleanly via temporal-sql-tool (setup-schema + update-schema).
  • The server boots on mssql2019 for both stores: all shards acquired, system namespace/workflows running, no driver errors.
  • End-to-end: namespace registration, workflow execution to completion, visibility ListWorkflowExecutions (including WorkflowType = '...' AND ExecutionStatus = 'Completed') and CountWorkflowExecutions return correct results.
  • All 44 shared persistence CRUD suites pass — 521 subtests, 0 failures — via the new TestMSSQL entry (common/persistence/tests/mssql_test.go), covering shard, execution mutable state (+tasks), every per-table history suite, matching tasks v1/v2, task queues, queue v1/v2, nexus endpoints, metadata, cluster metadata, and the full visibility filter/count/group-by/pagination paths.
  • Local load characterization (single all-in-one node, 128 shards): ~136 completed workflows/sec sustained with p95 latency ~520 ms on bare workflows.

go build ./..., go vet, and gofmt are clean across the tree.

Notes for reviewers

  • Known follow-ups are marked in code comments: multi-row inserts can exceed SQL Server's 2100-parameter batch cap on very large mutations (needs chunking); KeywordList predicates use OPENJSON at query time since SQL Server cannot index inside JSON arrays.
  • Happy to add a CI matrix entry (SQL Server container + mssql-server-fts) if you want this exercised in CI, and to split this PR into smaller pieces if that's easier to review.

Implements a new SQL plugin backed by github.com/microsoft/go-mssqldb
supporting both the default persistence store and the visibility store
on SQL Server 2019+ / Azure SQL:

- Plugin package with per-table query implementations translated from
  the postgresql plugin: MERGE WITH (HOLDLOCK) upserts (executed per-row
  for slices, since sqlx cannot expand MERGE), UPDLOCK/REPEATABLEREAD
  table hints for locking reads, TOP/OFFSET-FETCH pagination, expanded
  row-value tuple comparisons, and central ?->@pn rebinding.
- Session layer with sqlserver:// DSNs, passwordCommand support via
  RefreshingConnector, and TLS-to-DSN-parameter mapping.
- T-SQL schemas for main store (37 tables) and single-table visibility
  with PERSISTED computed search-attribute columns, a persisted
  coalesce_close_time ordering column, SHA2_256 hash PK for
  chasm_node_maps (900-byte key limit), and full-text search over Text
  attributes via a row_key IDENTITY surrogate key.
- Unified and legacy visibility query converters (OPENJSON for
  KeywordList, CONTAINS for Text, BIT-safe boolean literal rewriting via
  a new optional pluginBoolValueConverterLegacy hook).
- Admin operations issue single-statement batches and classify Azure
  SQL transient fault codes for Azure SQL Database compatibility.
- Blank-import the mssql plugin in the server and temporal-sql-tool
  binaries so init() registration runs.
- Include schema/mssql in PathsByDB so --schema-name validation accepts
  the embedded mssql schema directories.
- Add mssql2019 to AllowListForValidation (SQL visibility stores do not
  support list-of-values custom search attributes) and to the
  functional-test UseSQLVisibility driver switch.
- Add GetMSSQLTestClusterOption and MSSQL_SEEDS/MSSQL_PORT environment
  helpers for the persistence test harness.
- SQL Server 2022 Developer container (includes Full-Text Search, which
  the mssql visibility schema requires) in the dev docker-compose.
- make install-schema-mssql2019 target with MSSQL_USER/MSSQL_PASSWORD
  variables matching the container's SA credentials.
- config/development-mssql.yaml mirroring the other development-*.yaml
  configs for running the server against local SQL Server.
Found by applying the schema to SQL Server 2022 in Docker:
- Remove the IF NOT EXISTS guards from the full-text catalog/index:
  Temporal's schema statement splitter parses IF as a PL/pgSQL block
  needing END IF, so T-SQL IF guards produce 'unmatched i'. To re-apply,
  drop and recreate the database instead of setup-schema --overwrite.
- Make coalesce_close_time deterministic so it can be PERSISTED:
  CAST('...' AS DATETIME2) is non-deterministic (depends on session
  language); use CONVERT(DATETIME2(6), '...', 126) (ISO 8601) instead.
- Document that the stock SQL Server container does not bundle Full-Text
  Search and must have mssql-server-fts installed.

Verified end to end: schema applies cleanly, the server boots on
mssql2019, and a workflow runs to completion and is returned by
visibility list and count queries.
Mirror the postgresql test entry files so the shared persistence suites
run against SQL Server via -run TestMSSQL. Requires a running SQL Server
with mssql-server-fts installed (see docker-compose).

All 44 suites pass against SQL Server 2022: 521 subtests, 0 failures —
covering shard/execution/history/task/queue/metadata/cluster-metadata
CRUD, every per-table history suite, and the full visibility filter,
count, group-by and pagination paths.
@ashish24142
ashish24142 requested review from a team as code owners July 14, 2026 18:47
@CLAassistant

CLAassistant commented Jul 14, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@ashish24142

Copy link
Copy Markdown
Author

Note for maintainers on the failing crossrepo/saas-compatibility check: the run lives in the private saas-temporal repo, so I can't see the logs from outside. My best guess at the cause is the new dependency this PR introduces (github.com/microsoft/go-mssqldb, plus its two small golang-sql/* transitive modules), which enters the server's dependency graph the same way the other SQL drivers do — if the internal build uses a module allowlist/mirror, it won't have seen these yet. The shared-code changes outside the new plugin are additive only (an unexported optional interface in the legacy visibility converter and new switch cases), so a compile break seems less likely.

Happy to adjust whatever the internal run actually flags — and to add a public CI matrix entry for SQL Server if you'd like this exercised in this repo's CI as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants