Fix PostgreSQL visibility failure on Text search attributes containing ':'

[garfieldnate]

Context

I work at Nonsense, and I run a lot of workflows on movies. We use internal IDs for most things, but for search attributes we like to use human-readable text such as movie titles. Since introducing these attributes, our Temporal UI has become very difficult to use because hundreds of workflows that show as "Running" are actually completed, which we don't know until we click on them.

The diagnosis and fix below are AI-/Fable-generated (but still edited/vetted by me).

Problem

On PostgreSQL advanced visibility, any Text search attribute value containing a character significant to tsvector literal syntax — most commonly :, but also an unbalanced ' — fails the entire executions_visibility row INSERT/UPDATE:

pq: syntax error in tsvector: "The Lost World: Jurassic Park" (42601)

The cause is the generated column definition in schema/postgresql/v12/visibility/schema.sql, which uses a raw cast:

Text01 TSVECTOR GENERATED ALWAYS AS ((search_attributes->>'Text01')::tsvector) STORED

A raw ::tsvector cast doesn't tokenize free text — it parses the value as a tsvector literal, in which : introduces lexeme positions (valid literal syntax looks like 'jurassic':4 'park':5). Parsing The Lost World: Jurassic Park, Postgres reads the token World: and expects position digits after the colon; the space that follows makes the literal invalid and the cast throws. Without the colon it happens to work — 'The Lost World Jurassic Park'::tsvector parses fine as five lexemes — which is why the bug stays hidden until real-world punctuated text shows up. The cast runs while Postgres computes the generated column, so the whole row write is rejected.

Smallest demonstration (this is exactly what the generated column executes):

SELECT 'The Lost World: Jurassic Park'::tsvector;
-- ERROR:  syntax error in tsvector: "The Lost World: Jurassic Park"

Consequences in production

• RecordWorkflowExecutionClosed fails (service/history/visibility_queue_task_executor.go → processCloseExecution); the visibility task retries and is eventually sent to the DLQ ("Marking task as terminally failed, will send to DLQ").
• The workflow's visibility record — inserted at start, before the Text attribute was upserted — is never updated, so a completed workflow shows as Running in the UI/List API forever, until namespace retention deletes the record.
• UpsertWorkflowSearchAttributes visibility tasks fail the same way.
• On our cluster (v1.28.1, ~39k workflows/day, phrase/movie-title text in a Text attribute) ~0.7% of workflows hit this, producing a permanent, self-replenishing pool of ~800 phantom "Running" workflows and a steady stream of DLQ'd visibility tasks.

This has been reported by the community with no fix to date:
https://community.temporal.io/t/temporal-workflow-visibility-error-pq-syntax-error-in-tsvector-mocked-traceid-5c060ad0-5ec8-4ebb-b344-5558a9f11111/10625

Verified on v1.28.1; the schema and converters are unchanged on main and v1.30.1. MySQL (TEXT + FULLTEXT) and SQLite (TEXT + FTS5) store the value as plain text and are unaffected — only PostgreSQL uses the raw cast.

The read side has the sibling bug

The query converters (common/persistence/sql/sqlplugin/postgresql/query_converter.go and the legacy common/persistence/visibility/store/sql/query_converter_legacy_postgresql.go) tokenize the query value on spaces and emit a raw ::tsquery cast. Filtering on the very title that was stored — WHERE Title = 'The Lost World: Jurassic Park' — becomes Text01 @@ 'The | Lost | World: | Jurassic | Park'::tsquery, and the token World: throws syntax error in tsquery. Any colon-bearing token (a URL, key:value tags) does the same, so ListWorkflowExecutions/CountWorkflowExecutions filtering on such a Text value errors as well.

Fix

Convert both sides with the PostgreSQL text-search functions instead of raw casts, using the simple text search configuration on both sides so they stay consistent:

• Write (schema): to_tsvector('simple', search_attributes->>'TextNN').
  to_tsvector tokenizes arbitrary text and never fails on special characters.
• Read (query converter): each whitespace token is turned into
  plainto_tsquery('simple', <token>) and the tokens are OR-combined with the
  tsquery || operator, e.g.
  Text01 @@ (plainto_tsquery('simple', 'The') || plainto_tsquery('simple', 'Lost') || ...).
  plainto_tsquery normalizes the token, so significant characters no longer
  cause a parse error, and the OR-of-tokens semantics of the previous
  implementation is preserved.

simple folds case and splits on non-word characters without stemming or stopword removal. This keeps the token-based matching semantics of the old implementation, aligns case handling with the MySQL and SQLite stores (both are case-insensitive), and — importantly — matches the to_tsvector config so the plainto_tsquery lexemes still match the stored tsvector lexemes. (A language config like 'english' was rejected: Temporal has no per-namespace language knowledge, and stemming/stopword removal would make some exact-token queries silently match nothing.)

The read side is fixed in both query-converter implementations that exist on main:

• common/persistence/sql/sqlplugin/postgresql/query_converter.go
• common/persistence/visibility/store/sql/query_converter_legacy_postgresql.go

Alternatives considered

• Escaping values into valid tsvector literal syntax in Go before storing: avoids a migration and preserves case-sensitive matching, but the escaped form would land in search_attributes JSONB (which the API/UI return, so users would see mangled values), the escaping rules are fiddly, and the read-side ::tsquery bug would need separate escaping.
• Rejecting such values at the API boundary, or dropping the offending attribute on write: both are complementary hardening at best — they make ordinary human text an error (or silently drop it) on one backend only, and don't fix Text search itself.

Migration

• schema/postgresql/v12/visibility/schema.sql is updated for new
  installations.
• A new versioned upgrade script,
  schema/postgresql/v12/visibility/versioned/v1.15/fix_text_search_attributes_tsvector.sql,
  drops and re-adds the three generated columns with the new expression (which
  also drops and recreates the dependent by_text_0N GIN indexes).
• The embedded visibility schema version constant
  (schema/postgresql/v12.VisibilityVersion) is bumped 1.14 -> 1.15.

The migration recomputes the stored tsvector for every existing row. This is
safe: to_tsvector never errors, and any row already in the table must have had
a value the old raw cast accepted (rows with bad values were rejected at write
time and never stored), so nothing that previously stored can fail to recompute.

Compatibility considerations

• Existing stored tsvectors are recomputed by the migration. They change
  representation: the old raw cast preserved case ('World'), the new
  to_tsvector('simple', ...) lowercases ('world'). This is why the read side
  must change in lockstep — a raw ::tsquery (case-preserving) would stop
  matching the lowercased stored values.
• Read behavior change: Text matching becomes case-insensitive on
  PostgreSQL, matching MySQL and SQLite. OR-of-tokens matching is unchanged
  (maintainers may prefer AND-of-tokens; happy to change, but OR is what the
  current code does).
• The three-column rewrite in the migration rewrites the visibility table; on
  large visibility tables this deserves a maintenance window or a concurrent
  strategy.
• No change to MySQL, SQLite, Cassandra, or Elasticsearch visibility.

Tests / evidence

Run against a real PostgreSQL 13.5 (the repo's make start-dependencies
container), Go 1.26, -tags test_dep.

New end-to-end persistence test (write path)

common/persistence/sql/sqlplugin/tests visibility suite:
TestInsertReplaceSelect_TextSearchAttributesWithSpecialChars records a closed
workflow (InsertIntoVisibility + ReplaceIntoVisibility, the
RecordWorkflowExecutionClosed path) with
Text01 = "The Lost World: Jurassic Park" and Text02 = "it's a wonderful life", then reads the row back.

Before the schema fix (raw ::tsvector cast restored), against real Postgres:

--- FAIL: TestPQ/TestPostgreSQLVisibilitySuite/TestInsertReplaceSelect_TextSearchAttributesWithSpecialChars
    Received unexpected error:
      pq: syntax error in tsvector: "The Lost World: Jurassic Park" (42601)

After the fix:

--- PASS: TestPQ/TestPostgreSQLVisibilitySuite/TestInsertReplaceSelect_TextSearchAttributesWithSpecialChars (0.03s)

New query-converter unit tests (read path)

value with colon token cases added to both
common/persistence/sql/sqlplugin/postgresql and
common/persistence/visibility/store/sql converter tests: a value
The Lost World:Jurassic Park now renders as OR-combined plainto_tsquery
calls instead of an invalid ::tsquery cast. Both pass.

Migration test

tools/tests TestPostgres*/TestPostgreSQLUpdateSchemaTestSuite applies the
full versioned migration chain and reaches 1.15; passes for both the pq and
pgx plugins.

Regression

The full TestPQ/TestPostgreSQLVisibilitySuite, the schema embed test, and
the common/persistence/visibility/store/query tests all pass.

[CLAassistant]

All committers have signed the CLA.