feat(db): add MySQL as metadata database option, with parity fixes and CI coverage

[lyingbug]

Description

This builds on #2235 by @yuefanxiao, which adds MySQL as a business/metadata database option for issue Tencent#1418. That PR is the strongest implementation of the batch: it introduces a real dialect layer (internal/database/dialect.go plus capability predicates such as dialectSupportsRowLocking) instead of scattered if dialect == "mysql" branches, and its MySQL schema matches the PostgreSQL schema table-for-table and column-for-column.

I merged it onto current main (resolving one test-file conflict by keeping both sides' new tests) and then fixed the gaps that a review against a live MySQL 8.0 server and a live PostgreSQL 17 reference turned up.

1. The metadata prefix sweep could not run on MySQL

FindByMetadataKeyPrefix built its JSON extraction by hand as metadata->>'<key>'. MySQL requires a '$.key' path there and rejects the bare-key form with Error 3143 as soon as any row holds non-null JSON. Its only caller logs and returns, so the failure was silent: on MySQL, re-syncing a datasource stopped sweeping sub-items that had disappeared upstream, accumulating orphan knowledge rows behind a single warning line.

The expression now goes through the dialect helper. A new JSONPathExprIndexed also substitutes the generated column MySQL indexes, because MySQL rewrites an indexed generated column only for equality-shaped predicates and never for LIKE — the mapping lives in one place in internal/database rather than as a bare column name at each call site.

2. Schema divergences from PostgreSQL

Found by diffing information_schema between a MySQL database built from migrations/mysql and a PostgreSQL database built from migrations/versioned:

• chunks.seq_id and knowledge_tags.seq_id started at 1 instead of the sequence start values 100000000 and 10000000. FAQ import lets a caller pin a seq_id below the start value (documented on types.FAQImportEntry.ID), so that range is reserved for imports and generated values must stay out of it.
• sessions.fallback_response defaulted to the empty string instead of the seeded answer, so a MySQL deployment answered nothing where PostgreSQL answers a message.
• knowledges had no equivalent of the expression index behind the datasource external-ID lookups, and im_channel_sessions was missing the im_channel_id index, leaving both as full scans. The datasource sync runs one such lookup per item.

The generated column is LONGTEXT rather than VARCHAR(n) deliberately: the extracted value is unbounded on PostgreSQL, and a typed generated column fails the entire INSERT with Error 1406 once a value overflows it (verified on the server).

3. A PostgreSQL-side regression in the regex guard

The portable-regex guard rejected every letter escape, which took \d, \s and \w with it. Those mean the same thing to PostgreSQL ARE, MySQL ICU and Go RE2, so banning them only cost the agent expressiveness on deployments that already worked.

\b stays rejected, and it is the reason the guard exists at all. Verified on both servers:

MySQL       SELECT REGEXP_LIKE('a rag b', '\\brag\\b', 'i')  -> 1     (word boundary)
PostgreSQL  SELECT 'a rag b' ~* '\brag\b'                    -> false (literal backspace)

One pattern, different matches per deployment — so it is genuinely non-portable, unlike the character classes.

4. The missing test dimension

Every repository test ran on SQLite, which accepts SQL that MySQL rejects — the bare-key ->> form above is exactly that shape of bug, and it passed the SQLite suite. So this adds an integration suite that applies migrations/mysql to a throwaway database and drives the real repositories through the dialect-sensitive query paths, plus a CI job with a mysql:8.0 service so the suite is not permanently skipped.

Type of Change

• 🐛 Bug fix
• ✨ New feature
• 💥 Breaking change
• 📚 Documentation update
• 🎨 Refactor
• ⚡ Performance improvement
• 🧪 Test
• 🔧 Configuration / Build / CI

Related Issue

Closes Tencent#1418

Builds on #2235 (@yuefanxiao). Related implementations of the same issue: #2320 (@WHUTcjh-2024), #2264 (@mingri31164), #2132 (@wei-yan1), #1904 (@Arreboi06).

Testing

Verified against a real MySQL 8.0.46 server and a real ParadeDB/PostgreSQL 17 server, not just SQLite.

MySQL integration tests — the new suite, all passing:

--- PASS: TestMySQLFindByMetadataKeyPrefix
--- PASS: TestMySQLFindByMetadataKey
--- PASS: TestMySQLMetadataExternalIDUsesIndex/equality
--- PASS: TestMySQLMetadataExternalIDUsesIndex/prefix
--- PASS: TestMySQLUpdateChunks
--- PASS: TestMySQLSeqIDStartsAboveReservedRange
--- PASS: TestMySQLSessionDefaultsMatchPostgres
--- PASS: TestMySQLCaseInsensitiveSearches
--- PASS: TestMySQLTaskQueueClaimAndFail
ok  github.com/Tencent/WeKnora/internal/application/repository

The suite actually catches the bug it was written for. Reverting only the FindByMetadataKeyPrefix fix and re-running:

Error 3143 (42000): Invalid JSON path expression. The error is around character position 1.
--- FAIL: TestMySQLFindByMetadataKeyPrefix

Schema parity — MySQL schema versus the PostgreSQL reference built from migrations/versioned, comparing tables, columns, nullability and defaults:

缺失表: 无
缺失列: 无
更严列: 2 -> ['sync_logs.started_at', 'wiki_page_revisions.aliases']   (both have defaults; inserts unaffected)
缺默认值(非 .id): 0                                                    (18 .id columns come from BeforeCreate hooks)

Migrations apply cleanly (51 tables) and 000000_init.down.sql drops back to 0 tables. The AUTO_INCREMENT start values now read chunks=100000000, knowledge_tags=10000000, tenants=10000.

Index usage confirmed with EXPLAIN: the equality form reaches the index through generated-column substitution, and the LIKE-prefix form reaches it by naming the generated column.

No PostgreSQL regression. migrations/versioned/ is untouched, and all 79 migrations still apply to a fresh PostgreSQL 17 database with 0 failures.

Repository-wide checks:

go vet ./...  (excluding docreader)   -> clean
go test ./... (excluding docreader)   -> 65 packages ok, 0 failures
gofmt -l <files changed in this PR>   -> clean
git diff --check upstream/main...HEAD -> clean

golangci-lint is not installed in this environment, so diff-scoped lint was not run.

Checklist

• git diff --check origin/main...HEAD passes
• Changed source files are formatted
• Targeted tests for the changed packages/components pass
• Diff-scoped lint passes where applicable (golangci-lint unavailable in this environment; go vet is clean)
• Full-repository checks were run, or any unrelated/environment-dependent failures are documented above
• Self-reviewed the code
• Added/updated tests covering the change
• Updated related documentation (docs/mysql-primary-database.md)
• Breaking changes are clearly called out in the description above

Screenshots / Recordings

Not applicable — no user-visible UI changes.

Known follow-ups (not in this PR)

• MySQL Wiki search stays a substring match rather than full-text; docs/mysql-primary-database.md documents the difference. An ngram FULLTEXT index would close it but changes ranking semantics, so it belongs in its own change.
• The utf8mb4_0900_ai_ci collation makes human-facing unique constraints (users.username, organizations.invite_code, various name columns) case-insensitive, where PostgreSQL is case-sensitive. This may well be the behavior MySQL users expect, but it deserves an explicit decision rather than being inherited from the database default.
• migrations/mysql is a single consolidated baseline, so nothing fails the build when a new migrations/versioned migration lands without a MySQL counterpart. The parity diff used here is scriptable and would make a good CI guard.