feat(knowledge): multi-level document folders with folder-scoped Q&A, made mergeable

[lyingbug]

Description

This is Tencent#2234 by @Alorun (multi-level folders for document knowledge bases plus folder-scoped Q&A), merged with current main and with the blockers that kept it out of the tree fixed. Credit for the feature itself belongs to that PR; the commits after the merge commit are the additions.

Why Tencent#2234 as the base

Of the 15 community PRs targeting Tencent#1311 I compared, Tencent#2234 is the one whose architecture holds up: the folder subtree filter for document listing is pushed into SQL as a recursive CTE instead of materializing an ID list, tenant isolation is enforced by composite foreign keys in PostgreSQL (with equivalent triggers on the Lite/SQLite path) rather than only in the application layer, FolderScope has a presence-aware JSON contract so a malformed restricted scope fails closed instead of silently widening to the whole knowledge base, and it is the only one that patches the shared agent-tool authorization helpers (searchTargetScope, searchTargetIsWholeKB) so a folder-scoped target cannot be treated as whole-KB access. It also has the most complete frontend (drag-and-drop moves, breadcrumb, resizable sidebar, folder mention scope) and all four locales translated with a key-completeness guard.

What this branch adds on top

The server could not start. The folder migration sat on version 000071, which main already uses for 000071_platform_api_keys. golang-migrate refuses to open a source directory holding two files with the same version, so RunMigrationsWithOptions failed on boot. This was already true on the PR's own branch, not just after a rebase. Moved to 000079 for PostgreSQL and 000002 for the separate sqlite sequence (it had jumped to 000071 there too, which would have made every future Lite migration between 000002 and 000070 unreachable).

The migration is now replayable. Added IF NOT EXISTS guards, plus a pg_constraint lookup for the foreign key that has no such form, matching how the surrounding migrations support dirty-state recovery. Replaying the original against an already-migrated database fails with relation "folders" already exists; this version skips every statement and exits 0.

Folder depth was unbounded. Any client could build a chain thousands of levels deep with ordinary API calls, and both the recursive CTE behind folder-filtered listing and the ancestor walk behind move validation grow with that depth, so one tenant's nesting choice became every later request's cost. Folders past MaxFolderDepth (16) are now rejected. A move is validated against the height of the subtree being moved, not just the target, because the subtree travels with the folder. SubtreeHeight carries its own depth counter in the recursion and stops at the supplied bound, so rows that are already deeper than the cap — or that form a parent cycle — produce a bounded result instead of an unbounded scan.

Every folder-scoped question took write locks. ListByIDs only checks that a requested scope still exists, but it took SELECT ... FOR UPDATE on those rows, so concurrent questions about the same folder serialized against each other and blocked renames and moves behind them. It is now lock-free; the structural writes that need a lock still go through GetByIDForUpdate.

A folder scope could resolve to an unbounded document filter. Folder scoping works by handing the retrieval engines an explicit document-ID filter built from the whole subtree, with no ceiling. At the hundred-thousand-document scale this issue describes, asking a question against a top-level folder builds a filter far past what the backends accept — Elasticsearch and OpenSearch cap terms clauses at 65536 entries, PostgreSQL caps bind parameters at 65535 — which either errors out or, worse, truncates the filter and answers from documents the user did not select. The scan now stops one row past MaxFolderScopeKnowledgeIDs and the request is rejected with a 400 pointing the user at a more specific folder or at asking across the whole knowledge base, which needs no ID filter at all. Failing loudly keeps the scope honest. Pushing folder_id down into the chunk indexes is the better long-term fix and would lift the limit entirely; that belongs in its own change since it touches all ten retriever implementations.

Two test suites were red. folder_migration_test.go pinned the old Lite migration version, and UploadConfirmDialog.test.ts asserts on the source text of KnowledgeBase.vue, whose executeUploadBatch and executeUrlImport now take the upload target as an argument. Both assertions were updated; the behaviour they guard is unchanged.

Known follow-ups, not addressed here

• The table is named folders, while comparable sub-resources use a knowledge_ prefix (knowledge_tags, knowledge_tag_relations). Renaming is cheap now and expensive later.
• Nothing restricts folders to document-type knowledge bases, so they can also be created on wiki-type bases that already have their own directory tree in WikiBrowser.vue.
• Folder listing has no pagination; all=true returns the whole tree for a knowledge base.

Type of Change

• 🐛 Bug fix
• ✨ New feature
• ⚡ Performance improvement
• 🧪 Test

Related Issue

Fixes Tencent#1311

Testing

All commands run on this branch. PostgreSQL 16 was installed locally for the migration checks; app.skip_embedding=true skips the one migration that needs pgvector.

Startup blocker — golang-migrate source directory, opened with the same driver the server uses:

--- PR #2234 as submitted:
OPEN FAILED for migrations/versioned: duplicate migration file: 000071_platform_api_keys.down.sql
OPEN OK for migrations/sqlite: 3 migrations, first=0 last=71
--- this branch:
OPEN OK for migrations/versioned: 80 migrations, first=0 last=79
OPEN OK for migrations/sqlite: 3 migrations, first=0 last=2

Full versioned chain up, folder migration down, then replayed, on real PostgreSQL 16:

UP OK       -> version=79 dirty=false
DOWN OK     -> version=78 dirty=false
REPLAY OK   -> version=79 dirty=false

Idempotency, replaying the up migration against an already-migrated database:

--- PR #2234 as submitted:
ERROR:  relation "folders" already exists
--- this branch:
psql exit=0, 7 statements skipped

New tests, plus the existing folder and search-target tests:

--- PASS: TestFolderServiceCoreLifecycleAndSafety
--- PASS: TestFolderServiceCreateRejectsFoldersBeyondMaxDepth
--- PASS: TestFolderServiceMoveRejectsSubtreeThatWouldExceedMaxDepth
--- PASS: TestFolderServiceMoveToRootAlwaysFitsWithinMaxDepth
--- PASS: TestFolderRepositorySubtreeHeight
--- PASS: TestFolderRepositorySubtreeHeightTerminatesOnCyclicRows
--- PASS: TestKnowledgeRepositoryListIDsByFolderScopesStopsOneRowPastLimit
--- PASS: TestBuildSearchTargetsRejectsOversizedFolderScope
--- PASS: TestSQLiteFolderMigrationUpDownAndScopedConstraints
(plus all pre-existing TestFolder*, TestKnowledgeRepositoryListIDsByFolderScope*, TestBuildSearchTargets* cases)

CI-equivalent suite:

go vet ./... (excluding docreader):            OK
go test ./... (excluding docreader): exit=0    64 packages ok, 0 FAIL
go build ./cmd/server:                         OK
gofmt on changed Go files:                     clean
golangci-lint run --new-from-rev:              0 issues for this branch's diff
frontend: npm test                             331 tests, 331 pass, 0 fail
frontend: npm run type-check                   exit=0
frontend: npm run build                        exit=0

Checklist

• git diff --check origin/main...HEAD passes
• Changed source files are formatted
• Targeted tests for the changed packages/components pass
• Diff-scoped lint passes where applicable (golangci-lint run --new-from-rev reports 0 issues for this branch's diff; the pre-existing lll/revive findings inherited from Feat/multi level folders Tencent/WeKnora#2234 are listed as follow-ups)
• Full-repository checks were run
• Self-reviewed the code
• Added/updated tests covering the change
• Updated related documentation — Swagger annotations for the folder endpoints come from Feat/multi level folders Tencent/WeKnora#2234 and were not regenerated here
• Breaking changes are clearly called out in the description above

Screenshots / Recordings

No UI change on top of Tencent#2234. The additions are the migration versioning, the depth cap, the read-path lock removal, and the folder-scope size bound, all of which are covered by the migration runs and tests above.