[Feat]: knowledge-base agent (PDF/PNG/URL → markdown → RAG → graphRAG)

[FireTable]

Closes #13

Problem

No way to ingest a PDF into the app. KB v1 had a JSON store + injector but it was ditched for v2, and v3 (hybrid RRF + Reranker + @-mention resolution) is what this branch actually ships.

What changed

Backend

• backend/agent/kb-agent.ts — full ingestion pipeline: prepareKBDataNode → ocrNode → pageToMarkdownNode → generateChunkEmbedNode → rewriteMessagesNode. Wired as both a subgraph of mainAgent (chat path) and a top-level assistant (langgraph.json).
• lib/kb/{schema,queries,ingest,resolve,search,cache,extract,entityColor,screenshot,text,env}.ts — pgvector tables, hybrid RRF (BM25 + vector + tag), search_kb tool, LRU doc cache, LightRAG-style entity/relationship/theme extraction.
• lib/provider/model-registry.ts — five-way kind split (chat/ocr/embed/extract/rerank) + getOcrModelFromDB / getExtractModelFromDB / getEmbeddingModelFromDB / getRerankModelFromDB / RerankModel. kind= cache key prefix namespaces the LRU + round-robin counter per pool.
• lib/observability/{transform,aggregate}.ts + app/api/threads/[id]/observability/[parentMessageId]/route.ts — root chain detection for top-level runs.create invokes (Settings KB upload), aggregateRoot now consumes the transformed spans so failedCount is wire-accurate.
• lib/threads/{schema,queries}.ts — threads.kind ('chat' | 'kb') hides KB ingestion threads from the chat sidebar; sidebar filter is kind='chat'.
• backend/observability/callback.ts — CapturingHandler is a process-global singleton; wired on mainAgent, backgroundAgent, and kbAgent standalone compile.
• scripts/dev-stop.sh — replaced inline pnpm dev:stop (kills tsx watch wrappers + sweeps ports 2024/3000/3001).
• scripts/db-migrate.ts — seeded default model gets kind: ["chat"] so SELECT * FROM provider reads cleanly.

Frontend

• components/settings/kb-view/** — KB Settings tab (folder sidebar, doc table, doc detail, folder-level knowledge graph, observability list popover, live poll indicator).
• components/tool-ui/kb/** — chat-side rendering of search_kb (collapsible chunk list, leg badges, score formatting) + list_documents.
• components/assistant-ui/{directive-chip,directive-text,kb-mention,kb-mention-formatter}.{ts,tsx} — :kb-document[label]{documentId=…} / :kb-folder[label]{folderId=…} directive tokens survive the SDK wire because contentToParts strips sibling fields and filters unknown types.
• components/observability/{sheet,panel}.tsx — KB page wraps in <ObservabilitySheetProvider>; per-row Activity icon on <DocRow> opens the singleton Sheet against the right threadId/parentMessageId.
• components/landing/{features,how-it-works,cta}.tsx — KB bento card + How-it-works explainer.

API surface

• 9 KB endpoints (/api/kb/{upload,documents,documents/[id],documents/[id]/reprocess,documents/[id]/observability,folders,folders/[id]}) — see docs/APIS.md § Knowledge Base.
• 4 reprocess modes (full | chunksOnly | retryFailed | retryFailedChunks) with discriminated 4xx (409 PROCESSING / 409 ATTACHMENT_MISSING / 409 NOT_READY).

Docs

• docs/KNOWLEDGE_BASE.md — new (433 lines): pipeline, schema rationale, RRF math, Reranker stage, mention resolution, 4 reprocess modes, failure modes, env knob table, per-doc observability popover.
• docs/DB.md — kb_folder / kb_document / kb_chunk tables + 11 KB indexes.
• docs/APIS.md — all 9 KB endpoints + kind[] on the admin /api/admin/providers/** model endpoints.
• docs/OBSERVABILITY.md — KB ingestion runs (Settings → KB) section + Where capturingHandler is wired + kbAgent standalone compile.
• docs/TOOLS.md — search_kb / list_documents table + env knobs (Reranker, hybrid topK, mention budget). ALL_TOOLS → CHAT_TOOLS.
• docs/PROVIDERS.md — five-way kind split + new getters + RerankModel shape.
• docs/MEMORY.md — 2 graphs → 3 graphs (add kbAgent to topology).
• docs/ATTACHMENTS.md — link out to KNOWLEDGE_BASE.md now that KB v3 ships.
• docs/LANDING.md, README.md, CLAUDE.md — KB feature bullet + doc index entry.

Tests

1165/1165 passing (pnpm test, vitest). New coverage:

• tests/api/kb/{documents,folder,reprocess,documents/observability}.test.ts — every status code path on each KB route.
• tests/backend/{kb-agent,tool/kb,agent/resolve-entity-aliases}.test.ts — pipeline + entity alias resolution.
• tests/lib/kb/{cache,extract,ingest,queries,resolve,screenshot,search,entity-color}.test.ts — DB helpers, extract heuristics, ingest dispatch, resolve text rewriting.
• tests/lib/observability/aggregate.test.ts — aggregateRoot(capturedSpans, transformedSpans) wire-accuracy.
• tests/lib/provider/model-registry.test.ts — five-way round-robin + cache invalidation.
• tests/lib/threads/queries.test.ts — kind='chat' sidebar filter.
• tests/frontend/{directive-chip,directive-text,kb-list-documents-card,live-poll-indicator,landing/sections,settings/memory-view-group-threads}.test.tsx.
• tests/integration/kb-observability-roundtrip.test.ts — end-to-end upload → kb_observability row → popover list.

Pre-commit hooks (lint + typecheck + test) all green on the tip.

Docs

Per CLAUDE.md rules #1 and #10 — every API endpoint under app/api/ is reflected in docs/APIS.md; every tool added is reflected in docs/TOOLS.md; new feature lives in docs/KNOWLEDGE_BASE.md with cross-links to DB / observability / providers / attachments. Doc index in CLAUDE.md updated.

🤖 Generated with Claude Code

---

Summary by cubic

Ships the v3 Knowledge Base (issue #13): ingest PDFs to markdown with persisted pages, index chunks with 1024‑dim embeddings and LightRAG entities/relationships/themes, and search via hybrid RRF with optional Reranker, folder/doc filters, and graph views. Also fixes the vector-search leg and extract‑model fallback, and adds folder‑scoped lists, live polling, and tighter observability.

• New Features

  • kbAgent pipeline (OCR → markdown → chunks → embeddings → entities/relationships/themes) with concurrency, strict structured-output, and background indexing; pages are persisted for preview and fallback.
  • Hybrid search (BM25 + vector + entity-tag) now embeds the query and supports an optional Reranker with min‑score filtering; tools search_kb and list_documents return concise, folder‑grouped results.
  • KB Settings: folder sidebar, doc table, per‑doc detail (Markdown/Pages/Chunks/Graph), folder‑level knowledge graph with graphRAG coloring, hover tooltips, live polling indicator, and per‑doc run history popover; chunk/page badges show pending/parsing/success/failed.
  • Composer @‑mentions (:kb-document[label]{id=…} / :kb-folder[label]{id=…}) resolve to tool calls before routing; tool results inject numbered chunk context and sources render inline in chat.
  • 9 KB API routes (upload, folders, documents, reprocess, observability) and 4 reprocess modes (full, chunksOnly, retryFailed, retryFailedChunks), with accurate per‑page and per‑chunk statuses.
  • Model registry split into five kinds (chat, ocr, embed, extract, rerank) with admin UI support, per‑kind pools, caching, and clean fallbacks; threads gain kind='chat' | 'kb' to hide ingest threads.
  • Observability: wire‑accurate failed‑count aggregation, per‑turn sheets, doc‑scoped run lists, and inline failed‑span markers.
  • Docs updated: Knowledge Base design (pipeline, hybrid/Reranker, mentions, graphs), DB schema, APIs, tools, providers, observability, attachments, landing.

• Migration

  • Run DB migrations to install pgvector and create/alter KB tables/indexes (embedding dim 1024; entities to jsonb, relationships/themes added; per‑chunk status).
  • Configure provider models with kind[] for chat, ocr, embed, extract, and optional rerank (admin UI supports it).
  • Set KB env knobs in .env as needed (topK, token budget, chunk size, rerank min score).
  • If enabling Reranker, add a rerank model entry; otherwise search falls back to RRF.

^{Written for commit d94a16f. Summary will update on new commits.}

Greptile Summary

This PR ships the v3 Knowledge Base feature end-to-end: PDF/PNG/URL → OCR → markdown → chunks → pgvector embeddings → hybrid RRF search (BM25 + vector + entity-tag) + optional Reranker, with a full Settings UI, @-mention resolution in chat, and 9 new API endpoints. Two bugs in the search/model layer need fixing before the vector leg and extract-model fallback work as intended.

• lib/kb/search.ts: The auto-embedded query vector is stored in a local qvec variable but qvecLiteral still reads from args.qvec, which is null on every search_kb tool call — the pgvector leg silently no-ops for all normal searches.
• lib/provider/model-registry.ts: getExtractModelFromDB uses return promise inside try/catch without await; the fallback to the chat pool never runs when no extract-kind model is registered.
• Migration gap: 0005 creates embedding vector(1536) while the app expects vector(1024); 0006 fixes it, but a DB state where only the first migration has applied will reject every chunk insert.

Confidence Score: 3/5

The core RAG pipeline ships with the vector search leg completely disabled for all normal tool calls, and the extract-model fallback silently broken — both in new code that has not existed before in production.

The search_kb tool never activates the pgvector leg in any real call because qvecLiteral reads args.qvec (always null from the tool) instead of the locally-computed qvec. Every knowledge-base query runs keyword+tag only, silently degrading retrieval quality without surfacing any error. The extract-model fallback in getExtractModelFromDB is similarly inert: the missing await means the catch block is unreachable for async rejections, so deployments without an extract-tagged model will error instead of gracefully falling back to chat. Both issues are confined to two small code sites with straightforward one-line fixes, but they affect the primary value-add paths of the entire KB feature.

lib/kb/search.ts (vector leg disabled) and lib/provider/model-registry.ts (extract fallback broken) are the two files that need fixes before the KB feature works as described. The migration in db/migrations/0005_past_grey_gargoyle.sql is worth a second look for environments that run migrations incrementally.

Important Files Changed

Filename	Overview
lib/kb/search.ts	Hybrid RRF search (BM25 + vector + entity-tag). Critical bug: auto-embedded qvec local variable is never used in qvecLiteral — args.qvec is checked instead, so the vector leg never fires in the normal search_kb call path.
lib/provider/model-registry.ts	Five-way model pool (chat/ocr/embed/extract/rerank) with round-robin. getExtractModelFromDB fallback to chat pool is broken: missing await means the try/catch never intercepts the async rejection when no extract-tagged model exists.
backend/agent/kb-agent.ts	1399-line ingestion pipeline (prepareKBData → OCR → pageToMarkdown → generateChunkEmbed → rewriteMessages). Logic appears sound; relies on the search/extract fixes elsewhere.
lib/kb/ingest.ts	Shared fire-and-forget ingestion dispatch via langGraphClient.runs.create with multitaskStrategy: 'interrupt'. Thread upsert, messageId stamping, and mode/source plumbing all look correct.
app/api/kb/upload/route.ts	Upload endpoint: attachment ownership check, folder ownership check, content-hash dedup, doc row creation, fire-and-forget dispatch. Status guards and error handling look correct.
app/api/kb/documents/[id]/reprocess/route.ts	Four reprocess modes (full/chunksOnly/retryFailed/retryFailedChunks) with discriminated 409 guards. Mode parsing handles both ?mode= and legacy ?chunksOnly=true params correctly.
lib/observability/aggregate.ts	Signature extended to accept both CapturedSpan[] and WireSpanData[]; countRootFailures avoids double-counting by walking only leaf or childless-failed nodes. All callers updated correctly.
db/migrations/0005_past_grey_gargoyle.sql	Initial KB table DDL creates embedding vector(1536) but the application uses EMBEDDING_DIM=1024; the corrective ALTER is split into a separate migration 0006, leaving a transient dimension mismatch window.
lib/kb/env.ts	Typed env reader for KB knobs with sensible defaults. Module-level cache with test-only reset hook; correct use of parseInt/parseFloat with finite/positive guards.
lib/threads/schema.ts	Adds kind column (`'chat'

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant UI as Settings / Chat UI
    participant Upload as POST /api/kb/upload
    participant Ingest as lib/kb/ingest.ts
    participant kbAgent as kbAgent (LangGraph)
    participant OCR as ocrNode (vision LLM)
    participant Embed as generateChunkEmbedNode
    participant PG as Postgres (pgvector)
    participant Search as search_kb tool
    participant Rerank as RerankModel (optional)

    UI->>Upload: folderId attachmentId title
    Upload->>PG: "INSERT kb_document status=pending"
    Upload->>Ingest: fireIngestionRun
    Ingest->>kbAgent: "runs.create multitaskStrategy=interrupt"
    Upload-->>UI: 202 docId
    kbAgent->>OCR: PDF pages to markdown
    OCR-->>kbAgent: pages markdown
    kbAgent->>Embed: split chunks embeddings entity extract
    Embed->>PG: INSERT kb_chunk embedding entities themes tsv
    Embed->>PG: "UPDATE kb_document status=success"
    UI->>Search: search_kb query documentId
    Search->>PG: hybrid RRF BM25 kw vector vec entity tag
    PG-->>Search: ranked chunks
    Search->>Rerank: rerank query chunks if configured
    Rerank-->>Search: scored filtered results
    Search-->>UI: formatted chunk list

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant UI as Settings / Chat UI
    participant Upload as POST /api/kb/upload
    participant Ingest as lib/kb/ingest.ts
    participant kbAgent as kbAgent (LangGraph)
    participant OCR as ocrNode (vision LLM)
    participant Embed as generateChunkEmbedNode
    participant PG as Postgres (pgvector)
    participant Search as search_kb tool
    participant Rerank as RerankModel (optional)

    UI->>Upload: folderId attachmentId title
    Upload->>PG: "INSERT kb_document status=pending"
    Upload->>Ingest: fireIngestionRun
    Ingest->>kbAgent: "runs.create multitaskStrategy=interrupt"
    Upload-->>UI: 202 docId
    kbAgent->>OCR: PDF pages to markdown
    OCR-->>kbAgent: pages markdown
    kbAgent->>Embed: split chunks embeddings entity extract
    Embed->>PG: INSERT kb_chunk embedding entities themes tsv
    Embed->>PG: "UPDATE kb_document status=success"
    UI->>Search: search_kb query documentId
    Search->>PG: hybrid RRF BM25 kw vector vec entity tag
    PG-->>Search: ranked chunks
    Search->>Rerank: rerank query chunks if configured
    Rerank-->>Search: scored filtered results
    Search-->>UI: formatted chunk list

Loading

Prompt To Fix All With AI

Fix the following 3 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 3
lib/kb/search.ts:191
**Vector search leg silently no-ops on every `search_kb` call.** The auto-embedded vector is stored in the local `qvec` variable but `qvecLiteral` reads from `args.qvec`, which is always `null` when `search_kb` calls `hybridSearch` without pre-computing an embedding. As a result `hasVec` is always `false`, the `vec` CTE is never injected, and the hybrid search runs as keyword+tag only — the entire pgvector index is dead code for the tool path.

```suggestion
  const qvecLiteral = qvec != null ? vectorLiteral(qvec) : null;
```

### Issue 2 of 3
lib/provider/model-registry.ts:118-127
**`try/catch` never catches the async rejection — fallback to `chat` pool is dead code.** `getModelFromDB` is an `async` function that returns a `Promise`. Without `await`, the `try` block exits after handing back the Promise; if that Promise later rejects (e.g. no `extract`-tagged model registered), the rejection propagates up from the async function directly and the `catch` block is never entered. The designed fallback to the chat pool therefore never runs, and callers see an unhandled rejection instead.

```suggestion
export async function getExtractModelFromDB(opts: GetModelOpts = {}): Promise<BaseChatModel> {
  try {
    return await (getModelFromDB({ ...opts, kind: "extract" }) as Promise<BaseChatModel>);
  } catch (err) {
    // ponytail: no extract-tagged model registered yet → fall back to
    // chat pool. Same retry loop exists in the chat getter, so this
    // is the right pattern for "treat the pool as a soft contract".
    return getModelFromDB({ ...opts, kind: "chat" }) as Promise<BaseChatModel>;
  }
}
```

### Issue 3 of 3
db/migrations/0005_past_grey_gargoyle.sql:8
**Embedding column dimension mismatch between migrations.** `0005` creates `kb_chunk.embedding` as `vector(1536)`, but `lib/kb/schema.ts` sets `EMBEDDING_DIM = 1024` and migration `0006` immediately issues `ALTER COLUMN … SET DATA TYPE vector(1024)`. This two-step is harmless once both migrations run, but any environment that runs only `0005` ends up with a `1536`-dim column while the application inserts `1024`-dim vectors, producing a pgvector `22P02` error on every chunk insert.

_{Reviews (1): Last reviewed commit: "docs(kb): sync providers + memory topolo..." | Re-trigger Greptile}

Greptile also left 3 inline comments on this PR.

[FireTable]

@greptile-apps review please

[cubic-dev-ai]

All reported issues were addressed across 173 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[claude]

PR #38 Review — KB v3

Reviewed the KB feature end-to-end (backend pipeline, provider 5-way kind split, observability, threads filter, API endpoints, frontend, CLAUDE.md compliance). The scope is large; design is solid (dedup, reprocess modes, RRF + Reranker, per-doc observability). Four blockers and several majors below should land before merge.

Blockers

• B1. kbAgent standalone graph has no auth boundary. The Next.js route layer (POST /api/kb/upload, reprocess) is withAuth-wrapped, but langGraphClient.runs.create(..., "kbAgent", { config: { configurable: { userId } } }) trusts whatever userId is on the wire. A caller that can reach LANGGRAPH_API_URL can spoof config.configurable.userId = "victim-id" and ingest attacker-chosen content into the victim's KB namespace (data corruption / cross-user namespace injection). The chat resolver drops unknown kb_refs so this is corruption, not exfil — but it still violates the per-user boundary. Fix: have prepareKBDataNode (or a middleware) verify threads.userId === configurable.userId AND threads.kind === "kb".

• B2. mode=retryFailed is a no-op due to the chunksOnly short-circuit at backend/agent/kb-agent.ts:521. The reprocess route returns 202 "OCR re-running on failed pages", but the early-return swallows it. generateChunkEmbedNode's isRetryFailedChunks branch finds zero status='parsing' chunks and exits. The doc is left at status='parsing' with no OCR work having run. Fix: narrow the chunksOnly guard, or drop mode=retryFailed and forward to mode=full.

• B3. Full reprocess wipes data before validating the attachment (app/api/kb/documents/[id]/reprocess/route.ts:196-211). Order: (1) withKbTx { deleteKbChunksByDocumentId; resetKbDocumentForReprocess }, then (2) if (!doc.attachmentId) return 409 ATTACHMENT_MISSING. A 409 here means the derived data was already destroyed. Two clicks → permanent loss of indexed chunks. The 409 must be side-effect-free — validate attachment BEFORE entering the reset tx.

• B4. modelKwargs: { reasoning_split: true } is hard-coded for every chat/ocr pick (lib/provider/model-registry.ts:250). CLAUDE.md says "strip when switching providers". The inline comment "only minimax reads this" understates the impact: the kwarg is sent for OpenAI/Anthropic/Google when an admin adds them via the provider UI. Strict upstreams (or future 4xx-on-unknown-fields adapters) will break. Mirror lib/credit/build-model.ts:40-46 and gate on picked.providerId === "minimax".

Major

• M1. currentUserId module-level mutable in backend/tool/kb/user-id.ts is dead code in production (setKbToolUserId has zero prod callers) AND is a TOCTOU hazard under concurrent requests. Remove or wire up.
• M2. CapturingHandler singleton race on currentParentMessageId (lib/observability/callback.ts:260 + :281-283 + :531) — concurrent invokes overwrite each other's parent message id. The DB backfill only handles NULL cases, not misattribution. Use Map<runId, string | null> keyed by ancestor chain.
• M3. Missing FK on kb_observability.thread_id (db/migrations/0010_happy_thanos.sql:7 + lib/kb/schema.ts:233). Thread delete cascades observability_spans but leaves kb_observability rows pointing at a deleted thread — the popover returns a phantom threadId.
• M4. invalidateModelCache(key) clears ALL round-robin counters (lib/provider/model-registry.ts:259-266) — wiping unrelated pool rotation state. nextTupleIndexByKey.delete(key) instead.
• M5. getExtractModel lacks env fallback (backend/model.ts:110-112) — fresh deploy with no DB providers crashes on first ingest. Mirror getChatModel shape.
• M6. RerankModel defaults to Cohere URL for non-Jina providers (lib/provider/model-registry.ts:150-157). Adding a Voyage provider silently 401s on Cohere. Require baseUrl on the picked tuple.
• M7. R2 page PNGs are never cleaned up after doc delete/reprocess (lib/kb/queries.ts:613-636 + backend/agent/kb-agent.ts:457-459). Heavy KB users balloon R2 quota. Enqueue a R2 batch-delete on doc delete.
• M8. mode query param is not validated — unknown values silently default to "full" (app/api/kb/documents/[id]/reprocess/route.ts:46-62). A typo retryFailedChunk triggers destructive full reprocess. Parse with z.enum([...]), return 400 INVALID_MODE.
• M9. fireIngestionRun failure returns 202 — leaves the doc in a permanently unretryable state (app/api/kb/documents/[id]/reprocess/route.ts:106-129,168-188,212-228 + app/api/kb/upload/route.ts:92-104). Row stays at pending/parsing, then 409 PROCESSING blocks retry. Either mark failed and return 502/503, or only 202 when runs.create resolved.
• M10. application/pdf MIME not enforced in /api/kb/upload (app/api/kb/upload/route.ts:35-42 + .env.example:172) — non-PDFs are accepted and stuck pending. Default R2_ALLOWED_CONTENT_TYPES excludes PDF, so Settings upload rejects PDFs at presign under default config. Add application/pdf to allowlist default + reject non-PDF in /api/kb/upload with 400 UNSUPPORTED_CONTENT_TYPE.

Frontend / Tool-UI (CLAUDE.md rules #6, #7)

• components/settings/kb-view/doc-detail-dialog.tsx:490,564,713 — shadow-sm/hover:shadow-md on cards. Same in components/settings/kb-view/knowledge-graph.tsx:555,608,654,724,774.
• components/tool-ui/kb/list-documents-card.tsx:47-61 — CollapsibleTrigger renders ChevronRightIcon + FolderIcon + folder name + count inside a single button (rule feat(memory): cross-conversation memory + thread summarize + dual-graph runtime #7).
• components/settings/kb-view/doc-detail-dialog.tsx:449-464 — Copy button renders <Copy /> + "Copy" inside one button.
• components/settings/kb-view/folder-sidebar.tsx:127-158 — dropdown trigger is opacity-0 until row-hover; keyboard users never see it. Add group-focus-within/folder:opacity-100.
• components/landing/how-it-works.tsx:7-12 — all six landing demos (incl. new 499-line kb-explainer-demo + 8 lucide icons + framer-motion) are statically imported. Wrap with next/dynamic({ ssr: false }).

CLAUDE.md rule #1 (APIS.md sync) gaps

• ?mention=1 documented as flat {docs:[…]} but route returns grouped {folders:[…]}.
• Single-doc GET docs miss pages, entities, relationships, themes, per-chunk status/errorMessage, and call the position index instead of ordinal.
• Full reprocess docs claim 202 returns mode but route returns {docId} only.
• GET /api/kb/folders/[id] is undocumented.
• in_flight_runs docs include thread_id/passthrough metadata but the route strips both.

Other (follow-up)

• lib/kb/queries.ts:529-534 markAllKbChunksParsingForDocInTx and :572-578 getKbChunkFailureCount are exported but unreferenced in the diff — dead or future use; remove if unused.
• app/api/kb/folders/route.ts:14-16 + app/api/kb/folders/[id]/route.ts:37-39 Zod .min(1).max(64).trim() ordering — .trim() runs AFTER length checks, so a 64-space string passes and gets normalized to empty. Use .trim().min(1).max(64).
• app/api/kb/upload/route.ts:50-88 content-hash dedup race — two concurrent uploads both miss lookup, one gets 23505 instead of deduped 200. Handle wrapped .cause.code === "23505".
• app/api/kb/folders/[id]/route.ts:86-99 folder delete race — concurrent doc insert triggers 23503 RESTRICT → undocumented 500. Use tx or catch wrapped 23503.
• app/api/kb/folders/route.ts:37-44 + :72-75 dup-name handlers check only err.code; drizzle wraps 23505 under err.cause.code.
• app/api/kb/documents/[id]/reprocess/route.ts:147-188 retryFailedChunks returns 202 on no-op when no failed chunks exist.
• lib/kb/search.ts:411-414 textArrayLiteral escapes \\ and " only — defense-in-depth: also escape \n.
• lib/kb/search.ts:404-406 vectorLiteral does not check Number.isFinite(v) — NaN/Infinity produces [NaN,Infinity,…] which pgvector rejects as 22P02.
• lib/observability/queries.ts:13-44 FORBIDDEN_VALUE_RE is narrower than FORBIDDEN detection — a custom apikey/secret_key key logs a warn but is not actually masked.
• lib/provider/model-registry.ts:159-171 RerankModel.rerank fetch has no AbortSignal.timeout.
• lib/provider/model-registry.ts:351-357 (in search.ts) rerankResult.map(item => normalizedRows[item.index]) has no out-of-bounds guard.
• lib/provider/model-registry.ts:67 stale "Four prefixes" comment — should be five.
• lib/provider/model-registry.ts:274-276 resetRoundRobinCounters is redundant with invalidateModelCache().
• backend/model.ts:118-124 getRerankModel wrapper is dead code (no importers).
• backend/agent/kb-agent.ts:458 state.userId interpolated into R2 key path with no format validation (currently safe; defense-in-depth missing).
• lib/kb/screenshot.ts:21-40 + lib/kb/text.ts:27-44 mupdf Document never destroyed — native memory leak per ingest.
• app/api/threads/[id]/observability/[parentMessageId]/route.ts:26-39 + lib/observability/callback.ts:585-596 route comments claim spans persist on Start; callback only persists on End/Error. Pick a coherent lifecycle and update docs.
• tests/lib/provider/model-registry.test.ts:245 describe block titled "kind partition (ocr / embed)" — extract + rerank paths uncovered.
• components/assistant-ui/lexical-directive-chip.tsx:1 missing "use client" directive.
• components/settings/kb-view/doc-detail-dialog.tsx:156-245 skeleton dialog missing aria-busy="true" on <DialogContent>.
• components/settings/kb-view/doc-table.tsx:247-308 role="row" on wrapper div with no table/rowgroup ancestor (dangling ARIA).
• components/settings/kb-view/knowledge-graph.tsx:236-254 dedupFingerprint recomputes on every poll.

Posting inline comments on the four blockers. Happy to follow up on any specific finding.

[greptile-apps]

Too many files changed for review. (178 files found, 100 file limit)

Bypass the limit by tagging @greptile-apps to review.

[claude]

PR 38 review (KB v3)

A lot of moving parts in this PR (~31k adds, 2 new graphs, 9 new routes, 3 split migrations). The KB feature is well-scoped overall — auth wrapping, cross-user 404s, dedup + idempotency on uploads, and the claude-runtime + withAuth patterns are all in place. Five things worth fixing before merge; the top three affect the primary value-add paths.

Critical (these break primary flows)

1. lib/kb/search.ts:191 — vector leg dead code in search_kb tool path. Auto-embedded qvec is local; qvecLiteral reads args.qvec (always null from the tool handler). hasVec is false, the vec CTE is dropped, every search_kb call degrades to BM25 + tag only. Fix: const qvecLiteral = qvec != null ? vectorLiteral(qvec) : null;

2. lib/provider/model-registry.ts:118-127 — getExtractModelFromDB chat fallback is unreachable. Missing await on getModelFromDB(...); the returned Promise's rejection bypasses the try/catch. Deployments without an extract-kind model error instead of falling back to chat. getChatModelFromDB works only because it's a one-liner async return (auto-unwraps); the bug is unique to the two-step fallback.

3. db/migrations/0005_past_grey_gargoyle.sql:8 vs 0006_strong_karen_page.sql — column created as vector(1536) then immediately ALTER … SET DATA TYPE vector(1024). Normal pnpm db:migrate runs both so it's invisible, but a partial migration by hand leaves the schema on 1536 while the app inserts 1024-dim vectors → pgvector 22P02 on every chunk insert. Same user-visible failure as issue feat(001): stage 1 — user auth (Better Auth + email verification + thread ownership) #1 (vector leg silently disabled) from a totally different angle. Fix: collapse into one migration at the right dim, or add a loud dependency note.

Moderate

4. tests/lib/kb/search.test.ts — the qvec bug has no test coverage. Every case passes qvec: makeEmbedding(...) explicitly, so the auto-embed path through the tool handler is never exercised. Mocking getEmbeddingModel is in place (line 17-27) but unused. A single test that calls hybridSearch({ query, userId }) without qvec and expects legsHit: ["vec"] would have caught this.

5. lib/kb/queries.ts:469 — comment still says "1536-dim embedding". Schema is 1024. Cosmetic, but the dim reference in a comment will mislead the next person who touches findKbChunksContentByDocumentId.

Verified-clean areas (so authors know what was actually checked)

• API auth: all 9 /api/kb/** routes wrapped with withAuth, cross-user doc/folder ids return 404, no auth leaks on the reprocess/observability routes.
• User scoping: lib/kb/search.ts (valid_docs CTE, lines 301-304), lib/kb/queries.ts findKbDocumentById / findKbChunksByDocumentId / listKbDocumentsGroupedWithAttachment, and lib/kb/cache.ts (keyed by ${userId}:${docId}) all honor user.id. Cross-user kb_ref mentions are dropped to null in lib/kb/resolve.ts:124-128.
• Docs sync (rule feat(001): stage 1 — user auth (Better Auth + email verification + thread ownership) #1, [Bug]: Memory tab Socials row missing email per linked provider #10): docs/APIS.md § Knowledge Base covers all 9 endpoints with status codes + 409 discrimination; docs/TOOLS.md table updated for search_kb / list_documents; new docs/KNOWLEDGE_BASE.md (433 lines) plus cross-links in DB.md / OBSERVABILITY.md / PROVIDERS.md / MEMORY.md / ATTACHMENTS.md; CLAUDE.md doc-index row added; README.md bullet + link present. Two minor doc nits:
  • docs/TOOLS.md:109-110 references the source file as kb.ts, but the tool sources are now in backend/tool/kb/ (search-kb.ts, list-documents.ts).
• Env contract (rule [Feat]: chat attachments backed by Cloudflare R2 #12): KB env reads go through lib/kb/env.ts (typed, validated, defaults). No new NEXT_PUBLIC_* introduced — R2_PUBLIC_BASE_URL is server-only (NextResponse + fireIngestionRun).
• Reprocess modes: all 4 modes (full, chunksOnly, retryFailed, retryFailedChunks) gated cleanly with the documented 409 shapes (PROCESSING / ATTACHMENT_MISSING / NOT_READY); legacy ?chunksOnly=true still parsed.
• fireIngestionRun uses multitaskStrategy: 'interrupt' (latest-wins on same thread) — correct vs triggerBackgroundAgentNode which uses 'enqueue'.

Out-of-scope but worth a follow-up

• lib/kb/search.ts:33-38 flags kb_chunk.page_numbers as a missing column that's punted. UI degrades to "Page N" hover from kb_document.pages only. Fine for v3, will bite v4.
• app/api/kb/documents/[id]/route.ts:46 — DELETE on a doc leaves the R2 attachment + kb-tmp/* PNGs behind ("v3 retention sweep can clean them up"). Worth filing as a leak until the sweep exists.

Posted inline comments at the four critical lines above.