perf: stop reading whole session files, and index them incrementally

[Davidb-2107]

Summary

Claude session transcripts can grow to hundreds of megabytes. Reading the entire file again on each append creates large transient allocations in Switchboard's main process, even when only a little session metadata has changed.

This PR keeps @Davidb-2107's original chunked-reader and incremental-indexing contribution, adapted to the current harness and database architecture. Current main was merged into the contributor branch without rewriting the original commits.

Changes

• Move the incremental parser into harnesses/claude.js, preserving the current runtime, sessionFile, fileMtime, title handling and transcript timestamp bounds. Cached parser state survives database reopen, so subsequent appends resume at the previous byte offset.
• Add resume-state columns through schema inspection, including databases whose version was advanced by another branch. The additions preserve existing rows, folder metadata, user names, stars, archive state and search data.
• Keep frequent bulk cache queries free of parser text/state while retaining the fields needed by Claude and Codex consumers. Codex parsing remains unchanged.
• Use bounded reads for project-path discovery, session-slug lookup and the scheduler's JSONL fallback. Retain perf(scheduler): resolve projectPath from cache_meta instead of re-reading JSONLs #65's cache-first scheduler lookup.
• Handle long JSONL lines with one concatenation per line, avoiding quadratic buffer copying. Preserve UTF-8 characters split across chunks.
• Scan the same descriptor used for validation, stopping at the initial file size. Read errors do not return a successful partial parse. Record the folder's index timestamp before scanning so concurrent appends remain eligible for reconciliation.

Resume safety

Reset state when the head changes, the file is replaced or truncated, or a changed file has not grown. Version the saved state so older parser rows without timestamp accumulators receive a full read. A final line without a newline is displayed where parseable but disables resume until a later full pass can account for it safely.

This remains an optimization for append-only transcripts: an in-place edit beyond the 4 KiB guard that also grows the same file can evade validation and requires a full re-index. It does not claim to detect arbitrary edits throughout a transcript.

Validation

npm test: 192 passed, 0 failed on the updated branch (172 current-main tests plus 20 indexing/scanner/regression tests).

Coverage includes:

• Incremental results versus full reads; titles and raw/displayed timestamp bounds.
• Truncation, atomic replacement with an unchanged prefix, same-size rewrites, partial records, and legacy resume state.
• UTF-8 across chunk boundaries, linear copying for multi-megabyte lines, bounded reads, concurrent appends and read failures.
• Version-4 and higher-version/partially migrated SQLite databases, preservation of metadata/search, and idempotent reopen.
• A complete cache refresh after database reopen: an append to a 2 MiB fixture reads less than 512 KiB total, including project-path discovery and hash validation. Search, user naming, display timestamps and Codex rows remain intact.
• Syntax checks and git diff --cached --check pass.

Tests used existing local dependencies via NODE_PATH, with SQLite checks under Electron's Node mode in disposable data directories. No live-user transcript, running application, or original checkout was modified. The contributor's original whole-application RSS benchmark has not been rerun on this port; the current performance checks measure bounded I/O and copying. GitHub platform builds run on the pushed commit.

[abasiri]

Reviewed this properly — it's careful work and the diagnosis holds up on my machine too. For calibration: 3,290 session files, 3.1 GB, largest 136.6 MB. Under the current code an append to that file re-reads all of it into a JS string — roughly 273 MB of transient allocation — every few seconds while the session is live. Your sawtooth reproduces.

Heads up: I pushed a merge commit to this branch (5016379) fixing the two things that were blocking it. Both were merge-order artifacts, not defects in your work — #60 and #65 landed after you wrote this. Details below so nothing is a surprise.

What's done right

jsonl-scan.js avoids both classic chunked-reader bugs, which is not the common outcome. It carries the partial line as a Buffer rather than a string, so a multibyte UTF-8 sequence split across a chunk boundary survives; and it copies out of the reused allocUnsafe scratch buffer instead of aliasing it into data. The byte accounting is exact — data always begins at consumed — which is what makes the resume offset trustworthy rather than approximately right.

The resume-safety triad is the part I'd have most expected to find wrong, and it's complete: headHash catches an in-place rewrite, indexedBytes <= stat.size catches truncation, and a trailing line with no newline sets resumable = false so it can't be double-counted on the next pass.

The migration is additive with no cache wipe — NULL indexedBytes reads as "can't resume", so each session re-reads once and goes incremental after. And the tests assert the property that actually matters: incremental == full re-read.

I also specifically audited the riskiest refactor here, narrowing cacheGetAll from SELECT * to an explicit list. buildProjectsFromCache reads only aiTitle, created, firstPrompt, messageCount, modified, projectPath, sessionId, slug, summary — all still selected — and the customTitle consumers operate on freshly-read sessions, not cached rows. Safe.

What I changed in 5016379

schedule-runner.js conflict. #65 made scanSchedules prefer a cache_meta folder→projectPath lookup, falling back to a JSONL head read only for folders missing from the cache. That collided textually with your change to the same read — but the two compose. I kept #65's structure and moved readHead(…, 4096) inside its readProjectPathFromJsonl fallback, so the common path does no file read at all and the fallback no longer loads a 136 MB file to inspect its first line.

test/reconcile-cache.test.js was failing. refreshFolder now fetches the cached row as resume state, and that test's fake db predates the method:

✖ reconcileCacheFromFilesystem indexes new and stale folders …
  getCachedSession → undefined

I added getCachedSession() { return null; } to the fake, modelling a session with nothing indexed yet. Fixed in the fake rather than guarding the call site — the real db provides the method, and a guard would mask genuine wiring errors. 27/27 pass now.

Remaining — your call

Long lines are quadratic. pending = Buffer.concat([pending, chunk]) recopies on every chunk, so a line longer than CHUNK_BYTES costs O(n²) in line length. This isn't hypothetical: my largest session file has 38 lines over 256 KB, longest 2.6 MB, which is ~10 recopies of a growing multi-MB buffer for a single line. Collecting chunks into an array and concatenating once at the newline would fix it.

Two test gaps. The truncation path (indexedBytes > stat.size → full re-read) is implemented but unasserted. More importantly, a multibyte character split across a chunk boundary — the subtlest correctness property in the PR — currently rests on inspection alone. A file with, say, an emoji straddling the 256 KB mark would pin it.

Smaller notes. readHead can split a multibyte character at maxBytes, leaving U+FFFD in the final partial line — harmless, since that line fails JSON.parse inside the existing try/catch, but worth a comment. It's also a quiet semantic change from .slice(0, 4000) (characters) to 4096 bytes. And a file under 4 KB hashes its whole contents, so crossing 4 KB changes headHash and forces one extra full re-read per session — correct, just non-obvious.

Nothing above blocks merging. Happy to take it as-is and file the long-line concat separately if you'd rather land the win now.