perf: stream wire.jsonl reads to reduce peak memory consumption

[bowenliang123]

Related Issue

No linked issue — the problem is explained below.

Problem

As the current situation, the wire.jsonl file is:

• Agent-level: each agent’s append-only event journal, used to restore its conversation, tool activity, configuration, tasks, and runtime state.
• Apend-only:New records are appended whenever a persistent agent event occurs, such as a prompt, model output, tool call/result, state change, or usage update.
• No compaction: Context compaction reduces the model’s active context but does not remove old wire records, so the file continues to grow.

So it's a key to improve the reading performance, as it's never compacted and repeatedly read.

Three wire.jsonl readers still load the whole log into memory before parsing a single record:

• SnapshotReader.readWireRecords (kap-server) — backs GET /sessions/{sid}/snapshot and the cold transcript rebuild; session wire logs routinely reach tens to hundreds of MB.
• readWireRecords (agent-core services/message/transcript.ts) — backs the daemon message-history view.
• scanSessionWire (agent-core session/export/wire-scan.ts) — backs debug-bundle export activity scans.

Each one did readFile + raw.split('\n'), so a read transiently held the raw file string plus a per-line string array plus the parsed records — roughly 3× the file size in peak heap on a server that holds many sessions.

What changed

Convert all three to streaming line-by-line reads, reusing existing components where semantics allow:

• kap-server SnapshotReader.readWireRecords: chunk-buffered line framing over createReadStream (the same pattern already used by FileSystemAgentRecordPersistence, v2 AppendLogStore, and sessionEventJournal). The exact error contract is preserved: a torn final line is dropped, corruption anywhere else throws wire.jsonl: corrupted line N in <path> with identical line numbering, and a missing file still rejects with ENOENT (the cold-snapshot route depends on it). A hand-rolled splitter is used instead of node:readline deliberately: readline cannot distinguish "last line without a trailing newline" (torn, must be dropped) from "last line with one" (corrupt, must throw).
• agent-core readWireRecords: delegates to the canonical FileSystemAgentRecordPersistence.read() async generator it always claimed to mirror — identical crash tolerance, no duplicated framing logic. One deliberate semantic alignment: a missing file now yields zero records (same as the persistence reader) instead of rejecting; every caller pre-stats the file and treats unreadable as "fall back to live view", so behavior is unchanged in practice.
• agent-core scanSessionWire: node:readline over createReadStream (this scan tolerates all malformed lines, so readline's final-line blindness is a non-issue). Missing/unreadable file still degrades to an empty scan.

No behavior change otherwise: same records returned, same per-line tolerance, same reducer inputs. Existing unit tests cover the contract (message-transcript.test.ts, snapshotReader.unit.test.ts — torn-line drop, mid-file corruption throw, blobref rehydration, snapshot cache); full agent-core (3996) and kap-server (737) suites pass.

Benchmark

Fixtures: synthetic wire.jsonl at 1 / 10 / 50 / 240 / 500 MB with realistic record shapes (context.append_message, context.append_loop_event, turn_begin; ~1,340 records per MB). Each implementation runs in a fresh process, 5 runs per cell, median reported; heap sampled in-process. macOS, Node 24.

readWireRecords (old readFile+split vs new streaming):

file size	wall old	wall new	peak Δheap old	peak Δheap new	peak Δ
1 MB	4 ms	7 ms	3.3 MB	2.3 MB	−30%
10 MB	25 ms	29 ms	32.7 MB	22.9 MB	−30%
50 MB	101 ms	120 ms	112.9 MB	100.4 MB	−11%
240 MB	491 ms	489 ms	779.2 MB	327.3 MB	−58%
500 MB	1070 ms	982 ms	1127 MB	641.5 MB	−43%

scanSessionWire (old vs new):

file size	wall old	wall new	peak Δheap old	peak Δheap new	peak Δ
1 MB	4 ms	8 ms	3.4 MB	1.2 MB	−65%
10 MB	24 ms	28 ms	27.9 MB	2.9 MB	−90%
50 MB	99 ms	106 ms	162.3 MB	4.1 MB	−97%
240 MB	448 ms	450 ms	555.8 MB	8.5 MB	−98%
500 MB	942 ms	928 ms	1045.4 MB	15.8 MB	−98%

Reading the tables:

• Small files (the common case): streaming adds ~3–4 ms of fixed stream-setup overhead — invisible in practice (snapshot reads are (size, mtime)-cached, so this cost is paid once per file change, not per request). Memory win is modest because the file is small anyway.
• Large files (the painful case): peak heap drops ~2× for readWireRecords and ~65× for scanSessionWire; the scan's peak stays ~flat (1.2 → 15.8 MB across 1 → 500 MB) instead of growing linearly with the log.
• Wall time is parity, then flips: at 500 MB the streaming readers are actually faster (982 vs 1070 ms; 928 vs 942 ms) — the old version's multi-hundred-MB single allocations start costing more than chunking does. The workload is JSON.parse-bound and both implementations parse every line identically.
• The remaining peak in readWireRecords (327 MB at 240 MB) is the returned records array itself — eliminating it requires reducer-level streaming (see below).

Follow-ups deliberately left out of this PR:

• Reducer-level streaming: reduceWireRecords already accepts Iterable; widening it and v2's reduceContextTranscript to AsyncIterable would let readWireTranscript / the snapshot cache fold records as they stream, removing the retained records array entirely (up to another ~2× on the numbers above). That touches public reducer signatures in two packages (apiSurface snapshot), so it deserves its own PR.
• The hand-rolled line-framing loop now exists in 4 places across 3 packages; not worth a cross-package util for ~15 lines, but worth revisiting together with the reducer streaming.

Checklist

• I have read the CONTRIBUTING document.
• I have linked a related issue, or explained the problem above.
• I have added tests that prove my feature works. (Behavior-preserving change; covered by existing unit tests, full suites pass.)
• Ran gen-changesets skill, or this PR needs no changeset.
• Ran gen-docs skill, or this PR needs no doc update. (Internal perf/memory change, no user-facing behavior or docs impact.)

[changeset-bot]

🦋 Changeset detected

Latest commit: 27f97ef

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@moonshot-ai/kimi-code	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR