Skip to content

fix: split the memory pipeline's internal LLM prompts into system and user messages#938

Draft
gorkem2020 wants to merge 12 commits into
CortexReach:masterfrom
gorkem2020:fix/prompt-architecture
Draft

fix: split the memory pipeline's internal LLM prompts into system and user messages#938
gorkem2020 wants to merge 12 commits into
CortexReach:masterfrom
gorkem2020:fix/prompt-architecture

Conversation

@gorkem2020

Copy link
Copy Markdown
Contributor

The memory pipeline makes four internal LLM calls: extraction, admission scoring, dedup decision, and merge. All four packed instructions, criteria, output-format contract, and the per-call data into a single user-role message, with no system prompt beyond a generic one-liner shared by every call ("You are a memory extraction assistant..."). This meant the admission judge, the dedup decider, and the merge writer all introduced themselves with extraction-assistant wording, and every call mixed static instructions in with the untrusted conversation excerpt in the same message.

What changed

Each of the four prompt builders (buildExtractionPrompt, buildDedupPrompt, buildMergePrompt in src/extraction-prompts.ts, and buildUtilityPrompt in src/admission-control.ts) now returns {system, user}: static instructions, criteria, identity, and the output-format contract live in system; the per-call conversation excerpt, candidate rows, or neighbor rows live in user.

Each stage gets an honest, distinct identity instead of sharing generic wording:

  • extraction: "You are an extraction agent."
  • admission scoring: "You are an admission judge."
  • dedup: "You are a dedup decider."
  • merge: "You are a merge writer."

LlmClient.completeJson() gains an optional third argument, systemPrompt. The api-key path threads it into the messages array's system role. The OAuth path threads it into the Responses API's existing instructions field; that field already carried system-prompt semantics (it previously held the same generic string), so no degrade-to-prepend fallback was needed; the request shape stays exactly as it was, just with a per-call value instead of one fixed string. Both paths default to the historical generic text when systemPrompt is omitted, so the one caller outside this pipeline (memory-upgrader.ts's one-off migration prompt) is unaffected.

The merge prompt template had a live-observed formatting bug: broken bold markers rendered as ** Category **:, progressively indented list items, and a tokenization artifact ("up - to - date" from a space-hyphen-space pattern in the source). Rebuilt the template cleanly; added a snapshot-style test asserting the assembled prompt contains none of those artifacts.

Reflection-sourced admission gating previously reused the extraction path's "Conversation excerpt:" framing even when the excerpt was actually a reflection document. buildUtilityPrompt and AdmissionController.evaluate() now take an optional sourceKind: "conversation" | "reflection" (default "conversation"), and the admission judge sees "Source document (agent reflection):" for reflection-sourced rows instead. The admission judge only ever scores the excerpt as given; it never rewrites candidates.

No scoring weights, thresholds, or the decision protocol changed. Same decisions in and out; better-formed prompts.

Test plan

  • New test file covering all four builders' system/user split, per-stage identity strings, the merge template's absence of broken-bold/indentation/tokenization artifacts, and the reflection source-kind framing.
  • LlmClient tests confirming the api-key path's system message and the OAuth path's instructions field both honor an explicit systemPrompt and fall back to the historical default when omitted.
  • Updated three existing test files whose mock LLM servers matched on the old prompt-opener phrases (now identity phrases, since those phrases moved to the system message).
  • Full local test chain green; tsc build green.

@gorkem2020
gorkem2020 force-pushed the fix/prompt-architecture branch 3 times, most recently from 2b42580 to e13f4b0 Compare July 15, 2026 18:08
gorkem2020 added a commit to gorkem2020/memory-lancedb-pro that referenced this pull request Jul 15, 2026
Checkpoint recut of deploy/fleet-2026-07-15b (21 PRs) surfaced several
merge-time regressions and stale test fixtures once the full test/tsc/build
gate ran end to end:

- index.ts: a plain (non-conflicted) auto-merge reintroduced CRLF into the
  modelRun:true block inside generateReflectionTextUnbounded; restored to LF.
- index.ts: restored PR CortexReach#934's "regex fallback skipped" debug log + return
  in the below-threshold defer branch, dropped when PR CortexReach#951's merge took the
  persistence-based rollback wholesale instead of unioning both changes.
- index.ts: removed a duplicate minMessages declaration left over from
  reconciling PR CortexReach#939's assistant-context accumulation block with PR CortexReach#951's
  own minMessages usage earlier in the same function.
- src/admission-control.ts: hoisted the constructed-durable short-circuit
  from evaluateWithUtility() into evaluate() itself, so a constructed
  durable candidate is rejected before scoreUtility() spends an LLM call,
  not just before its score is blended in.
- src/admission-control.ts: restored the Grounding/Conversation register
  fields in buildUtilityPrompt's user template (dropped during the PR CortexReach#938
  split-prompt merge), since PR CortexReach#927's own admission tests depend on the
  judge seeing them inline.
- Updated several PR CortexReach#927/CortexReach#938/CortexReach#939 test assertions that predate the
  split-prompt {system, user} return shape to destructure the right half
  instead of matching the whole object, and updated stale
  "## Recent Conversation" / "Evaluate whether this candidate" mock-routing
  strings to the current prompt text.
- test/assistant-context-capture-counting-symmetry.test.mjs: gave each
  captureAssistant/payloadShape scenario its own directory so the new
  watermark sidecar file (keyed by dirname(dbPath) + sessionKey) doesn't
  bleed state between scenarios that previously only shared in-memory state;
  updated the delta-only sanity check to reflect the documented rollback
  characteristic where per-turn deltas below minMessages never accumulate.
- test/smart-extractor-branches.mjs: updated a fixture comment/payload that
  had already predicted and documented this exact combination, switching to
  full accumulated history per turn as prescribed.

Full suite (minus the environment-only Ollama port-conflict test), tsc, and
build all green after these fixes.
@gorkem2020
gorkem2020 force-pushed the fix/prompt-architecture branch 2 times, most recently from 71765a9 to 45d012f Compare July 17, 2026 21:50
All four internal calls (extract-candidates, admission-utility,
dedup-decision, merge-memory) packed instructions, criteria, identity, and
the output-format contract into a single user-role message with no system
prompt. Restructure each builder to return {system, user}: static
instructions/criteria/format go in system, the per-call conversation
excerpt / candidate rows / neighbor rows go in user.

Give each stage an honest, distinct identity (extraction agent, dedup
decider, merge writer, admission judge) instead of every call introducing
itself with the same generic extraction-assistant wording.

Rebuild the merge prompt template cleanly: the old version rendered
broken bold markers ("** Category **:"), progressively indented list
items, and split-hyphen tokenization artifacts ("up - to - date").

Thread an optional sourceKind through the admission prompt builder so a
reflection-sourced excerpt is honestly framed as "Source document (agent
reflection)" rather than "Conversation excerpt" (default unchanged). The
actual reflection call site lives on a not-yet-merged branch and will
pass sourceKind: "reflection" when that branch is integrated.

LlmClient.completeJson() gains an optional systemPrompt third argument.
The api-key path threads it into the messages array's system role; the
OAuth path threads it into the Responses API's existing `instructions`
field (which already carried system-prompt semantics, so no prepend
fallback was needed). Both default to the historical generic system text
when omitted, so memory-upgrader.ts's unrelated call site is unaffected.

Update three existing test files whose mock LLM servers routed responses
by matching prompt-opener phrases in the user message; those phrases now
live in the system message, so the mocks combine both messages before
matching. No scoring weights, thresholds, or decision protocol changed.
…uations

- buildDedupPrompt and buildMergePrompt: separate each Abstract/Overview/
  Content field with a blank line so markdown embedded in Overview/Content
  doesn't visually collapse into the label line above it.
- admission-control's buildUtilityPrompt: indent continuation lines of a
  multi-line Overview/Content so they stay nested under their bullet
  instead of landing flush-left mid-list.
- smart-extractor's dedup candidate listing: extract
  formatExistingMemoryForDedupPrompt so the same continuation-line
  indentation applies to the numbered existing-memories list, and the
  formatting is directly unit-testable.
…ission few-shot example

buildUtilityPrompt's Overview/Content bullets were glued to Category/
Abstract with a single newline; give them their own blank-line-separated
block like the dedup/merge prompts already do. The system prompt had no
few-shot example at all, so add one wrapped in explicit "--- EXAMPLE
(not part of the live data) ---" / "--- END EXAMPLE ---" markers and a
clearly labeled "Example response:" line, matching the fencing convention
already established for the batch-utility prompt on a sibling branch.

No batch variant of buildUtilityPrompt exists on this branch (or on
master) to formatting-match against -- it lives on the separate,
unmerged feat/admission-batch-utility branch, out of scope here.
buildReflectionPrompt returned one flat string with the distiller's
identity, heading contract, hard rules, section rules, governance format,
and output template all glued to the per-call tool-error-signal and
conversation payload. Split it into {system, user} like the other four
LLM call sites: system carries every static instruction (now opening
with an explicit "You are a memory reflection distiller." identity
line, matching the extraction/dedup/merge/admission stage-identity
convention), user carries only the per-call tool error signals and the
fenced conversation INPUT block.

generateReflectionTextUnbounded rejoins system + "\n\n" + user before
handing the result to the embedded-agent runner and the CLI fallback,
since neither transport (runEmbeddedPiAgent's params bag, the CLI's
--message arg) has a distinct system-prompt field to split into -- same
mechanical-merge posture already used on the sibling admission-batch-
utility branch for its own not-yet-split call site. A test asserts the
rejoined text is byte-identical to the prior single-string prompt, so
this is a pure prompt-architecture/formatting change with no behavior
delta: same headings, same rules, same output-format contract.

index.ts carries this repo's known mixed CRLF/LF encoding; both the
function body and its call site were rewritten with a latin1 Buffer
splice (never Edit) to avoid normalizing the file, per this fork's
CLAUDE.md. Diff is 10 lines.
Durable/recurring state and habit changes (e.g. "switched my commute
to the M4", "Spanish lesson before breakfast") were getting
classified as "events" by the extraction LLM and then hugging the
admission threshold instead of persisting as the user's new ongoing
state. Add a compact rubric bullet to the extraction prompt's Common
Confusion section: recurring/durable state and habit changes route
to preferences or patterns; events stays reserved for genuinely
one-off occurrences.
Wire test/extraction-category-rubric.test.mjs into package.json's
local test chain and scripts/ci-test-manifest.mjs's
llm-clients-and-auth group, alongside its sibling
prompt-architecture.test.mjs.
@gorkem2020
gorkem2020 force-pushed the fix/prompt-architecture branch from 45d012f to 035d6a2 Compare July 18, 2026 15:47
…tities, markdown payload shape, raw-JSON contracts

Fleet-proven prompt-architecture foundation, ported from the deployed
build:
- src/prompt-blocks.ts single-sources the shared static prompt content:
  agent identities (every prompt opens 'You are a memory ...'), the
  six-category taxonomy line, the scoring rubric, bare JSON shape
  rendering (jsonShape - examples are shown exactly as the model must
  emit them, unfenced; fenced examples taught fence-mimicking models to
  fence their output), and the markdown candidate/memory formatters.
- Candidate and existing-memory payloads render as markdown blocks
  ('## Candidates' / '### 1. <category>') through one shared formatter.
- Every output contract reads 'Return JSON only (the raw object, no
  markdown code fences)' and shows the shape bare.
- The admission judge is transcript-free: it scores the candidate's own
  content plus store-derived features, never a conversation excerpt.
- The dedup decider carries the anti-category-wall doctrine (category
  labels never decide a verdict by themselves) and the extraction
  transcript rides a fenced block like the distiller's INPUT.
- Distiller opener names the domain and forbids fenced output.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant