fix: layer memory_category onto reflection-mapped rows (write-time, read-time, backfill)

[gorkem2020]

Problem

Reflection-mapped rows (the "User model deltas", "Agent model deltas", "Lessons & pitfalls", and "Decisions (durable)" sections written by the reflection writer) carry a mappedCategory/mappedKind in their metadata, but never a memory_category in the 6-category sense that the rest of the store uses. Readers fall back to deriving one from the row's legacy store category via reverseMapLegacyCategory, and that fallback sends mapped decision rows straight into events, an append-only category. Since durable operator/system decisions are not one-off occurrences, they end up permanently shielded from consolidation and dedup for the wrong reason: not because they are append-only in nature, but because of a mapping gap.

This PR closes that gap in three layers, following the write-time stamping pattern from #950 and building on the single-sourced category map from #935.

Layer 1: write-time stamping

buildReflectionMappedMetadata now stamps memory_category directly from the row's mappedKind, since the kind is known structurally at write time (each one comes from a fixed reflection section) rather than needing to be guessed from text:

• user-model, agent-model -> preferences
• lesson, decision -> cases

This is forward-only and has zero impact on existing stores; only new rows written after this lands get the stamp.

Layer 2: read-time fallback correction

For rows that never migrate (written before this lands, or from any other source relying on the fallback), reverseMapLegacyCategory's decision case now resolves through the same branch as fact instead of defaulting to events. This is a behavior change for any pre-existing row that hits this fallback with a legacy decision category: it will now read as cases (or profile for personal-identity-shaped text, same heuristic fact already used) instead of events.

Layer 3: opt-in backfill

memory-pro upgrade gains a --categories-only flag that runs a new, narrower one-shot pass (MemoryUpgrader.normalizeMappedRowCategories) to re-stamp memory_category on existing reflection-mapped rows using the same mapping as layer 1. It is separate from the general legacy-upgrade sweep (which deliberately excludes reflection rows), touches only that one field, and only when the value is missing or wrong. Idempotent: a row that's already correct is left alone, so a second run makes zero further changes. Supports --dry-run and --scope, matching the existing command's conventions.

Testing

• test/reverse-map-legacy-category.test.mjs
• test/reflection-mapped-category-stamping.test.mjs
• test/memory-upgrader-category-normalization.test.mjs

All three are registered in package.json's test chain and scripts/ci-test-manifest.mjs's storage-and-schema group. Full local suite, tsc --noEmit, and npm run build all pass clean.

Update (2026-07-18)

Rebased onto current master. This revision also single-sources the heading-to-taxonomy map: the metadata stamp and the stored row category both read REFLECTION_MAPPED_MEMORY_CATEGORY, agent self-observations now map to patterns instead of polluting user preferences, and mapped rows mint their smart taxonomy category as the row category rather than the legacy preference/fact/decision names. The upgrader's mapped-row normalization re-stamps existing rows through the same map. Verified live on our fleet.

[rwmjhb]

Reviewed head aa79cc6. The targeted tests, full suite, and repository CI pass, but the write path violates the primary category-column contract.

index.ts stores smart taxonomy values (preferences, patterns, cases) directly in MemoryEntry.category via a cast, while that column is defined in the legacy storage vocabulary (preference, fact, decision, etc.) and the repository already provides getStorageCategoryForMemoryCategory for this conversion. Direct consumers therefore see an incompatible value. A focused compaction check showed both cases and preferences sources being reconstructed with memory_category: patterns; the new preferences shape also derives a different default layer from an equivalent legacy-backed row. With compaction enabled, correctly stamped source rows can be deleted and replaced by a misclassified merged row.

Please keep entry.category in the legacy storage vocabulary and store the six-category value only in metadata.memory_category. Add production-path tests covering compaction and default-layer parity between newly written and backfilled rows.

Important follow-ups: paginate --categories-only beyond the fixed 10,000-row first page, keep admission and persistence on the same centralized mapping, restrict legacy-decision fallback to identifiable mapped rows, and merge category metadata against the latest locked row to avoid lost updates.

[gorkem2020]

Fixed in 69fab12. The write site now derives the stored column through the central smart-to-storage mapping (getStorageCategoryForMemoryCategory via a new getReflectionMappedStorageCategory helper), and the six-category value lives only in metadata.memory_category.

Read side, two hardenings for rows already written with six-category column values: reverseMapLegacyCategory reads a six-category value back as itself instead of defaulting to patterns, and the decision-to-cases redirect is now gated on the row's own mapped identity (metadata type), so a bare legacy decision row keeps the canonical decision-to-events mapping your sibling translators use. Layer derivation prefers a valid stamped memory_category, which restores default-layer parity between newly written and backfilled rows.

Follow-ups from your review, dispositions:

• Pagination: fixed. The categories-only backfill pages the whole store (pageSize option, default 1000) until a short page, no fixed first-page cap.
• Centralized mapping: fixed. Admission scoring now reads the same kind-to-category table the persisted stamp comes from; a consequence is that mapped decision rows are judged under cases, matching their stored register (they scored under events before, which disagreed with their own stamp).
• Legacy-decision fallback restriction: fixed as above.
• Merge against the latest row: the backfill now re-reads each row per chunk immediately before building its patch, so concurrent metadata writes between scan and write survive; the store-level replace of the metadata string is still last-writer-wins inside that millisecond window. A true atomic field merge (the mergeInsert pattern applyManualRecallMetadataBatch uses) is the durable shape and stays tracked as a follow-up.

Validation: new red-proofed cells for the compaction reconstruction (cases and preferences sources no longer collapse to patterns), default-layer parity, the storage-column vocabulary pins, backfill pagination, column repair, and fresh-read patch building (10 cells red pre-fix, all green post). Full suite 454/454, tsc clean, cli-smoke group green, dist rebuilt in the same commit.

[rwmjhb]

Thanks for addressing the primary storage-category blocker in 69fab12. The current head now has merge conflicts with the latest master, so I cannot reliably verify the fix against the code that will be merged. Please rebase onto the latest master, resolve the conflicts, rerun the focused category/backfill tests and full CI suite, and push the resolved head for re-review.

[gorkem2020]

Rebased onto current master (post #927/#943/#974 merges). Semantic reconciliation was confined to the reflection persist site: the branch now layers its mappedKind taxonomy and storage-category mapping on top of the merged #943 mirror callback (both behaviors preserved). Everything else was mechanical test-registration unions. Gates green on the new head: typecheck, full suite, manifest verifier, fresh dist.

[rwmjhb]

Re-reviewed rebased head 05457c0. The prior storage-contract blocker is fixed: mapped rows keep the legacy storage vocabulary in entry.category, while the six-category value remains in metadata.memory_category. The rebased persistence path preserves the merged #943 mirror callback, and the updated pagination, admission mapping, legacy-decision fallback, compaction/parity regressions, full suite, packaging checks, and repository CI all pass. I found no remaining HIGH/CRITICAL issue. Approving.

Follow-ups worth addressing: derive unstamped historical agent-model rows from mappedKind; make the category backfill an atomic field-level metadata merge; preserve valid member category stamps during compaction; honor or reject --limit explicitly in --categories-only; document that the command may also repair the storage column; and remove the whole-file index.ts line-ending churn/dead import.