fix: gate the auto-capture regex fallback behind smart extraction and admission

[gorkem2020]

Problem

When the smart extractor does not handle a turn, the auto-capture hook falls through to the legacy regex capture, which stores matching texts verbatim via bulkStore, with no distillation, no admission, and none of the extraction pipeline's safeguards. Two trigger paths:

1. cumulativeCount < extractMinMessages: every content-rich first message of a session is eligible for raw capture before the extractor ever gets to judge it.
2. The boundary-skip continuation after an extraction that persisted nothing.

Observed live: the first message of a fresh session (a roleplay opener) was stored verbatim as a confirmed fact row within seconds, bypassing the grounding and admission stack entirely. Sessions that open with a short greeting never hit this, which is why it stayed hidden; sessions that open with a rich message (how real users start conversations) hit it immediately.

Fix

1. Below-threshold turns with smart extraction enabled: skip the regex fallback and defer. The cursor is rolled back for history-carrying sessions so the deferred text re-enters the next turn's extraction input (proven by regression test); ingress-fed sessions keep their accumulator semantics, where a rollback would wedge the minMessages window.
2. When the fallback legitimately runs (smart extraction disabled, or the boundary-skip continuation): captures are routed through admission control when it is enabled, after the cheap dedup checks, with the audit persisted under provenance: "auto-capture-regex-fallback". Admission disabled keeps the legacy passthrough.

Tests

9 cases, 3 red-proven against the previous behavior: the below-threshold skip, deferred-text re-inclusion in the next extraction, gated boundary-skip continuation, plus pins for legacy behavior with smart extraction disabled.

Verification on a live gateway

The identical rich first message that previously produced a verbatim fact row now produces nothing; the follow-up turn's extraction input contained the deferred first-message content, produced correctly grounded candidates, and logged "skipping regex fallback" on the empty-outcome path.

[app3apps]

After #919 was merged, this PR now has merge conflicts with the latest main. Please rebase the branch onto the current main, resolve all conflicts, and push the updated head. We will review the updated revision once it is clean and mergeable.

[app3apps]

Reviewed head cd3c56f. The direction is valuable and the targeted/full suites pass, but the below-threshold change introduces two blocking data-loss regressions.

1. In ingress-fed sessions, pendingIngressTexts is copied and removed and the accumulator advances, but the rollback happens only when that snapshot is empty. The function then returns before regex fallback. A below-threshold ingress turn is therefore neither persisted nor available to a later extraction.
2. This also removes the deterministic behavior for explicit memory requests. Two existing cases containing 请记住我的饮品偏好是乌龙茶 were changed from asserting a regex capture to asserting that fallback is skipped. Persistence now requires another turn and a successful discretionary LLM extraction; a single-turn session loses the requested memory entirely.

Please retain below-threshold ingress text for the next extraction or preserve a safely admission-gated explicit-memory path, and add regressions covering real message_received ingress and a single-turn explicit remember request.

Also, admission evaluation currently fails open without storing a durable failed-open marker. Since this PR exists to close an admission bypass, please either fail closed for raw fallback text or persist provenance that makes bypassed rows auditable.

[app3apps]

Re-reviewed rebased head f9bf6ed. The rebase is clean and CI is green, but the production-path data-loss blocker remains.

In ingress-fed sessions, pendingIngressTexts is consumed and removed, the cursor advances, and the below-threshold branch returns before the fallback path. That permanently drops the ingress text, including explicit remember-style requests.

Please preserve or requeue deferred ingress text until it is actually extracted, and add production-path coverage for message_received plus a single-turn remember request. A history-carrying session that ends below threshold also needs an end-of-session case so its deferred text cannot disappear. Fail-open admission should leave durable provenance on stored rows.

Requesting changes until the ingress text is no longer lossy.

[app3apps]

Re-reviewed head ce16a61. CI and the full suite pass, but the new expectations still codify deferral without proving eventual persistence.

For ingress-fed sessions, pendingIngressTexts is consumed and deleted, the seen cursor advances, and the below-threshold branch returns before fallback. A single-turn explicit remember request can therefore become unrecoverable. A history-backed session that ends before extractMinMessages has the same no-flush outcome.

Please preserve or requeue deferred text until extraction succeeds, or persist explicit capture through a safely gated path. Add production-path regressions using real message_received ingress, a single-turn remember request, and a session ending below threshold, proving the content is eventually stored exactly once. Admission failures that intentionally fail open should also leave durable provenance on the stored row.

Requesting changes.

[gorkem2020]

Fixed two findings in 63eec3f, and want to be direct about how the other three relate rather than claim more than I've actually closed.

Fixed:
2. For ingress-fed sessions, a below-threshold turn consumed and deleted the pending ingress queue, counted it toward the threshold, then discarded the actual text. The counter advanced correctly but the content of every deferred turn except the most recent was gone by the time a later turn crossed the threshold. Now re-queues the consumed texts back into the pending map on defer, mirroring what the existing cursor rollback already does for history-carrying sessions.
5. The regex-fallback gate's fail-open path now gets a durable audit record on the row itself (failedOpen: true), not just an ephemeral log line, matching the same fix just applied to the sibling mapped-row admission gate on another PR.

Not separately fixed:

1. The test-coverage ask is substantially closed by the new ingress-flow test added alongside the fix above, which proves a second deferred turn's text survives into the extraction that finally crosses the threshold.
2. The single-turn explicit-remember scenario is the same underlying gap as finding 2 for ingress-fed sessions (now fixed) and was already handled for history-carrying sessions by the existing rollback. The one case that is genuinely unrecoverable either way is finding 4 below.
3. A session that ends before crossing the threshold has no flush path, and that's real. I looked at adding a session_end handler to force a final extraction, but the risk is concrete: if the last agent_end call already crossed the threshold and reset the counter, a naive session_end flush reading the same accumulated-text buffer could re-extract content that was just successfully persisted, which would be a worse bug (duplicate memories) than the current silent gap. Getting that distinction right needs its own careful pass, not a same-session addition alongside everything else here.

[app3apps]

Thanks for the update in 63eec3f. I reran the orchestrator against this head, but GitHub currently reports mergeable=CONFLICTING / merge_state_status=DIRTY, so the review stopped at the conflict gate after R0/R1.

Please rebase onto the latest master, resolve all conflicts, and push the clean result. The ingress requeue and failed-open provenance changes are recorded; we will verify them against the actual merge result after the rebase.

[gorkem2020]

Rebased onto current master as requested, the branch is mergeable again. New head: b0f05a3. Conflicts were limited to the test registration files (package.json test chain and scripts/ci-test-manifest.mjs), resolved by keeping both sides' entries; dist is rebuilt from the resolved sources and the PR's test suite plus typecheck pass on the new base.

[app3apps]

Re-reviewed the current head b0f05a3. The ingress requeue and failed-open provenance fixes are useful, and the full suite passes, but two behavior gaps still block this change:

1. The below-threshold branch returns before regex fallback and only preserves data for a later turn. If the session ends before extractMinMessages, an explicit remember-style request that the legacy fallback previously persisted is silently lost. Please add an end/session-boundary flush or another exactly-once path for intentional capture, with a regression for a short session that has no later turn.
2. The PR body says the legitimate fallback is routed through admission when smartExtraction=false, but the controller is obtained from smartExtractor?.getAdmissionController(). With smart extraction disabled that is always null, and gateRegexFallbackCapture admits unconditionally. Please construct/use admission independently of SmartExtractor for this configuration, or otherwise make the advertised safety boundary real and test the production configuration.

There is also an accounting edge to cover while fixing the first item: ingress cumulativeCount advances before deferral, while requeued texts are counted again and the queue is capped at six, so repeated deferrals can cross the threshold early while silently dropping older input. Add an exactly-once/count-and-cap regression around that flow.

[app3apps]

Since #924 merged, GitHub now reports this branch as conflicting with the latest master. Please rebase onto or merge the current master, resolve the conflicts, and push the updated head. The prior review blockers remain recorded and will be re-reviewed on the clean revision.

[gorkem2020]

Rebased onto the current master, dist rebuilt on the rebased tree, all checks green on the branch.

[app3apps]

Re-reviewed head 5d3dbd3 after the rebase. The build, focused suites, and GitHub CI pass, but the rebase leaves the two previously requested correctness fixes unchanged:

1. In the below-threshold agent_end path, the consumed text is rolled back or requeued and the hook returns. There is still no session-boundary flush. If a session ends before extractMinMessages (including a one-turn explicit "remember this" request), there is no later turn to consume the deferred content, so a capture that previously persisted is now silently lost. Please add an exactly-once safe flush and an end-to-end short-session regression that inspects storage.

2. When smartExtraction=false, no SmartExtractor or AdmissionController is created. The caller passes null to gateRegexFallbackCapture, whose null-controller branch unconditionally admits the raw regex capture. Thus admission remains completely bypassed in the configuration where regex fallback is the primary capture path, despite the PR's stated admission-gating behavior. Please make the controller available independently of smart extraction (or otherwise enforce the gate) and add a production-configuration rejection test.

These are behavioral blockers, not test-infrastructure issues.