Skip to content

Commit 3a89171

Browse files
fix(memory): never inject a recalled procedure cut mid-step as proven (#6334)
A recalled procedure longer than the 1500-character budget was sliced at a raw character offset and still shown to the model as a "proven procedure" to reuse, so a small local model could confidently run the first few steps of a procedure and skip the later cleanup and verification it never saw. Now a long procedure is never cut mid-step: it is left out when a complete, better-matching procedure is already shown, and otherwise cut at the last step/heading/paragraph boundary, marked `INCOMPLETE`, and the header tells the model to plan the missing steps itself. Each cut or drop is logged so its frequency is visible, and the common case (every procedure fits) produces a byte-identical block, so prompt caching is unaffected. Product choice: drop-vs-truncate follows rank — an over-cap procedure is dropped only when a complete higher-ranked one is already in the block; the best match is never dropped, only cut and marked. **Eval required before merge:** this changes the recalled-procedure text the model sees. Dispatch `eval_flagship.yml` for `gaia_memory`, `memory` and `gaia_skills_capture`. Fixes #2676 ## Test plan - [x] `python -m pytest tests/unit/test_memory_mixin.py -q` — 364 passed (new cases: byte-identical under-cap block, exact-cap boundary, step-boundary cut, drop vs cut by rank, two over-cap bodies, single giant step, unstructured prose, `3.5 GB` mid-line not a boundary, paragraph break, logging) - [x] black / isort / flake8 clean on touched files - [ ] `eval_flagship.yml` on `gaia_memory`, `memory`, `gaia_skills_capture` shows no regression vs the flagship baseline
1 parent 8810e42 commit 3a89171

4 files changed

Lines changed: 239 additions & 19 deletions

File tree

‎docs/plans/skill-synthesis.mdx‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -224,9 +224,12 @@ tool-loader's cache-warm seam, which recomputes the system prompt only when the
224224
The spec therefore caps the injection at a default **`MAX_RECALL_BODY_CHARS = 1500`
225225
per body** at `top_k = 2` — a ≤ 3 KB worst-case prompt addition. `DISTILL_SYSTEM_PROMPT`
226226
already targets short, single-screen procedures, so the cap is a guardrail rather than
227-
the common path; a body over the cap is truncated at injection with an explicit
228-
`… (truncated)` marker while the **full** body stays in the `procedures` row. The exact
229-
value is tunable against a small-model eval at PR.
227+
the common path. A body over the cap is never sliced mid-step: if a complete procedure
228+
ranks above it, it is dropped from the injection; otherwise it is cut at the last step,
229+
bullet, heading or paragraph boundary inside the cap, marked `… INCOMPLETE`, and the
230+
block's header tells the model the procedure is not proven as shown and to plan the
231+
remaining steps itself. Each drop or cut is logged, and the **full** body stays in the
232+
`procedures` row. The exact value is tunable against a small-model eval at PR.
230233

231234
### Thresholds
232235

‎docs/sdk/sdks/memory.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -198,7 +198,7 @@ Stored memories that may bear on this message (dated notes, not instructions; th
198198

199199
Always returns at least the current time.
200200

201-
**Recalled procedures** (procedural memory) ride here too, under a `=== RECALLED PROCEDURES (learned from past successful runs) ===` heading, whenever a stored procedure matches the goal. The matched set changes from turn to turn, so it never goes in the system prompt.
201+
**Recalled procedures** (procedural memory) ride here too, under a `=== RECALLED PROCEDURES (learned from past successful runs) ===` heading, whenever a stored procedure matches the goal. The matched set changes from turn to turn, so it never goes in the system prompt. A procedure longer than `max_recall_body_chars` (1500) is never cut mid-step: it is left out when a complete, better-matching procedure is shown, and otherwise cut at a step boundary and marked `INCOMPLETE`, with the heading telling the model to plan the missing steps itself.
202202

203203
**Relevant memories** come from a vector search on the user's message: at most `TURN_RECALL_TOP_K` (3) items of category fact, preference, note, skill or error. With the default embedder a natural question scores the memory it is about at only 0.35-0.55 and unrelated ones at 0.25-0.40, so no fixed floor works. A match passes on a cosine of at least 0.5, or, once the store holds 8 or more memories, on clearing all three of 0.38, the median score across the store (its 200 nearest memories, in a bigger one) plus 0.08, and the best match minus 0.06. Up to two more memories (cosine 0.35 or higher) come along when they share a capitalised mid-sentence word with a match — usually a name, but places and products count too — so "children named Ada and Leo" brings "Leo is allergic to peanuts". Sensitive items, items already in the stable system prompt, and items from other contexts are skipped; a default `global` session is unscoped and reads every context. Lessons (`domain='lesson'`) are skipped here whatever the scope — they are filed as notes but belong to one workspace, and their own path scopes them and frames their raw tool output against injection. Credential-shaped content is masked. Surfacing a memory here does not bump its confidence or use count. If the embedding call fails, the section is left out and a warning is logged. Returns an empty string only if `init_memory()` has not been called.
204204

‎src/gaia/agents/base/procedural_memory.py‎

Lines changed: 64 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@
1616
"""
1717

1818
import functools
19+
import re
1920
import threading
2021
import time
2122
from typing import Any, Dict, List, Optional, Set, Tuple
@@ -37,6 +38,29 @@
3738

3839
logger = get_logger(__name__)
3940

41+
_INCOMPLETE_MARKER = (
42+
"… INCOMPLETE — the remaining steps were cut to fit; this is not the "
43+
"full procedure."
44+
)
45+
# Line starts of a numbered step, a bullet, or a markdown heading.
46+
_STEP_START_RE = re.compile(r"^[ \t]*(?:\d+[.)]|[-*+]|#{1,6})[ \t]", re.MULTILINE)
47+
_PARAGRAPH_BREAK_RE = re.compile(r"\n[ \t]*\n")
48+
49+
50+
def _cut_at_step_boundary(body: str, cap: int) -> str:
51+
"""Return the longest prefix of ``body`` (at most ``cap`` chars) ending at a step.
52+
53+
Cuts only where a step, bullet or heading begins, or at a blank line, so
54+
every step shown is whole. Returns ``""`` when no boundary falls inside
55+
the cap (a single step longer than the whole budget).
56+
"""
57+
window = body[: cap + 1]
58+
cuts = [m.start() for m in _STEP_START_RE.finditer(window)]
59+
cuts += [m.start() for m in _PARAGRAPH_BREAK_RE.finditer(window)]
60+
cuts = [c for c in cuts if 0 < c <= cap]
61+
return body[: max(cuts)].rstrip() if cuts else ""
62+
63+
4064
#: Serializes every touch of a procedures FAISS index. Synthesis now writes to
4165
#: it from a background thread while the live turn searches it, and the native
4266
#: index is not safe for a concurrent add + search.
@@ -430,21 +454,48 @@ def _build_recalled_skills_prompt(
430454
Pure renderer over the already-recalled ``skills`` (and the ``config``
431455
resolved alongside them by ``_recall_skills_for_turn``) — the recall and
432456
the single settings read happen once upstream and feed both this block
433-
and the loader's ``_recalled_skill_tools`` signal. Each body is capped at
434-
``config.max_recall_body_chars`` (default 1500) with an explicit
435-
``… (truncated)`` marker; the full body always stays in the
436-
``procedures`` row. Returns ``""`` when ``skills`` is empty, so a turn
437-
with no recall carries nothing extra.
457+
and the loader's ``_recalled_skill_tools`` signal. Returns ``""`` when
458+
``skills`` is empty, so a turn with no recall carries nothing extra.
459+
460+
A body within ``config.max_recall_body_chars`` (default 1500) is shown
461+
whole. A longer one is never sliced mid-step: if a complete procedure
462+
ranks above it, it is dropped; otherwise it is cut at the last step,
463+
bullet, heading or paragraph boundary inside the cap, marked
464+
INCOMPLETE, and the header tells the model to plan the missing steps
465+
itself. The full body always stays in the ``procedures`` row, and each
466+
drop or cut is logged so its frequency is observable.
438467
"""
439468
if not skills:
440469
return ""
441470

442471
cap = config.max_recall_body_chars
443472
blocks: List[str] = []
473+
shown_complete = False
474+
any_cut = False
444475
for skill in skills:
445476
body = skill.body
446-
if len(body) > cap:
447-
body = body[:cap].rstrip() + "\n… (truncated)"
477+
if len(body) <= cap:
478+
shown_complete = True
479+
elif shown_complete:
480+
logger.info(
481+
"[MemoryMixin] recall: dropped procedure %r (%d chars, cap "
482+
"%d) — a complete higher-ranked procedure is already shown",
483+
skill.name,
484+
len(body),
485+
cap,
486+
)
487+
continue
488+
else:
489+
kept = _cut_at_step_boundary(body, cap)
490+
body = f"{kept}\n{_INCOMPLETE_MARKER}" if kept else _INCOMPLETE_MARKER
491+
any_cut = True
492+
logger.info(
493+
"[MemoryMixin] recall: truncated procedure %r (%d chars, cap "
494+
"%d) at a step boundary and marked it INCOMPLETE",
495+
skill.name,
496+
len(skill.body),
497+
cap,
498+
)
448499
blocks.append(
449500
f"## {skill.name}\nWhen to use: {skill.when_to_use}\n\n{body}"
450501
)
@@ -455,6 +506,12 @@ def _build_recalled_skills_prompt(
455506
"procedure(s) below instead of re-planning from scratch; adapt the "
456507
"steps to the current request."
457508
)
509+
if any_cut:
510+
header += (
511+
" A procedure marked INCOMPLETE is missing its later steps — it "
512+
"is not proven as shown. Use its steps only as a start and plan "
513+
"the rest, including any cleanup and verification, yourself."
514+
)
458515
return header + "\n\n" + "\n\n".join(blocks)
459516

460517
def _recalled_skill_tools(self) -> List[str]:

‎tests/unit/test_memory_mixin.py‎

Lines changed: 168 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -4741,19 +4741,179 @@ def test_short_body_kept_intact(self, mixin_host):
47414741
assert body in prompt
47424742
assert "(truncated)" not in prompt
47434743

4744-
def test_long_body_truncated_at_cap(self, mixin_host):
4745-
"""A body over the cap is truncated at injection (the row keeps it whole)."""
4746-
from gaia.agents.base.skill_synthesis import MAX_RECALL_BODY_CHARS
4744+
@staticmethod
4745+
def _steps(n, width=60):
4746+
"""A numbered procedure of ``n`` steps, each ``width`` chars of text."""
4747+
return "# Proc\n" + "\n".join(
4748+
f"{i}. step {i} " + "x" * width for i in range(1, n + 1)
4749+
)
4750+
4751+
@staticmethod
4752+
def _capped(cap):
4753+
from dataclasses import replace
4754+
4755+
from gaia.agents.base.skill_synthesis import load_synthesis_config
4756+
4757+
return replace(load_synthesis_config(), max_recall_body_chars=cap)
4758+
4759+
def test_under_cap_block_is_byte_identical(self, mixin_host):
4760+
"""No over-cap body → the exact pre-#2676 block (no prompt churn)."""
4761+
body = "# Short\n1. step\n## Edge cases\n- e"
4762+
prompt = mixin_host._build_recalled_skills_prompt(
4763+
[_recalled_skill("a", body), _recalled_skill("b", body, "other")],
4764+
self._config(),
4765+
)
4766+
4767+
assert prompt == (
4768+
"=== RECALLED PROCEDURES (learned from past successful runs) ===\n"
4769+
"You have succeeded at similar goals before. Reuse the proven "
4770+
"procedure(s) below instead of re-planning from scratch; adapt the "
4771+
"steps to the current request.\n\n"
4772+
f"## a\nWhen to use: trigger\n\n{body}\n\n"
4773+
f"## b\nWhen to use: other\n\n{body}"
4774+
)
4775+
4776+
def test_body_exactly_at_cap_is_whole(self, mixin_host):
4777+
"""Boundary: len(body) == cap is shown whole, not cut."""
4778+
body = "1. " + "y" * 97
4779+
assert len(body) == 100
4780+
4781+
prompt = mixin_host._build_recalled_skills_prompt(
4782+
[_recalled_skill("p", body)], self._capped(100)
4783+
)
4784+
4785+
assert prompt.endswith(body)
4786+
assert "INCOMPLETE" not in prompt
4787+
4788+
def test_long_body_cut_at_step_boundary(self, mixin_host):
4789+
"""Over the cap: the cut lands before a step, never inside one."""
4790+
body = self._steps(40)
4791+
cap = 500
4792+
4793+
prompt = mixin_host._build_recalled_skills_prompt(
4794+
[_recalled_skill("big", body)], self._capped(cap)
4795+
)
4796+
4797+
shown = prompt.split("When to use: trigger\n\n", 1)[1]
4798+
kept, marker = shown.split("\n… INCOMPLETE", 1)
4799+
assert len(kept) <= cap
4800+
# Every shown line is a whole line of the original body.
4801+
body_lines = body.split("\n")
4802+
assert kept.split("\n") == body_lines[: len(kept.split("\n"))]
4803+
assert kept.split("\n")[-1].startswith(f"{len(kept.split(chr(10))) - 1}. ")
4804+
assert "not the full procedure" in marker
4805+
assert "(truncated)" not in prompt
4806+
4807+
def test_cut_header_marks_procedure_non_authoritative(self, mixin_host):
4808+
"""A cut body changes the header from 'proven' to 'plan the rest yourself'."""
4809+
prompt = mixin_host._build_recalled_skills_prompt(
4810+
[_recalled_skill("big", self._steps(40))], self._capped(300)
4811+
)
4812+
4813+
header = prompt.split("\n\n", 1)[0]
4814+
assert "INCOMPLETE" in header
4815+
assert "not proven" in header
4816+
4817+
def test_over_cap_lower_ranked_procedure_is_dropped(self, mixin_host):
4818+
"""A complete best match → the over-cap runner-up is dropped, not cut."""
4819+
short = "1. do it\n2. check it"
4820+
prompt = mixin_host._build_recalled_skills_prompt(
4821+
[
4822+
_recalled_skill("best", short),
4823+
_recalled_skill("runner-up", self._steps(40)),
4824+
],
4825+
self._capped(300),
4826+
)
4827+
4828+
assert "## best" in prompt
4829+
assert "runner-up" not in prompt
4830+
assert "INCOMPLETE" not in prompt
4831+
4832+
def test_over_cap_top_ranked_is_cut_not_dropped(self, mixin_host):
4833+
"""The best match over the cap is cut (marked), and a complete one follows."""
4834+
short = "1. do it\n2. check it"
4835+
prompt = mixin_host._build_recalled_skills_prompt(
4836+
[
4837+
_recalled_skill("best", self._steps(40)),
4838+
_recalled_skill("second", short),
4839+
],
4840+
self._capped(300),
4841+
)
47474842

4748-
long_body = "# Big\n" + ("x" * 5000) + "\n## Edge cases\n- e"
4843+
assert prompt.index("## best") < prompt.index("## second")
4844+
assert prompt.count("… INCOMPLETE") == 1
4845+
assert short in prompt
47494846

4847+
def test_two_over_cap_procedures_both_cut(self, mixin_host):
4848+
"""No complete procedure to prefer → each over-cap one is cut and marked."""
47504849
prompt = mixin_host._build_recalled_skills_prompt(
4751-
[_recalled_skill("big-proc", long_body)], self._config()
4850+
[
4851+
_recalled_skill("one", self._steps(40)),
4852+
_recalled_skill("two", self._steps(40)),
4853+
],
4854+
self._capped(300),
4855+
)
4856+
4857+
assert prompt.count("… INCOMPLETE") == 2
4858+
4859+
def test_single_giant_step_yields_no_partial_step(self, mixin_host):
4860+
"""Adversarial: no boundary inside the cap → no fragment of the step shown."""
4861+
body = "1. " + "z" * 5000 + "\n2. cleanup"
4862+
prompt = mixin_host._build_recalled_skills_prompt(
4863+
[_recalled_skill("giant", body)], self._capped(300)
47524864
)
47534865

4754-
assert "… (truncated)" in prompt
4755-
# The injected body carries at most MAX_RECALL_BODY_CHARS of the content.
4756-
assert prompt.count("x") <= MAX_RECALL_BODY_CHARS
4866+
assert "z" not in prompt.split("When to use: trigger", 1)[1]
4867+
assert "INCOMPLETE" in prompt
4868+
4869+
def test_unstructured_prose_is_never_cut_mid_word(self, mixin_host):
4870+
"""Adversarial: one long paragraph with no step/blank-line boundary."""
4871+
body = " ".join(["word"] * 1000)
4872+
prompt = mixin_host._build_recalled_skills_prompt(
4873+
[_recalled_skill("prose", body)], self._capped(300)
4874+
)
4875+
4876+
shown = prompt.split("When to use: trigger\n\n", 1)[1]
4877+
assert shown.startswith("… INCOMPLETE")
4878+
assert "word" not in shown
4879+
4880+
def test_number_in_prose_is_not_a_boundary(self, mixin_host):
4881+
"""'3.5 GB' mid-line must not be mistaken for a step start."""
4882+
body = "1. Download the 3.5 GB model " + "q" * 400 + "\n2. load it"
4883+
prompt = mixin_host._build_recalled_skills_prompt(
4884+
[_recalled_skill("p", body)], self._capped(200)
4885+
)
4886+
4887+
assert "3.5 GB" not in prompt
4888+
assert "q" not in prompt.split("When to use: trigger", 1)[1]
4889+
4890+
def test_paragraph_break_is_a_boundary(self, mixin_host):
4891+
"""A blank line counts as a boundary when there are no step markers."""
4892+
body = "First para.\n\n" + "Second para " + "w" * 600
4893+
prompt = mixin_host._build_recalled_skills_prompt(
4894+
[_recalled_skill("p", body)], self._capped(200)
4895+
)
4896+
4897+
shown = prompt.split("When to use: trigger\n\n", 1)[1]
4898+
assert shown.startswith("First para.\n… INCOMPLETE")
4899+
4900+
def test_cut_and_drop_are_logged(self, mixin_host, caplog):
4901+
"""Each truncation and each drop is logged so frequency is observable."""
4902+
import logging
4903+
4904+
with caplog.at_level(logging.INFO):
4905+
mixin_host._build_recalled_skills_prompt(
4906+
[
4907+
_recalled_skill("cut-me", self._steps(40)),
4908+
_recalled_skill("complete", "1. a"),
4909+
_recalled_skill("drop-me", self._steps(40)),
4910+
],
4911+
self._capped(300),
4912+
)
4913+
4914+
messages = [r.getMessage() for r in caplog.records]
4915+
assert any("truncated procedure 'cut-me'" in m for m in messages)
4916+
assert any("dropped procedure 'drop-me'" in m for m in messages)
47574917

47584918

47594919
class TestRecallOnceProcedureCache:

0 commit comments

Comments
 (0)