Skip to content

Add GoodMemory as a third backend (local HTTP bridge) - #17

Open
hjqcan wants to merge 7 commits into
mem0ai:mainfrom
hjqcan:add-goodmemory-backend
Open

hjqcan wants to merge 7 commits into
mem0ai:mainfrom
hjqcan:add-goodmemory-backend

Conversation

@hjqcan

@hjqcan hjqcan commented Jul 5, 2026 •

Copy link
Copy Markdown

Adds GoodMemory v0.7.5 as a local HTTP backend next to Mem0 Cloud/OSS.

What is included

  • benchmarks/common/goodmemory_client.py: an async adapter around the published goodmemory-client==0.1.0 package. The official client owns the HTTP wire contract, caller/scope headers, bearer auth, retries, and routing metadata.
  • --backend goodmemory plus --goodmemory-host in the LoCoMo, LongMemEval, and BEAM runners.
  • README setup using goodmemory-http-bridge --recommended; embeddings are optional.

Contract and evidence boundaries

  • Deterministic fixture ingestion: every benchmark turn is imported as a verified fact with its timestamp preserved. GoodMemory 0.7.5's default bridge policy does not persist assistant assertions directly, so benchmark assistant turns are imported as verified fixture facts while their original role is preserved both in the indexed text and in metadataPatch.attributes.sourceRole.
  • Run isolation: each project-derived user id creates a separate GoodMemory scope. The documented in-memory bridge is fresh per process, and each run should use a fresh --project-name.
  • Observable routing: the adapter requests auto by default and reports strategy fallback instead of hiding a retrieval downgrade.
  • Published recall limit: the 0.7.5 recall-context contract returns at most 12 selected items and has no caller-controlled item limit. The documented comparable configuration therefore uses --top-k 10 --top-k-cutoffs 10; requests above 12 emit an explicit warning.
  • No invented relevance score: the bridge publishes ranked items but no numeric score. The adapter preserves bridge order and records 0.0 instead of synthesizing a number that could influence the retrieval judge.
  • No benchmark claim: the smoke below verifies integration behavior only; it is not evidence of benchmark quality or uplift.

Validation

Environment: Python 3.12.13, Node v22.14.0, Bun 1.3.14, npm 10.9.2.

  • python -m unittest discover -s tests -v — 6/6 passed.
  • python -m py_compile for the adapter and all three runners.
  • All three runner --help paths load and expose the GoodMemory backend under Python 3.12.
  • python -m pip check — no broken requirements.
  • git diff --check.
  • Live smoke against the exact published goodmemory@0.7.5 package with bearer auth and an in-memory recommended bridge: two timestamped fixture facts accepted, a same-scope query recalled the relevant deployment-region fact, and a different scope returned no results. The bridge reported retrievalTier: preset-recommended; embeddings were intentionally disabled for this contract smoke.

No benchmark answerer, judge, paid model, or score was run as part of this validation.

Adds benchmarks/common/goodmemory_client.py - an async client for the
GoodMemory HTTP bridge (npm i -g goodmemory; goodmemory-http-bridge) exposing
the same add/search/delete_user interface as Mem0Client - and wires
--backend goodmemory (+ --goodmemory-host) into the LOCOMO, LongMemEval, and
BEAM runners.

Notes:
- Bearer-token auth (GOODMEMORY_HTTP_BRIDGE_TOKEN) and the bridge's
  per-request caller header are handled automatically.
- Benchmark seeding writes every turn deterministically (verified fact
  annotations + rules-only extraction), mirroring how GoodMemory's own
  benchmark harnesses ingest conversations; assistant turns are stored as
  user-scoped facts with the original role preserved in the content, since
  assistant-authored writes are blocked by the product write policy.
- Runs are isolated by scope (user ids embed the project name); the bridge
  has no bulk delete endpoint, so delete_user relies on fresh project names.
- Smoke-tested end to end against a live bridge (add -> search round trip).
hjqcan added 2 commits July 6, 2026 09:39
…tack

The bridge coerces any non-hybrid recall strategy (including auto) to a
rules-only lexical floor, which cannot surface semantically-relevant facts.
Default recall_strategy to hybrid so representative recall works out of the
box (override via GOODMEMORY_RECALL_STRATEGY), and document the bridge config
it needs: an embedding endpoint, the recommended retrieval preset (semantic
candidate union), and in-memory storage. Deterministic verbatim seeding is
unchanged (no LLM extractor required). Validated 4/4 on cross-topic recall
probes (the query-relevant fact ranks first).
@hjqcan

hjqcan commented Jul 13, 2026

Copy link
Copy Markdown
Author

Refreshed the GoodMemory adapter in commit 6e3b204 to preserve each runner's timestamp / observation_date as a UTC observation prefix before deterministic ingestion. This closes a temporal-fidelity gap affecting BEAM, LongMemEval, and LoCoMo.

Validation:

  • python -m unittest discover -s tests -v (3/3)
  • python -m py_compile for the adapter and all three runners
  • live smoke against the current GoodMemory HTTP bridge: write with Unix timestamp, recall returns [Observed at 2023-05-08T00:00:00Z] ...

The PR remains ready for maintainer review.

@hjqcan

hjqcan commented Jul 31, 2026

Copy link
Copy Markdown
Author

Refreshed this integration for the published GoodMemory v0.7.0 in acf0d21. The benchmark adapter now delegates the wire protocol to the official goodmemory-client package instead of maintaining a second HTTP implementation, reducing the adapter from 298 to 202 lines while adding caller/scope correctness and observable recall routing. The setup no longer requires an embedding provider: goodmemory-http-bridge --recommended provides the local BM25/entity/RRF tier, with dense retrieval optional.

Validation is current: 5/5 unit tests, all three runners compile, git diff --check passes, and a live bearer-authenticated write/recall smoke succeeded against npm goodmemory@0.7.0. GitHub reports the branch as mergeable; it is still awaiting maintainer review.

@hjqcan

hjqcan commented Aug 1, 2026 •

Copy link
Copy Markdown
Author

Current verification update: commit 5f70b8b refreshes the integration evidence and setup examples to the published GoodMemory v0.7.5.

Validation on Python 3.12.13, Node v22.14.0, Bun 1.3.14, and npm 10.9.2:

  • 6/6 unit tests passed
  • adapter and all three runners compile
  • all three runner --help paths expose the GoodMemory backend
  • pip check and git diff --check are clean
  • live bearer-authenticated smoke against goodmemory@0.7.5: timestamped fixture ingestion, same-scope recall, and different-scope isolation

This is integration/runtime evidence only. No benchmark answerer, judge, paid model, or score was run. The PR is mergeable and still awaiting maintainer review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant