feat: add `delete` RUN_TYPE to retrieval-agent benchmarks by sscargal · Pull Request #1363 · MemMachine/MemMachine

sscargal · 2026-04-21T22:42:24Z

Purpose of the change

Standardizes cleanup of ingested benchmark data across locomo, wikimultihop,
hotpotqa, and longmemeval. Before this, only a broken stand-alone locomo
delete script existed; iterative experiments had to clear data by hand.

Description

Wired delete RUN_TYPE across all four retrieval-agent benchmarks.

evaluation/retrieval_agent/locomo_delete.py — rewrote using the current agent_utils.init_memmachine_params + EpisodicMemory.delete_session_episodes() pattern (old code called a removed API and pointed at a stale config file).
evaluation/retrieval_agent/wikimultihop_delete.py — new file, deletes session group1.
evaluation/retrieval_agent/hotpotQA_test.py — added hotpotqa_delete() + --run-type delete branch (session hotpotqa_group).
evaluation/retrieval_agent/longmemeval_test.py — added longmemeval_delete() + --run-type delete branch (session from --session-id).
evaluation/retrieval_agent/run_test.sh — updated per-benchmark usage strings, validate_args (delete = 4 positional args for all), conditional positional unpack, DELETE_CMD per benchmark, and a new elif delete dispatch branch. Skips rm -f
on result files when run-type is delete.
evaluation/retrieval_agent/README.md — added a "Deleting Ingested Data" section with per-benchmark invocations.
evaluation/retrieval_agent/test_run_test.py — 5 new tests: help text, concurrency-flag rejection for delete, extra-positional rejection, and an end-to-end delete dispatch check using a tmp run_test.sh copy.

Fixes/Closes

Fixes #1235

Type of change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration.

Manual verification (list step-by-step instructions)

Test Results: [Attach logs, screenshots, or relevant output]

Verified: bash -n passes, ruff check and ruff format --check pass, all 10 test_run_test.py tests pass. Help output for every benchmark now documents delete. Real end-to-end delete requires a configured Neo4j/postgres backend and was not exercised in this session.

Checklist

I have signed the commit(s) within this pull request
My code follows the style guidelines of this project (See STYLE_GUIDE.md)
I have performed a self-review of my own code
I have commented my code
My changes generate no new warnings
New and existing unit tests pass locally with my changes
I have checked my code and corrected any misspellings

Maintainer Checklist

Confirmed all checks passed
Contributor has signed the commit(s)
Reviewed the code
Run, Tested, and Verified the change(s) work as expected

Screenshots/Gifs

N/A

Further comments

None

Standardizes cleanup of ingested benchmark data across locomo, wikimultihop, hotpotqa, and longmemeval. Before this, only a broken stand-alone locomo delete script existed; iterative experiments had to clear data by hand. - Rewrite locomo_delete.py against the current agent_utils + EpisodicMemory.delete_session_episodes() API (old code called a removed method and pointed at a stale config filename). - Add wikimultihop_delete.py and extend hotpotQA_test.py / longmemeval_test.py with a `delete` --run-type branch that reuses each benchmark's ingest session key. - Wire `delete` through run_test.sh: per-benchmark usage, 4-arg validation, conditional positional unpack, DELETE_CMD, new dispatch branch, and skip the result-file rm on delete. - Document in README and cover help/validation/dispatch in test_run_test.py. Signed-off-by: Steve Scargall <[email protected]>

Copilot

Pull request overview

Adds a standardized delete run type to the retrieval-agent benchmark harness to remove previously ingested episodic data, making iterative benchmark runs reproducible without manual database cleanup.

Changes:

Wires delete dispatch into evaluation/retrieval_agent/run_test.sh for locomo, wikimultihop, hotpotqa, and longmemeval.
Implements/updates per-benchmark delete behaviors (new wikimultihop_delete.py, rewired locomo_delete.py, and new delete branches in HotpotQA/LongMemEval runners).
Documents delete usage and adds unit tests covering help/arg-validation and delete dispatch.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
evaluation/retrieval_agent/wikimultihop_delete.py	New script to delete WikiMultiHop session (`group1`).
evaluation/retrieval_agent/locomo_delete.py	Rewritten LoCoMo delete to use `agent_utils.init_memmachine_params()` + `delete_session_episodes()`.
evaluation/retrieval_agent/hotpotQA_test.py	Adds `--run-type delete` pathway deleting `hotpotqa_group`.
evaluation/retrieval_agent/longmemeval_test.py	Adds `--run-type delete` pathway deleting the configured `--session-id`.
evaluation/retrieval_agent/run_test.sh	Adds `delete` run type across benchmarks; updates usage, arg validation, and dispatch.
evaluation/retrieval_agent/README.md	Documents “Deleting Ingested Data” with per-benchmark commands/notes.
evaluation/retrieval_agent/test_run_test.py	Adds tests for help text, delete arg validation, and end-to-end delete dispatch (longmemeval).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-21T22:47:12Z

 usage_wiki() {
-    echo "WikiMultihop Usage: wikimultihop $0 RESULT_POSTFIX RUN_TYPE TEST_TARGET LENGTH"
+    echo "WikiMultihop Usage: wikimultihop $0 RESULT_POSTFIX RUN_TYPE TEST_TARGET [LENGTH]"
    echo
    echo "Arguments:"
    echo "  RESULT_POSTFIX    Custom postfix for output files"
-    echo "  RUN_TYPE          Run ingestion or search [ingest | search]"
+    echo "  RUN_TYPE          Run ingestion, search, or delete [ingest | search | delete]"
    echo "  TEST_TARGET       [memmachine | retrieval_agent | llm]"
-    echo "  LENGTH            Number of examples to run [1 - 12576]"
+    echo "  LENGTH            Number of examples to run [1 - 12576] (ingest/search only)"


usage_wiki help text is misleading/inconsistent with the script’s actual argument validation: it shows wikimultihop $0 ... [LENGTH] (1) the $0 placement is reversed vs the other usage blocks and (2) [LENGTH] reads as optional for all run types, but validate_args still requires a LENGTH positional for ingest/search. Consider switching to a two-line usage format like HotpotQA/LongMemEval (separate {ingest|search} vs delete forms) and keep $0 first for consistency.

Copilot · 2026-04-21T22:47:12Z

+    async def process_conversation(idx, item):
        if "conversation" not in item:
            return

-        conversation = item["conversation"]
-        speaker_a = conversation["speaker_a"]
-        speaker_b = conversation["speaker_b"]
-
-        print(
-            f"Processing conversation for group {idx} with speakers {speaker_a} and {speaker_b}..."
-        )
-
        group_id = f"group_{idx}"
+        print(f"Deleting episodes for group {group_id}...")

-        memory = cast(
-            EpisodicMemory,
-            await memory_manager.get_episodic_memory_instance(
-                group_id=group_id,
-                session_id=group_id,
-                user_id=[speaker_a, speaker_b],
-            ),
+        memory, _, _ = await agent_utils.init_memmachine_params(
+            resource_manager=resource_manager,
+            session_id=group_id,
        )
+        await memory.delete_session_episodes()

-        await memory.delete_data()
-        await memory.close()
-
-    tasks = [
-        process_conversation(idx, item, memory_manager)
-        for idx, item in enumerate(locomo_data)
-    ]
+    tasks = [process_conversation(idx, item) for idx, item in enumerate(locomo_data)]
    await asyncio.gather(*tasks)


locomo_delete.py launches one async task per conversation with asyncio.gather(*) and no concurrency limit. If --data-path ever points at a larger LoCoMo JSON (more than the current 10 groups), this can create a large burst of init_memmachine_params() + delete calls and potentially overwhelm the backing stores. Consider adding a --concurrency arg (mirroring locomo_ingest.py) and gating deletions with a semaphore (or reusing memmachine_server.common.utils.async_with).

- Add beam_delete.py for deleting ingested BEAM data - Update run_test.sh to support delete run type for BEAM - Update README.md with BEAM delete documentation Aligns with MemMachine#1363 (feat: add delete RUN_TYPE to retrieval-agent benchmarks) Signed-off-by: Junhyeok Park <[email protected]>

* docs: document BEAM benchmark usage and dataset download * feat: add BEAM benchmark support to test runner * fix: make generate_scores.py compatible with BEAM evaluation output * feat: add BEAM benchmark search script * feat: add BEAM benchmark ingestion script * feat: add BEAM-specific rubric-based evaluation * feat: add BEAM dataset download utility * feat: integrate official BEAM evaluation approach - Use official BEAM unified_llm_judge_base_prompt from https://github.com/mohammadtavakoli78/BEAM - Implement 0.0/0.5/1.0 scoring scale (float preserved) - Add event ordering evaluation with Kendall tau-b normalized - Include responsiveness check anchored to the question - Add semantic tolerance rules (paraphrases, synonyms, numeric equivalence) - Add style neutrality to prevent style contamination - Output llm_judge_responses with per-criterion scores and reasons This aligns MemMachine's BEAM evaluation with the official reference implementation, enabling fair comparison across memory providers. Signed-off-by: Junhyeok Park <[email protected]> * docs: add BEAM dependency requirements for scipy and datasets Document that BEAM evaluation requires additional packages: - scipy: for Kendall tau-b correlation in event ordering evaluation - datasets: for downloading BEAM dataset from HuggingFace Signed-off-by: Junhyeok Park <[email protected]> * refactor: consolidate BEAM benchmark files into dedicated directory - Move beam_*.py files from evaluation/retrieval_agent/ to evaluation/retrieval_agent/beam/ - Move beam_download.py from evaluation/data/ to evaluation/retrieval_agent/beam/ - Create evaluation/retrieval_agent/beam/README.md with full BEAM documentation - Simplify evaluation/retrieval_agent/README.md to reference beam/README.md - Update run_test.sh paths to point to new beam/ directory - Add __init__.py for beam package Signed-off-by: Junhyeok Park <[email protected]> * refactor: address Copilot review comments for BEAM benchmark - beam_ingest.py: Remove unused concurrency parameter - beam_search.py: Fix concurrency batching for last category - beam_evaluate.py: Write results once after all evaluations complete - generate_scores.py: Fix llm_score check to use strict equality - beam_download.py: Remove unused os import These changes improve code quality and fix performance issues identified during AI-assisted code review. Signed-off-by: Junhyeok Park <[email protected]> * feat: add LLM-based fact alignment for BEAM event ordering - Add llm_equivalence() to check semantic equivalence of facts - Add align_with_llm() for LLM-based fact alignment - Use tau_norm as primary metric for event_ordering category Signed-off-by: Junhyeok Park <[email protected]> * fix: resolve ruff linting errors (SIM118, B905, F841) and format code Signed-off-by: Junhyeok Park <[email protected]> * docs: add note about official BEAM 0.5 score dropping behavior - Document that official BEAM code casts scores to int(), dropping 0.5 - Clarify that this implementation preserves float scores for partial compliance - Note that scores may differ by a few percentage points Signed-off-by: Junhyeok Park <[email protected]> * feat: support 10M BEAM dataset format in ingestion and search - Add format detection for 100K/500K/1M (nested plan-X) vs 10M (flat) - Flatten nested structure in beam_ingest.py - Flatten nested structure in beam_search.py for llm target Signed-off-by: Junhyeok Park <[email protected]> * feat: support 10M BEAM dataset download - Add Mohammadta/BEAM-10M dataset mapping - Add '10M' to choices in argument parser Signed-off-by: Junhyeok Park <[email protected]> * docs: update README to reflect 10M dataset support - Remove '10M not supported' note - Add 10M download example Signed-off-by: Junhyeok Park <[email protected]> * style: fix ruff SIM118 warnings in beam module - Remove unnecessary .keys() calls in iteration Signed-off-by: Junhyeok Park <[email protected]> * feat: support 10M BEAM dataset in beam_download.py - Add Mohammadta/BEAM-10M dataset mapping - Add '10M' to choices in argument parser - Fix convert_chats_to_json to handle 10M plan-based structure - Split conversion logic into separate functions for clarity Signed-off-by: Junhyeok Park <[email protected]> * feat: add delete RUN_TYPE to BEAM benchmark - Add beam_delete.py for deleting ingested BEAM data - Update run_test.sh to support delete run type for BEAM - Update README.md with BEAM delete documentation Aligns with #1363 (feat: add delete RUN_TYPE to retrieval-agent benchmarks) Signed-off-by: Junhyeok Park <[email protected]> * refactor: address Copilot review comments and ruff failures - README.md: Fix broken markdown code block nesting (line 560-573) - generate_scores.py: Remove unused beam_rubric variable - generate_scores.py: Keep llm_score == 1 check (intentional for BEAM 0.5 scoring) - run_test.sh: Fix BEAM ingest to require 5 args (not 6), QUESTIONS_PATH not needed - beam/README.md: Update ingest usage (QUESTIONS_PATH not needed) - beam_ingest.py: Fix metadata falsy value handling (use 'is not None' check) - beam_ingest.py: Add format validation with clear error messages - beam_search.py: Add format validation with clear error messages - beam_evaluate.py: Filter question echoes from extract_facts_from_response - beam_evaluate.py: Add logging for LLM judge parse failures - beam_evaluate.py: Add optional dependency checks for json_repair/scipy Fixes #1235 (follow-up to delete RUN_TYPE + separate LLM models) Signed-off-by: Junhyeok Park <[email protected]> --------- Signed-off-by: Junhyeok Park <[email protected]>

sscargal requested review from Tianyang-Zhang, Copilot, edwinyyyu and malatewang April 21, 2026 22:42

Copilot started reviewing on behalf of sscargal April 21, 2026 22:43 View session

Copilot AI reviewed Apr 21, 2026

View reviewed changes

sscargal added this to the v0.3.6 milestone Apr 21, 2026

edwinyyyu approved these changes Apr 21, 2026

View reviewed changes

jealous approved these changes Apr 21, 2026

View reviewed changes

sscargal merged commit 3476f63 into MemMachine:main Apr 21, 2026
47 of 48 checks passed

Navigation Menu

Repository navigation

feat: add `delete` RUN_TYPE to retrieval-agent benchmarks - #1363

feat: add `delete` RUN_TYPE to retrieval-agent benchmarks#1363
sscargal merged 1 commit into
MemMachine:mainfrom
sscargal:feat/1235-add-delete-command-to-remove-ingested-data

sscargal commented Apr 21, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 21, 2026

Uh oh!

Copilot AI Apr 21, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

sscargal commented Apr 21, 2026

Purpose of the change

Description

Fixes/Closes

Type of change

How Has This Been Tested?

Checklist

Maintainer Checklist

Screenshots/Gifs

Further comments

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants