[Feat]: Add delete command support to run_test.sh for benchmark evaluation #1235

Activity

1. hyeongtakji

   added the

   Feature

   issue type on Mar 17, 2026
2. sscargal commented on Apr 21, 2026

   

   sscargal

   on Apr 21, 2026

   Contributor

   More actions

   Here is a proposed plan.

   Context

   The retrieval-agent benchmark runner (evaluation/retrieval_agent/run_test.sh) supports
   ingest and search for four benchmarks (locomo, wikimultihop, hotpotqa, longmemeval).
   There is no standardized way to remove previously-ingested data, which makes iterative
   experimentation cumbersome. Only locomo_delete.py exists today, and it is:

   • never wired into run_test.sh
   • undocumented
   • broken against the current code path: it calls
     EpisodicMemoryManager.create_episodic_memory_manager("locomo_config.yaml") (stale
     config filename) and memory.delete_data() which no longer exists on EpisodicMemory
     (the current API is EpisodicMemory.delete_session_episodes() — see
     packages/server/src/memmachine_server/episodic_memory/episodic_memory.py:279).

   Goal: add a uniform delete RUN_TYPE across all four benchmarks, fix locomo_delete.py,
   add wikimultihop_delete.py, and extend the existing hotpotQA_test.py /
   longmemeval_test.py (which already dispatch on --run-type) with a delete branch.

   Proposed user-facing syntax (matches issue #1235):

   ./run_test.sh locomo exp1 delete retrieval_agent
   ./run_test.sh wikimultihop exp1 delete retrieval_agent
   ./run_test.sh hotpotqa exp1 delete retrieval_agent
   ./run_test.sh longmemeval exp1 delete retrieval_agent

   Four positional args for every benchmark: TEST RESULT_POSTFIX delete TEST_TARGET.
   No SPLIT_NAME or LENGTH needed — delete only needs the session identifier, which is
   derived from RESULT_POSTFIX (or is a benchmark-fixed constant). TEST_TARGET is
   accepted for symmetry with ingest/search but is not used.

   Approach

   1. Reuse the existing delete API

   EpisodicMemory.delete_session_episodes() already clears all long-term memory
   episodes bound to a session key. Build an EpisodicMemory instance via the same
   agent_utils.init_memmachine_params(resource_manager, session_id=...) path used by
   every ingest script, then await delete_session_episodes().

   No new server-side code is needed.

   2. Files to modify

   evaluation/retrieval_agent/locomo_delete.py — rewrite

   • Replace the stale EpisodicMemoryManager.create_episodic_memory_manager("locomo_config.yaml")
     plumbing with the current pattern used in locomo_ingest.py:43-75:
     • argparse for --data-path and --config-path
     • agent_utils.load_eval_config(config_path)
     • per-group agent_utils.init_memmachine_params(resource_manager, session_id=f"group_{idx}")
     • await memory.delete_session_episodes()
   • Keep the iteration over conversations (10 groups) because locomo ingest uses one
     session per conversation.

   evaluation/retrieval_agent/wikimultihop_delete.py — new file

   • Mirrors wikimultihop_ingest.py shape (single session "group1").
   • Args: --config-path only. No data file needed — wikimultihop uses a single session_id.
   • Init memory with session_id="group1", call delete_session_episodes().

   evaluation/retrieval_agent/hotpotQA_test.py — extend

   • Add a hotpotqa_delete(config_path: str) coroutine that mirrors the pattern above
     against the fixed session_id="hotpotqa_group" used by hotpotqa_ingest
     (hotpotQA_test.py:76-77).
   • Add "delete" as an accepted value in the existing --run-type argparse choice
     (the arg already exists in the current file). Route to hotpotqa_delete in main().

   evaluation/retrieval_agent/longmemeval_test.py — extend

   • Add a longmemeval_delete(config_path: str, session_id: str) coroutine (same pattern).
   • Accept "delete" as a value for --run-type. Route in main().
   • Reuses the existing --session-id argument.

   evaluation/retrieval_agent/run_test.sh — wire delete through

   1. Usage strings (usage_locomo, usage_wiki, usage_hotpotqa, usage_longmemeval):
      change [ingest | search] to [ingest | search | delete] and add a line explaining
      that delete ignores LENGTH/SPLIT_NAME but still expects TEST_TARGET.
   2. validate_args: for each benchmark, when $3 == "delete", require exactly
      4 positional args (TEST RESULT_POSTFIX delete TEST_TARGET). Concurrency flags
      should be rejected for delete (they are already rejected via the existing
      $3 != "ingest" / $3 != "search" checks — confirm each emits a clear error,
      and add --ingest-concurrency/--search-concurrency/--judge-concurrency
      rejection for delete if not already covered).
   3. run_test case arms: for each TEST, define DELETE_CMD and add an
      elif [[ "$INGEST" = "delete" ]]; then "${DELETE_CMD[@]}" branch. Do NOT run
      the result-file rm -f when deleting (the files aren't relevant to delete).
      Specifically:

   • locomo: DELETE_CMD=(python -u locomo_delete.py --data-path .../locomo10.json --config-path "$CONFIG_FILE")
   • wikimultihop: DELETE_CMD=(python -u wikimultihop_delete.py --config-path "$CONFIG_FILE")
   • hotpotqa: DELETE_CMD=(python -u hotpotQA_test.py --run-type delete --test-target "$TEST_TARGET" --config-path "$CONFIG_FILE")
   • longmemeval: DELETE_CMD=(uv run python -u longmemeval_test.py --run-type delete --test-target "$TEST_TARGET" --session-id "$SESSION_ID" --config-path "$CONFIG_FILE")

   4. Positional count in run_test: add delete to each TEST's positional unpack
      so RESULT_POSTFIX/TEST_TARGET are assigned identically to ingest. For
      hotpotqa/longmemeval — which currently unpack SPLIT_NAME and LENGTH at
      positions 4/6 — make the unpack conditional: if $3 == "delete", only assign
      RESULT_POSTFIX=$2, INGEST=$3, TEST_TARGET=$4.
   5. show_help "Unknown RUN_TYPE": update so delete is valid; unchanged
      fallthrough message is fine.

   evaluation/retrieval_agent/README.md — document

   • Add a "Deleting ingested data" subsection under "Running Benchmarks" with one
     invocation per benchmark.
   • Note that delete removes all episodes for the benchmark's session key(s),
     and that for locomo it iterates all conversation groups.

   evaluation/retrieval_agent/test_run_test.py — extend

   • Add tests analogous to the existing help/validation tests:
     • Each benchmark's help output mentions delete.
     • Each benchmark accepts 4-arg delete invocation through argument validation
       (i.e., does not fail at the validate_args layer). Use the same pattern the
       current tests use — invoke run_test.sh in a way that exits before touching
       Python (the existing tests rely on configuration.yml being absent to stop
       early, which is also convenient here).
     • --ingest-concurrency/--search-concurrency/--judge-concurrency are
       rejected when paired with delete.

   3. Reused utilities / code paths

   • evaluation/utils/agent_utils.py:36 — load_eval_config
   • evaluation/utils/agent_utils.py:383 — init_memmachine_params
   • packages/server/src/memmachine_server/episodic_memory/episodic_memory.py:279
     — EpisodicMemory.delete_session_episodes

   4. Anti-scope

   • No server-side changes. The existing delete_session_episodes is sufficient.
   • No schema changes to configuration.yml.
   • No changes to the REST API, client SDKs, or MCP surface. This issue is
     scoped to the evaluation/retrieval_agent/ benchmark runner only.

   Verification

   From a clean checkout with a working configuration.yml:

   cd evaluation/retrieval_agent

   1. Help strings mention delete
      ./run_test.sh --help
      ./run_test.sh locomo --help
      ./run_test.sh wikimultihop --help
      ./run_test.sh hotpotqa --help
      ./run_test.sh longmemeval --help

   2. Concurrency flags rejected with delete
      ./run_test.sh locomo exp1 delete retrieval_agent --ingest-concurrency 2 # must fail

   3. End-to-end ingest -> delete -> re-ingest on one benchmark (wikimultihop is fastest)
      ./run_test.sh wikimultihop exp1 ingest retrieval_agent 20
      ./run_test.sh wikimultihop exp1 delete retrieval_agent
      ./run_test.sh wikimultihop exp1 search retrieval_agent 20 # should retrieve nothing after delete

   4. LoCoMo roundtrip (verifies multi-group iteration)
      ./run_test.sh locomo exp1 ingest retrieval_agent
      ./run_test.sh locomo exp1 delete retrieval_agent

   5. Unit tests pass
      uv run pytest evaluation/retrieval_agent/test_run_test.py -v

   6. Lint + format clean
      uv run ruff check evaluation/retrieval_agent
      uv run ruff format --check evaluation/retrieval_agent

   For step 3, after delete the Neo4j vector store should contain no episodes for
   session_key="group1". Spot-check with a direct Cypher query if doubt exists:
   MATCH (e:Episode {session_key:"group1"}) RETURN count(e) — should be 0.
3. sscargal

   self-assigned this

   on Apr 21, 2026
4. sscargal

   closed this as completedin #1363on Apr 21, 2026
5. junttang

   added a commit that references this issue on May 19, 2026

   refactor: address Copilot review comments and ruff failures

   ...

   This issue will close once commit acf5049 is merged into the 'main' branch.

   acf5049
6. sscargal

   added a commit that references this issue on May 19, 2026

   feat: add BEAM benchmark support (#1317)

   ...

   This issue will close once commit 68e4fff is merged into the 'main' branch.

   68e4fff
7. honggyukim

   added

   skhynix

   on Sep 19, 2026