Improve embedding fidelity (speedkick) - #1587

Copy of #1449 onto speedkick. Cherry-picked cleanly; no conflicts and no changes were needed. Targeted suite server_tests/memmachine_server/common/embedder passes (5 passed).

Purpose of the change

Chunk-merged embeddings (texts longer than the model input limit are split, embedded per chunk, and averaged) reproduce the whole-text embedding more faithfully with greedy max-size chunks and a length-weighted average than with the current balanced chunks and unweighted mean. Measured on three encoders; numbers below.

OpenAI servers no longer error when special tokens are given to text-embedding-3-small, so the special-token sanitizer is removed (probe script and results attached as a PR comment below).

Description

• Use greedy max-size chunks (chunk_text) instead of balanced chunks (chunk_text_balanced) in the OpenAI, SentenceTransformer, and Amazon Bedrock embedders.
• Merge chunk embeddings with a length-weighted np.average instead of unweighted np.mean.
• Remove special token processing.

Measurements

Method: 10 hand-written texts in distinct registers (fiction, technical docs, chat transcript, news, legal, academic, recipe, business memo, product review, encyclopedia), each short enough to embed whole — the whole-text embedding is the observable ground truth. Each text is chunked at limits 800 / 1600 / 3200 chars under each policy, chunks embedded, merged, and compared to the whole-text vector by cosine similarity. Single-chunk cells excluded; 28 cells per configuration per model. Probe script and raw per-cell results are attached as PR comments below (deliberately not committed to the branch).

This PR's configuration (greedy + length-weighted) vs current behavior (balanced + unweighted), paired over the same 28 cells:

Full split × weighting grid (mean cosine to whole-text embedding, higher = more faithful):

Two structural checks that this is signal, not noise:

• Greedy requires the weighting: greedy+unweighted ranks last on all three encoders. The two changes in this PR are a package.
• Balanced is indifferent to weighting (its two rows agree to ~5 decimals on every encoder), as expected for equal-size chunks — a built-in sanity check on the harness.

Mechanism: to the extent an encoder mean-pools token states, the whole-text vector is approximately a length-weighted average over the text, which a merge approximates best when weights match chunk lengths and most characters sit in maximal-context chunks. The result holds on the OpenAI models even though their pooling is undocumented.

Caveat: per-cell variance is wider on the OpenAI models than on gemma (worst cell −0.019, best +0.048), so this is a consistent aggregate win rather than a uniform per-text one.

Type of change

• Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

• Unit Test
• Integration Test
• End-to-end Test
• Test Script (please provide)
• Manual verification (list step-by-step instructions)

Test Results: Fidelity tables above; probe scripts and raw results attached as PR comments below; embedder unit tests pass (server_tests/memmachine_server/common/embedder/).

Checklist

• I have signed the commit(s) within this pull request
• My code follows the style guidelines of this project (See STYLE_GUIDE.md)
• I have performed a self-review of my own code
• I have commented my code
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added unit tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged and published in downstream modules
• I have checked my code and corrected any misspellings

Maintainer Checklist

• Confirmed all checks passed
• Contributor has signed the commit(s)
• Reviewed the code
• Run, Tested, and Verified the change(s) work as expected

🤖 Generated with Claude Code

https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL