Skip to content

Fix local failing tests and use a single flag for integration tests - #1038

Merged
edwinyyyu merged 1 commit into
MemMachine:mainfrom
edwinyyyu:weird_tests
Feb 4, 2026
Merged

edwinyyyu merged 1 commit into
MemMachine:mainfrom
edwinyyyu:weird_tests

Conversation

@edwinyyyu

@edwinyyyu edwinyyyu commented Feb 3, 2026 •

Copy link
Copy Markdown
Contributor

Purpose of the change

Version test was failing locally. Tests have been failing in main for ages. This should make it somewhat better.

Tests were failing when sentence_transformers was not installed.
Tests were failing when Docker is not up.

Description

Used Claude Code.

Now tests should just work after clone.

Unit tests: uv run pytest or uv run --all-extras pytest
Integration tests: uv run pytest -m integration or uv run --all-extras pytest -m integration
All tests: uv run pytest -m "" or uv run --all-extras pytest -m ""

Limitations:

  • If CI suddenly doesn't have Docker, tests will be skipped instead of erroring. An alternative design would be to add flags indicating to run sentence_transformers or testcontainers tests. But a similar problem exists for missing API keys.

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Refactor (does not change functionality, e.g., code style improvements, linting)
  • Documentation update
  • Project Maintenance (updates to build scripts, CI, etc., that do not affect the main project)
  • Security (improves security without changing functionality)

How Has This Been Tested?

All tests pass locally or are skipped.

  • Unit Test
  • Integration Test
  • End-to-end Test
  • Test Script (please provide)
  • Manual verification (list step-by-step instructions)

Checklist

  • I have signed the commit(s) within this pull request
  • My code follows the style guidelines of this project (See STYLE_GUIDE.md)
  • I have performed a self-review of my own code
  • I have commented my code
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added unit tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published in downstream modules
  • I have checked my code and corrected any misspellings

Maintainer Checklist

  • Confirmed all checks passed
  • Contributor has signed the commit(s)
  • Reviewed the code
  • Run, Tested, and Verified the change(s) work as expected

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes failing tests that occurred when sentence_transformers is not installed or Docker is unavailable. The changes replace the command-line --integration flag with pytest markers and environment checks to gracefully skip tests when dependencies are missing.

Changes:

  • Removed the --integration command-line option in favor of automatic test skipping based on available dependencies
  • Added helper functions and decorators to detect Docker availability and sentence_transformers installation
  • Modified test fixtures to use lazy imports for optional dependencies
  • Updated the version test regex pattern to handle additional version formats

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
tests/memmachine/conftest.py Removed --integration CLI option; added is_docker_available(), requires_sentence_transformers, and requires_docker helpers
tests/memmachine/episodic_memory/long_term_memory/test_long_term_memory.py Moved imports inside fixtures and added skip decorators for sentence_transformers-dependent tests
tests/memmachine/episodic_memory/declarative_memory/test_declarative_memory.py Moved imports inside fixtures and added skip decorators for sentence_transformers-dependent tests
tests/memmachine/common/vector_graph_store/test_neo4j_vector_graph_store.py Added Docker availability check to neo4j_connection_info fixture
tests/memmachine/common/resource_manager/test_reranker_manager.py Added skip decorators for sentence_transformers-dependent tests
tests/memmachine/common/resource_manager/test_embedder_manager.py Added skip decorators for sentence_transformers-dependent tests
tests/memmachine/common/reranker/test_cross_encoder_reranker.py Added pytest.importorskip and simplified fixtures
tests/memmachine/common/embedder/test_sentence_transformer_embedder.py Added pytest.importorskip and simplified fixtures
tests/memmachine/common/api/test_version.py Updated version regex to support major.minor format in addition to major.minor.patch
pyproject.toml Added default pytest option to exclude integration tests
.github/workflows/pytest-integration.yml Removed --integration flag from pytest commands
.github/workflows/installation-test.yml Removed --integration flag from pytest commands

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread tests/memmachine/conftest.py
Comment thread tests/memmachine/common/api/test_version.py
@edwinyyyu
edwinyyyu merged commit c267260 into MemMachine:main Feb 4, 2026
40 of 41 checks passed
edwinyyyu added a commit to edwinyyyu/MemMachine that referenced this pull request Sep 10, 2026
Every pull request against speedkick fails the "Run Python unit tests"
job on ubuntu and macos with

    FAILED .../common/api/test_version.py::test_get_version
    AssertionError: Invalid server version: 0.3.9.post2.dev14+g2c69bb83e

The annotated tag v0.3.9-post1 (2026-08-31, "Post-release build from
speedkick, after v0.3.9") sits on speedkick's ancestry and is the
nearest tag for every commit since, so setuptools-scm renders each
build as 0.3.9.post2.devN+g<sha>. The test's hand-written pattern had
no post-release segment. main is unaffected because the tag is not
reachable from it.

The test is deleted rather than patched. get_version() is a wrapper
around importlib.metadata.version for two distribution names; nothing
in it produces a version shape. The shape comes from setuptools-scm,
the repository's tags and the working tree, so the test could only
ever fail for environmental reasons, and that is its whole history:
the regex was widened in MemMachine#993 (dotted local segment), MemMachine#1038
(two-component tag) and would have been again here, while the function
under test has not changed since MemMachine#951 wrote it. It also rejected a
documented output: MemMachine#951 specifies client_version as "not available"
when memmachine-client is not installed, which is the case for a wheel
install of the server, whose package does not depend on the client.
The test passed only because the uv workspace installs every member.
test_version_string stays; it tests the model's own string form.

The check moves to the layer that owns it. The Test Server Package
workflow already builds the wheels and installs them; a new step runs
the installed memmachine-server --version and requires the reported
server version to equal the version in the wheel's filename. That ties
the binary to the artifact it shipped in without asserting anything
about the version's format.

Verified locally on this branch: the trimmed test file passes and is
clean under ruff check and ruff format --check; the new step, run
verbatim against wheels built from this tree and installed into a
fresh virtualenv, passes.

The windows unit-test jobs report success only because the step runs
under pwsh, which does not stop on the first command's non-zero exit
and returns the last command's status (the client suite). Their server
run fails this test plus two windows-only tests
(test_nebula_graph_vector_literals charmap decode, test_index_persistence
flush guard). That masking is left as is: forcing bash on the step
would turn windows red until those two are fixed.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_019zu1CSWVso2NA4bWPYmw8Z
edwinyyyu added a commit that referenced this pull request Sep 10, 2026
…kick) (#1604)

Verify the version in the package workflow, not in a unit test

Every pull request against speedkick fails the "Run Python unit tests"
job on ubuntu and macos with

    FAILED .../common/api/test_version.py::test_get_version
    AssertionError: Invalid server version: 0.3.9.post2.dev14+g2c69bb83e

The annotated tag v0.3.9-post1 (2026-08-31, "Post-release build from
speedkick, after v0.3.9") sits on speedkick's ancestry and is the
nearest tag for every commit since, so setuptools-scm renders each
build as 0.3.9.post2.devN+g<sha>. The test's hand-written pattern had
no post-release segment. main is unaffected because the tag is not
reachable from it.

The test is deleted rather than patched. get_version() is a wrapper
around importlib.metadata.version for two distribution names; nothing
in it produces a version shape. The shape comes from setuptools-scm,
the repository's tags and the working tree, so the test could only
ever fail for environmental reasons, and that is its whole history:
the regex was widened in #993 (dotted local segment), #1038
(two-component tag) and would have been again here, while the function
under test has not changed since #951 wrote it. It also rejected a
documented output: #951 specifies client_version as "not available"
when memmachine-client is not installed, which is the case for a wheel
install of the server, whose package does not depend on the client.
The test passed only because the uv workspace installs every member.
test_version_string stays; it tests the model's own string form.

The check moves to the layer that owns it. The Test Server Package
workflow already builds the wheels and installs them; a new step runs
the installed memmachine-server --version and requires the reported
server version to equal the version in the wheel's filename. That ties
the binary to the artifact it shipped in without asserting anything
about the version's format.

Verified locally on this branch: the trimmed test file passes and is
clean under ruff check and ruff format --check; the new step, run
verbatim against wheels built from this tree and installed into a
fresh virtualenv, passes.

The windows unit-test jobs report success only because the step runs
under pwsh, which does not stop on the first command's non-zero exit
and returns the last command's status (the client suite). Their server
run fails this test plus two windows-only tests
(test_nebula_graph_vector_literals charmap decode, test_index_persistence
flush guard). That masking is left as is: forcing bash on the step
would turn windows red until those two are fixed.


Claude-Session: https://claude.ai/code/session_019zu1CSWVso2NA4bWPYmw8Z

Co-authored-by: Claude Fable 5.1 <[email protected]>
malatewang added a commit that referenced this pull request Sep 19, 2026
test_get_version asserts that get_version() returns a version matching a
hand-written pattern: release, optional .devN, optional local segment.
On speedkick that assertion fails on every pull request, because the
annotated tag v0.3.9-post1 (2026-08-31, "Post-release build from
speedkick, after v0.3.9") sits on that branch's ancestry and
setuptools-scm renders each build as 0.3.9.post2.devN+g<sha>, a shape
the pattern never allowed. main passes today only because that tag is
not yet reachable from it; it becomes reachable the moment speedkick
merges, and any post-release tag would do the same.

The test is deleted rather than patched. get_version() is a wrapper
around importlib.metadata.version for two distribution names; nothing
in it produces a version shape. The shape comes from setuptools-scm,
the repository's tags and the working tree, so the test could only
ever fail for environmental reasons, and that is its whole history:
the regex was widened in #993 (dotted local segment), #1038
(two-component tag) and would have been again here, while the function
under test has not changed since #951 wrote it. It also rejected a
documented output: #951 specifies client_version as "not available"
when memmachine-client is not installed, which is the case for a wheel
install of the server, whose package does not depend on the client.
The test passed only because the uv workspace installs every member.
test_version_string stays; it tests the model's own string form.

The check moves to the layer that owns it. The Test Server Package
workflow already builds the wheels and installs them; a new step runs
the installed memmachine-server --version and requires the reported
server version to equal the version in the wheel's filename. That ties
the binary to the artifact it shipped in without asserting anything
about the version's format.

Verified locally on this branch: the trimmed test file passes and is
clean under ruff check and ruff format --check; the new step, run
verbatim against wheels built from this tree and installed into a
fresh virtualenv, passes (and reports "client: not available", which
the deleted test would have rejected).

Unrelated but worth knowing: the pytest workflow's test step has no
shell key, so on windows it runs under pwsh, which does not stop on the
first command's non-zero exit and returns the last command's status
(the client suite). A server-suite failure on windows would not fail
the job. On main the windows server run passes today (1830 passed on
#1584's run), so nothing is hidden yet; on speedkick two windows-only
failures are. Forcing bash there is left for when those are fixed.

Same change as #1604 on speedkick, applied to main.

Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
Co-authored-by: Shu Wang <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants