Skip to content

MemoryError when writing junitxml for large test suite #641

Description

@pytestbot

Originally reported by: Michael Howitz (BitBucket: icemac, GitHub: icemac)


In a large project (> 3500 Tests) we get a MemoryError when writing the junit xml file.
(The host is our Jenkins CI server with 4 GB of RAM and 2 GB of swap.)

#!python

bin/py.test --nosugar --junitxml=parts/tests.xml --cov-report html --cov-report xml --cov-report term --cov src
============================= test session starts ==============================
platform linux2 -- Python 2.7.6 -- py-1.4.25 -- pytest-2.6.4
plugins: gocept.pytestlayer, cache, cov, sugar, remove-stale-bytecode
collected 3643 items

… Many, many tests which ran fine …

Traceback (most recent call last):
  File "bin/py.test", line 308, in <module>
    sys.exit(pytest.main())
  File "/srv/jenkins/eggs/pytest-2.6.4-py2.7.egg/_pytest/config.py", line 41, in main
    return config.hook.pytest_cmdline_main(config=config)
  File "/srv/jenkins/eggs/pytest-2.6.4-py2.7.egg/_pytest/core.py", line 413, in __call__
    return self._docall(methods, kwargs)
  File "/srv/jenkins/eggs/pytest-2.6.4-py2.7.egg/_pytest/core.py", line 424, in _docall
    res = mc.execute()
  File "/srv/jenkins/eggs/pytest-2.6.4-py2.7.egg/_pytest/core.py", line 315, in execute
    res = method(**kwargs)
  File "/srv/jenkins/eggs/pytest-2.6.4-py2.7.egg/_pytest/main.py", line 116, in pytest_cmdline_main
    return wrap_session(config, _main)
  File "/srv/jenkins/eggs/pytest-2.6.4-py2.7.egg/_pytest/main.py", line 109, in wrap_session
    exitstatus=session.exitstatus)
  File "/srv/jenkins/eggs/pytest-2.6.4-py2.7.egg/_pytest/core.py", line 413, in __call__
    return self._docall(methods, kwargs)
  File "/srv/jenkins/eggs/pytest-2.6.4-py2.7.egg/_pytest/core.py", line 424, in _docall
    res = mc.execute()
  File "/srv/jenkins/eggs/pytest-2.6.4-py2.7.egg/_pytest/core.py", line 315, in execute
    res = method(**kwargs)
  File "/srv/jenkins/eggs/pytest-2.6.4-py2.7.egg/_pytest/terminal.py", line 349, in pytest_sessionfinish
    __multicall__.execute()
  File "/srv/jenkins/eggs/pytest-2.6.4-py2.7.egg/_pytest/core.py", line 315, in execute
    res = method(**kwargs)
  File "/srv/jenkins/eggs/pytest-2.6.4-py2.7.egg/_pytest/junitxml.py", line 215, in pytest_sessionfinish
    ).unicode(indent=0))
  File "/srv/jenkins/eggs/py-1.4.25-py2.7.egg/py/_xmlgen.py", line 54, in unicode
    return u("").join(l)
MemoryError

Any suggestions?


Activity

  1. pytestbot commented on Dec 11, 2014

    @pytestbot
    ContributorAuthor

    Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):


    If you don't write the junitxml file py.test completes? How is memory usage evolving during the test run?

  2. pytestbot commented on Dec 11, 2014

    @pytestbot
    ContributorAuthor

    Original comment by Michael Howitz (BitBucket: icemac, GitHub: icemac):


    Without the option --junitxml py.test completes. During the test run up to 1,5 GB of RAM is needed. I was not yet able to see the peek when writing the XML file.

    When running the tests locally the resulting XML file is about 150 MB.

  3. pytestbot commented on Dec 12, 2014

    @pytestbot
    ContributorAuthor

    Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):


    ok, then it sounds like writing out the xml in a more iterative way should help. py._xmlgen.SimpleUnicodeVisitor might help to achieve that. Not going to try that myself soon but can review a PR.

  4. added this to the 3.0 milestone on Sep 13, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    type: bugproblem that needs to be addressed

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions