Repository navigation
MemoryError when writing junitxml for large test suite #641
Description
Originally reported by: Michael Howitz (BitBucket: icemac, GitHub: icemac)
In a large project (> 3500 Tests) we get a MemoryError when writing the junit xml file.
(The host is our Jenkins CI server with 4 GB of RAM and 2 GB of swap.)
#!python
bin/py.test --nosugar --junitxml=parts/tests.xml --cov-report html --cov-report xml --cov-report term --cov src
============================= test session starts ==============================
platform linux2 -- Python 2.7.6 -- py-1.4.25 -- pytest-2.6.4
plugins: gocept.pytestlayer, cache, cov, sugar, remove-stale-bytecode
collected 3643 items
… Many, many tests which ran fine …
Traceback (most recent call last):
File "bin/py.test", line 308, in <module>
sys.exit(pytest.main())
File "/srv/jenkins/eggs/pytest-2.6.4-py2.7.egg/_pytest/config.py", line 41, in main
return config.hook.pytest_cmdline_main(config=config)
File "/srv/jenkins/eggs/pytest-2.6.4-py2.7.egg/_pytest/core.py", line 413, in __call__
return self._docall(methods, kwargs)
File "/srv/jenkins/eggs/pytest-2.6.4-py2.7.egg/_pytest/core.py", line 424, in _docall
res = mc.execute()
File "/srv/jenkins/eggs/pytest-2.6.4-py2.7.egg/_pytest/core.py", line 315, in execute
res = method(**kwargs)
File "/srv/jenkins/eggs/pytest-2.6.4-py2.7.egg/_pytest/main.py", line 116, in pytest_cmdline_main
return wrap_session(config, _main)
File "/srv/jenkins/eggs/pytest-2.6.4-py2.7.egg/_pytest/main.py", line 109, in wrap_session
exitstatus=session.exitstatus)
File "/srv/jenkins/eggs/pytest-2.6.4-py2.7.egg/_pytest/core.py", line 413, in __call__
return self._docall(methods, kwargs)
File "/srv/jenkins/eggs/pytest-2.6.4-py2.7.egg/_pytest/core.py", line 424, in _docall
res = mc.execute()
File "/srv/jenkins/eggs/pytest-2.6.4-py2.7.egg/_pytest/core.py", line 315, in execute
res = method(**kwargs)
File "/srv/jenkins/eggs/pytest-2.6.4-py2.7.egg/_pytest/terminal.py", line 349, in pytest_sessionfinish
__multicall__.execute()
File "/srv/jenkins/eggs/pytest-2.6.4-py2.7.egg/_pytest/core.py", line 315, in execute
res = method(**kwargs)
File "/srv/jenkins/eggs/pytest-2.6.4-py2.7.egg/_pytest/junitxml.py", line 215, in pytest_sessionfinish
).unicode(indent=0))
File "/srv/jenkins/eggs/py-1.4.25-py2.7.egg/py/_xmlgen.py", line 54, in unicode
return u("").join(l)
MemoryError
Any suggestions?
Activity
pytestbot commentedon Dec 11, 2014
Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):
If you don't write the junitxml file py.test completes? How is memory usage evolving during the test run?
pytestbot commentedon Dec 11, 2014
Original comment by Michael Howitz (BitBucket: icemac, GitHub: icemac):
Without the option --junitxml py.test completes. During the test run up to 1,5 GB of RAM is needed. I was not yet able to see the peek when writing the XML file.
When running the tests locally the resulting XML file is about 150 MB.
pytestbot commentedon Dec 12, 2014
Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):
ok, then it sounds like writing out the xml in a more iterative way should help.
py._xmlgen.SimpleUnicodeVisitormight help to achieve that. Not going to try that myself soon but can review a PR.