Repository navigation
Discrepancy between test running and cleaning/printing failures #5872
Description
The run-tests script puts the test output in the current working directory but the --print-failure and --clean-failure arguments look for test output in the tests/ directory. As a result the two only work together if the working directory for running the tests is effectively the tests/ directory. I found this out when running tests by calling run-tests by full path from wherever the cwd is: it would never print failures.
It would be better if the script would stick to one thing: either read test output from the cwd as well, or else produce it in the tests/ directory (which would be my favorite: not a fan of relying on cwd except for commandline usage). The latter could be extended by allowing to specify the output directory as an argument. Ideas of which way to go?
Activity
dlech commentedon Apr 6, 2020
I would be in favor of having a default output directory that is excluded from git, similar to the
builddirectory for compiler output, e.g.tests/results/ortests/output/.dpgeorge commentedon Apr 7, 2020
+1
+1
stinos commentedon Apr 7, 2020
Note there's a similar problem with the default values of MICROPY_MICROPYTHON and MPYCROSS in run-tests: they are relative to the working directory so if you don't invoke run-tests from within it's own directory you have to specify them because the defaults are not correct anymore.
stinos commentedon Apr 7, 2020
I started on this but turns out it is actually quite involved to the point I'm not sure if it's even worth it: there's really a lot of things which expect they're ran relative to the tests directory. Get a glimpse here: https://github.com/stinos/micropython/tree/run-test-abspath (note: still doesn't work). Maybe there's an easier way? But even if it would be fixed, all other test runners suffer the same problem. Perhaps just making it required for the test runners to run from the tests directory and raising an error if not would be better..
edit Or implement it so that e.g. the problems described here (print/clean failures, specify output directory, MICROPY_MICROPYTHON etc being wrong) are fixed, but running builtin tests (i.e. anything which requires skip_tests logic and is relative to the tests directory) requires being ran from the tests directory.
tests/run-tests: Use absolute paths where possible.
Don't update status LED color on brightness change
tests/run-tests: Use absolute paths where possible.
tests/run-tests: Use absolute paths where possible.
tests/run-tests: Use absolute paths where possible.