Description
Motivation. CI runs the OpenCL tests on one GPU. A kernel bug that this device happens to tolerate stays invisible. Example: #3429, where the cholesky_decompose kernel returns a wrong factor on an AMD GPU because its barriers fence local memory only.
Oclgrind is an OpenCL device simulator. It needs no GPU and checks what real devices do not:
- data races between work items, including a barrier with the wrong fence flag (
--data-races),
- reads and writes outside a buffer (always on),
- use of uninitialized values (
--uninitialized).
It runs our test binaries unchanged:
oclgrind --data-races --log races.log ./test/unit/math/opencl/cholesky_decompose_test
On develop this reports read-write races in cholesky_decompose, already for the 3×3 case, and nothing once the barriers are fixed. A first local pass also reports races in tridiagonalization_householder.
Suggested steps.
- Add
oclgrind to the CI image.
- Add a CPU-only stage that builds
test/unit/math/opencl with STAN_OPENCL=true and runs each test binary through oclgrind --data-races --log <file>. runTests.py already prefixes MPI tests with mpirun; the same mechanism can prefix oclgrind.
- Fail the stage when a log is not empty.
- Start it as non-blocking and make it required once the existing reports are fixed.
Description
Motivation. CI runs the OpenCL tests on one GPU. A kernel bug that this device happens to tolerate stays invisible. Example: #3429, where the
cholesky_decomposekernel returns a wrong factor on an AMD GPU because its barriers fence local memory only.Oclgrind is an OpenCL device simulator. It needs no GPU and checks what real devices do not:
--data-races),--uninitialized).It runs our test binaries unchanged:
On develop this reports read-write races in
cholesky_decompose, already for the 3×3 case, and nothing once the barriers are fixed. A first local pass also reports races intridiagonalization_householder.Suggested steps.
oclgrindto the CI image.test/unit/math/openclwithSTAN_OPENCL=trueand runs each test binary throughoclgrind --data-races --log <file>.runTests.pyalready prefixes MPI tests withmpirun; the same mechanism can prefixoclgrind.