MathMixMatFun.add and MathMixMatFun.subtract tests seem to fail randomly?

Description

After #1424 we've had a couple PRs which look fine fail the MPI unit tests (but pass the GPU branch?)

Ex:

Not sure what's going on?

If we can find a quick patch for #1424 or understand why that's happening then cool and good otherwise let's do the revert and revert-revert PR

Current Version:

v3.0.0

Also the tests pass locally for me. Not sure if they are setting something off in the MPI that means we need a flag somewhere

I don't understand what's being proposed here. Did you want to revert #1424 for some reason? I couldn't tell from #1445 what's going wrong.

I'm looking into what's going on right now. Not sure if it's some weird interaction between something and MPI since the tests on the GPU node pass fine

@rok-cesnovar

Alright idk y but breaking them up into individual tests fixed it for one of the PRs. Gonna close this. @rok-cesnovar if you have time / think the matrix_cl<var> PR can get in tmrw we can just leave the fix in there otherwise I can slam out a quick PR that just has those fixed up

There are still a couple residual MPI-based tests that I didn't remove that do some setup. One is: test/unit/math/rev/mat/fun/multiply_test.cpp Do the other tests need similar config? I don't know how tests/MPI are calling everything from the outside.

…

On Nov 11, 2019, at 10:05 PM, Steve Bronder ***@***.***> wrote: I'm looking into what's going on right now. Not sure if it's some weird interaction between something and MPI since the tests on the GPU node pass fine — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub, or unsubscribe.

I'm not sure either, something something curing the symptom and not the wound but we can just deal with those if they pop up as issues

Develop has just now failed with a similar error (see https://jenkins.mc-stan.org/blue/organizations/jenkins/Math%20Pipeline/detail/develop/404/pipeline/142)

I will look into this tonight.

The failure is this:

[ RUN      ] MathMixMatFun.add_vec_mat
./test/unit/math/expect_near_rel.hpp:39: Failure
The difference between x1 and x2 is 0.0010334642265417964, which exceeds tol, where
x1 evaluates to -0.0010334642265417964,
x2 evaluates to 0, and
tol evaluates to 0.001.
expect_near_rel_finite(-0.0010334642265417964, 0, absolute tolerance = 0.001); absolute diff = 0.0010334642265417964    in: expect_near_rel; require items x1(i) = x2(i): hessian() Hessian
...

First, the failure's within acceptable bounds. I've had to lower precision on a lot of finite diff Hessian tests to 1e-2. So if I'd gotten this locally, I'd have just lowered tolerances.

The test (see below) doesn't have any randomness. That means arithmetic is changing on different runs. The test that's failing here is this one:

TEST(MathMixMatFun, add_vec_mat) {
  auto f = [](const auto& x, const auto& y) { return stan::math::add(x, y); };

  Eigen::VectorXd v5(5);
  v5 << 1, 2, 3, 4, 5;
  Eigen::VectorXd v5b(5);
  v5 << 2, 3, 4, 5, 6;
  Eigen::VectorXd rv5(5);
  v5 << 1, 2, 3, 4, 5;
  Eigen::VectorXd rv5b(5);
  v5 << 2, 3, 4, 5, 6;
  stan::test::expect_ad(f, v5, v5b);
  stan::test::expect_ad(f, rv5, rv5b);
}

The expect_ad code isn't using any kind of multi-threading or introducing any statics anywhere. I suspect it may be running with different compiler configs or different hardware config, both of which can change details of arithmetic precision.

The easiest way to fix these is to include:

stan::test::ad_tolerances tols;   // sets up with defaults
tols.hessian_hessian_ = 1e-2;  // increase acceptable Hessian finite diff error for rev
tols.hessian_fvar_hessian_ = 1e-2;  // " for fwd
...
expect_ad(tols, ...);

I added the forward mode because this report also shows up. The key is the hessian_fvar() Hessian line. That indicates that a call to the functional hessian_fvar() failed when computing the Hessian.

expect_near_rel_finite(-0.0010334642265417964, 0, absolute tolerance = 0.001); absolute diff = 0.0010334642265417964    in: expect_near_rel; require items x1(i) = x2(i): hessian_fvar() Hessian

The errors then follow that nomenclature in lower case to make it easy to update the tests without looking at the tolerances code (only took a few painful attempts to line up messages and constants before I did that).

Were other tests failing in different situations? It might make sense to just lower Hessian tolerance to 1e-2 globally, or even 5e-3, because we get lots of failures right around 1e-3.

@bob-carpenter

There were 7 merges to develop and no tests failed here so I am closing this. Thanks to @bob-carpenter for providing the solution.

Develop did fail once with /usr/local/opt/llvm@6/bin/clang++: /usr/local/opt/llvm@6/bin/clang++: cannot execute binary file but I am assuming that is hopefully one time Jenkins issue.

It would be nice to get to the bottom of why the deterministic arithmetic varies from run to run. There's nothing probabilistic in these tests. Globally setting .005 would be OK, too.

…

On Dec 3, 2019, at 11:12 AM, Marco Colombo ***@***.***> wrote: This has hit again: https://jenkins.mc-stan.org/blue/organizations/jenkins/Math%20Pipeline/detail/PR-1477/5/pipeline Strangely it still reports 0.001 as tolerance, although it was changed to 0.005 in #1469: ./test/unit/math/expect_near_rel.hpp:39: Failure The difference between x1 and x2 is 0.0010334642265417964, which exceeds tol, where x1 evaluates to 0.0010334642265417964, x2 evaluates to 0, and tol evaluates to 0.001. expect_near_rel_finite(0.0010334642265417964, 0, absolute tolerance = 0.001); absolute diff = 0.0010334642265417964 in: expect_near_rel; require items x1(i) = x2(i): hessian() Hessian — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

I would start from only setting the tolerance for the subtract_vec_mat test at 0.005. Up to now I think only the add and subtract tests have been playing funny, so perhaps this is enough.

On the other hand it would be interesting to see if it's possible to track down from the Jenkys or other backend logs if there is a unique instance of the Windows machines that is failing, and if so understand what is causing the failures (different compiler/hardware configurations?).

@serban-nicusor-toptal

@serban-nicusor-toptal would you be able to tell me if these two runs of "Windows Headers & Unit" were run on different Windows machines?

I know you told me once how to check this, but I forgot, sorry.

Hopefully its different machines, that would make this somewhat more expected behavior.

I tried looking at the pipeline.log in Artifacts but not sure what was run where.

Both ran on the same machine

Running on gelman-group-win2 in C:\Jenkins2\workspace\Math_Pipeline_PR-1351

To find on what machine they run, go to Jenkins PR page in this case, click on run #10 then go to Console Output -> Full output now CTRL + F for Running on

Thank you!

In that case it is something that needs further investigating. Will make a PR to fix the issue on develop and check if can reproduce this on Windows.

Hmm. If it's not the OS, is it possible it's a different compiler or different compiler settings? The problem is that deterministic tests are providing different behaviors at different times. Could this be some kind of multi-threading or multi-process issue?

…

On Dec 4, 2019, at 3:25 AM, Rok Češnovar ***@***.***> wrote: Thank you! In that case it is something that needs further investigating. Will make a PR to fix the issue on develop and check if can reproduce this on Windows. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

I think we will finally squashed this one with 744fea1

Turns out that we kept initializing the same variable while the rest were left uninitialized.

I ran valgrind on the test before the fix, and it did complain about uninitialised values. It's worth keeping it in mind for the next time we see some unexpected behaviour such as this. When dust settles we should try reverting #1469 and #1483.

@mcol

I am going to revert back to the default tolerance, which will essentially revert those PRs. Good call @mcol.

Navigation Menu

Repository navigation

MathMixMatFun.add and MathMixMatFun.subtract tests seem to fail randomly? #1453

Description

Current Version:

3 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

MathMixMatFun.add and MathMixMatFun.subtract tests seem to fail randomly? #1453

Description

Description

Current Version:

Activity

SteveBronder commented on Nov 11, 2019

bob-carpenter commented on Nov 12, 2019

SteveBronder commented on Nov 12, 2019

SteveBronder commented on Nov 12, 2019

bob-carpenter commented on Nov 13, 2019

SteveBronder commented on Nov 13, 2019

rok-cesnovar commented on Nov 28, 2019

bob-carpenter commented on Nov 28, 2019

rok-cesnovar commented on Nov 30, 2019

3 remaining items

bob-carpenter commented on Dec 3, 2019

mcol commented on Dec 3, 2019

rok-cesnovar commented on Dec 4, 2019

rok-cesnovar commented on Dec 4, 2019

serban-nicusor-toptal commented on Dec 4, 2019

rok-cesnovar commented on Dec 4, 2019

bob-carpenter commented on Dec 5, 2019

rok-cesnovar commented on Jan 5, 2020

mcol commented on Jan 5, 2020

rok-cesnovar commented on Jan 5, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions