[SyneRBI/SIRF] adjoint tests sometimes fails for Gadgetron (#637)
Kris Thielemans
(at least) one job on linux is failing after #636 https://travis-ci.org/github/SyneRBI/SIRF/builds/679325885. Error is simply
FAIL: test2.test_main
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/travis/.local/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
self.test(*self.arg)
File "/home/travis/build/SyneRBI/SIRF/src/xGadgetron/pGadgetron/tests/test2.py", line 70, in test_main
raise AssertionError("Gadgetron operator is not adjoint")
AssertionError: Gadgetron operator is not adjoint
https://travis-ci.org/github/SyneRBI/SIRF/jobs/679325894#L31331
It'd be more useful to have some diagnostics as well then.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
Kris Thielemans
likely a border case on the numerical threshold as there's nothing special for that job as far as I can see.
Kris Thielemans
It's only that one job https://travis-ci.org/github/SyneRBI/SIRF/builds/679325850
Kris Thielemans
@AnderBiguri I'll have a look later, unless you volunteer...
Biguri
@KrisThielemans im happy to have a look.
Biguri
@KrisThielemans I cannot reproduce. Gadgetron adjoint test always return results 2 orders of magnitude lower ([3~50]e-7) than the threshold (10e-5). Is there something particular from that job that can have an influence?I can see its Xcode, which is Mac (I dont have access to Mac), but the rest of the flag combination seems OK with other builds? Its hard to know as they are not always clear in the flag list.
Kris Thielemans
@AnderBiguri the job listed in the first comment here is a linux job actually. It's more than likely that you cannot reproduce it as the other jobs are fine.
i don't see anything in the flags really. I think best thing to do is to write diagnostics to stdout when the test fails. Otherwise we're just in the dark.
Biguri
@KrisThielemans fair. The fucntion has print statements, however they do not work in the test. Other tests use sys.stderr.write() to display, but I did not want to put this inside the function. A possible alternative is making is_operator_adjoint return multiple outputs, which can be the errors and dot products. Open to other ideas.
Kris Thielemans
any reason we wouldn't have a test like this
if not is_operator_adjoint(am, verbose=true): ...
(or whatever appropriate Python syntax is)
Biguri
@KrisThielemans yes, what I mentioned in my previous comment. print does not print on the ctests
Kris Thielemans
are you sure that print doesn't print? That is: ctest pipes everything to a log file. We run ctest --output-on-failure to get it to the Travis log as well, https://github.com/SyneRBI/SIRF-SuperBuild/blob/b1ea877a68b17daaebeae08e4c92a42f150d221b/.travis.yml#L491.
By the way, I think the verbose should default to True for your test, as the output is quite small (and helpful)
Biguri
Ah @KrisThielemans my bad. Still a lot of these things elude me. Defaulting to True should be a tiny change, yet I still do not know what to do to fix this issue.
Kris Thielemans
yet I still do not know what to do to fix this issue.
no sure. let's see the diagnostics first!
Kris Thielemans
funny. the relevant job now passed.
let's wait for the OSX tests but this update is good in any case.
Kris Thielemans
Closed #637.
Kris Thielemans
This seems to have disappeared, even though the only change in #638 was making verbose output the default. As there are no job failures anymore, I'll close this.
Kris Thielemans
@AnderBiguri We have one with a failure
Testing AcquisitionModel: Iteration 1/25
Pass, with a with normalized error of 1.1071005495298965e-05 (max: 0.0001)
Testing AcquisitionModel: Iteration 2/25
Pass, with a with normalized error of 5.671866958080409e-07 (max: 0.0001)
Testing AcquisitionModel: Iteration 3/25
Pass, with a with normalized error of 4.4371515091084564e-05 (max: 0.0001)
Testing AcquisitionModel: Iteration 4/25
AcquisitionModel is not adjoint, with normalized error of 0.00011278952485286853 (max: 0.0001)
seems indeed that our threshold is just too optimistisc. Change it to 0.0002?
Kris Thielemans
Reopened #637.