Is it possible to see at which breakpoint a test failed?

[Paolo Bizzarri]

Hi,

I am starting to use Thread Weaver - very very cool tool.

I would like to see in a test something like this:

- I execute a test

- the test performs the various interleavings

- one of them fails the assertion

- the test tells me where was the breakpoint when the test fails.

Is that possible or hard to achieve?

Regards

Paolo

[Alasdair Mackintosh]

You can set a debug option in the test runner.

public void testFoo() {
AnnotatedTestRunner runner = new AnnotatedTestRunner();
runner.setDebug(true); // Turns on internal debugging
...
runner.runTests(...)

This will print out some details of what interleavings are going on,
and where it's breaking. You should be able to use that to work out
what's going on.

Alasdair

[Paolo Bizzarri]

Debug option was fine and nice.

I am trying to figure out if there is an API that I can access from a test, to report the line where the breakpoint was set when we had a failure.

I would love something along the lines:

"Failed assertion XXX: method yyyy had breakpoint at line ZZZ".

Regards

Paolo

[Alasdair Mackintosh]

On Tue, Nov 25, 2014 at 1:46 AM, Paolo Bizzarri <pib...@gmail.com> wrote:
> Debug option was fine and nice.
>
> I am trying to figure out if there is an API that I can access from a test,
> to report the line where the breakpoint was set when we had a failure.
>
> I would love something along the lines:
>
> "Failed assertion XXX: method yyyy had breakpoint at line ZZZ".

Hi Paolo,

I'm afraid there isn't anything besides the debug output.

I would hope that the debug output would be enough to diagnose any
problem. I would assume that when a test failed, the developer is
going to have to look at the diagnostic output to see what went wrong.
We could provide an API that would get hold of that diagnostic output,
but is there anything to do with it besides print it out?

Also, the test framework doesn't know when something went wrong. If
you look at the example code in the Tutorial (See Tutorial.html in the
docs directory) it is only when the assertion failed in the
@ThreadedAfter method that we know there was a problem. The framework
itslef doesn't know that there was a race condition inside the code
that led to a test failure.

I suppose we could always record the debug output, and then provide an
API for accessing that if something goes wrong...

Alasdair

[Paolo Bizzarri]

Hi Alasdair, 

thanks for your reply. Please find my comments below.

Il giorno giovedì 27 novembre 2014 20:51:23 UTC+1, alasdair ha scritto:

On Tue, Nov 25, 2014 at 1:46 AM, Paolo Bizzarri <pib...@gmail.com> wrote:
> Debug option was fine and nice.
>
> I am trying to figure out if there is an API that I can access from a test,
> to report the line where the breakpoint was set when we had a failure.
>
> I would love something along the lines:
>
> "Failed assertion XXX: method yyyy had breakpoint at line ZZZ".

Hi Paolo,

I'm afraid there isn't anything besides the debug output.

 

I would hope that the debug output would be enough to diagnose any
problem. I would assume that when a test failed, the developer is
going to have to look at the diagnostic output to see what went wrong.
We could provide an API that would get hold of that diagnostic output,
but is there anything to do with it besides print it out?

The problem is that this is too much. 

Suppose that I have a problem and that my assertions have failed in the ThreadedAfter.

The most important information - for me - is to know exactly at which iteration the failure has happened. and when the breakpoint was at a certain line.

 I do not want to go through lots of debugging output just to know that failure happened at third iteration when breakpoint was at line 28.

Also, the test framework doesn't know when something went wrong. If
you look at the example code in the Tutorial (See Tutorial.html in the
docs directory) it is only when the assertion failed in the
@ThreadedAfter method that we know there was a problem. The framework
itslef doesn't know that there was a race condition inside the code
that led to a test failure.

I suppose we could always record the debug output, and then provide an
API for accessing that if something goes wrong...

No I do not think it is necessary. We can just memorize the number of iteration for each test execution and the line for the breakpoint. Then, in case of failure we can print this directly.

Something like

assert failed: 0 != 1 at iteration 5 when breakpoint was at line 257.

Regards.

Paolo

 

Alasdair

[Alasdair Mackintosh]

Hi Paolo,

Do you have a concrete example of a test failure that was hard to
debug just using the debugging output?

Although I can see why the mechanism that you describe might be
useful, it seems like a fair amount of work for something that I think
is likely to be used infrequently.

Alasdair

> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "Thread Weaver" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to thread-weave...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.