Retrying failed tests / Statement.evaluate() never seems to throw an exception

[PascalSchumacher]

Hello everybody,

I tried to use a custom TestRule to get spock to retry failed tests. The implementation is copied from http://stackoverflow.com/questions/8295100/how-to-re-run-failed-junit-tests-immediately and works with plain junit but not with spock. Is there a way to get this working? Is there another way to retry failed tests?

Thanks a lot,
Pascal

[PascalSchumacher]

Using a deprecated custom MethodRule as shown in https://gist.github.com/abyx/897229 works, but the setup and cleanup closures are skipped. I guess the best way to solve this is a custom spock extension? Sadly I could not figure out how to rerun features with a spock extension from the only explanation I found (http://ldaley.com/post/971946675/annotation-driven-extensions-with-spock). Any hints how to achieve this?

Thanks,
Pascal

[Peter Niederwieser]

Pascal,

this topic has been discussed before, both here and on Stack Overflow.

Spock's rule execution is fully compatible with JUnit's, except for one difference: In Spock, an exception occurring inside a test method is caught and handled before control is passed back to a TestRule, whereas in JUnit it isn't. This isn't something that's easy to change. MethodRule doesn't have this problem, and isn't deprecated anymore in latest JUnit. But as you said, it differs from TestRule in that it only wraps around the test method, but not around setup and cleanup.

In general, retrying with a TestRule doesn't seem ideal. If nothing else, the same test instance will be used each time (also in JUnit). If you can live with these kinds of limitations, you could implement a MethodRule and use setup/cleanup blocks instead of setup/cleanup methods. I can think of a few other potential (Spock-specific) solutions, but they aren't straightforward to explain or implement. Eventually, I'd like to enhance the extension SPI in a way that makes it easy to express repetition/retry.

Hope this helps. What's your motivation for retrying?

Cheers,

Peter

--
You received this message because you are subscribed to the Google Groups "Spock Framework - User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spockframewor...@googlegroups.com.
To post to this group, send email to spockfr...@googlegroups.com.
Visit this group at http://groups.google.com/group/spockframework?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[PascalSchumacher]

Hello Peter,

Am Donnerstag, 30. Mai 2013 23:59:51 UTC+2 schrieb Peter Niederwieser:

this topic has been discussed before, both here and on Stack Overflow.

 sorry for the duplication. I did not find any discussion other that a post on stackoverflow which suggests use of the method rule.

Spock's rule execution is fully compatible with JUnit's, except for one difference: In Spock, an exception occurring inside a test method is caught and handled before control is passed back to a TestRule, whereas in JUnit it isn't. This isn't something that's easy to change. MethodRule doesn't have this problem, and isn't deprecated anymore in latest JUnit. But as you said, it differs from TestRule in that it only wraps around the test method, but not around setup and cleanup.

In general, retrying with a TestRule doesn't seem ideal. If nothing else, the same test instance will be used each time (also in JUnit). If you can live with these kinds of limitations, you could implement a MethodRule and use setup/cleanup blocks instead of setup/cleanup methods. I can think of a few other potential (Spock-specific) solutions, but they aren't straightforward to explain or implement. Eventually, I'd like to enhance the extension SPI in a way that makes it easy to express repetition/retry.

O.k. thanks for the information. :)

Hope this helps. What's your motivation for retrying?

 
We use spock for end-to-end testing gray-box testing of our system. Most tests check the database or call webservices of external systems. Sometimes the response is a bit slow so that one or two out of over a hundred  tests fail. As rerunning all tests takes a lot of time I just want to repeat the failed ones, if I can a way to do this easily. In the long run I will try to adjust the scaffolding to make the test less brittle, but this takes time I do not have at the moment.

Thanks again for the input,
Pascal

[Levon Saldamli]

Hello Peter,

I'm trying to implement retry functionality that also can catch errors in setup methods, but I can't find any extension approach that would work. I'm working on spock-retry which adds an interceptor to the feature method, catches exceptions, and reruns proceed() a number of times. But this only works on feature methods, not on e.g. iteration interceptors or feature interceptors. Is there nowadays a way to implement a retry functionality which can run setup/feature/cleanup and catch errors and do retries?

/Levon