Soft assertions in Spock - RFC

[Marcin Zajączkowski]

Hi,

Recently I wanted to make a soft condition verification (collect all the
failures and display them instead of fail on the first one [1]) in the
integration test in Spock and I haven't found any satisfactory solution.
I did a quick (and successful) PoC and it seems quite easy to add it to
Spock. Before polishing the implementation and making a PR I would like
to discuss the API first.

[1] -
https://github.com/joel-costigliola/assertj-core/wiki/New-and-noteworthy#soft-assertions
- Soft assertions in AssertJ


As each statement has to be executed separately (and potential
AssertionError be caught) I see no easy way to pass all the things as
one big Closure (and later split into chunks) and every condition has to
be passed in a separate Closure:
void assert(Closure<?>... conditions)`

def softly = new SoftConditions()
(...)

then:
softly.assert(
{ assert 1 },
{ assert foo == "Bar" }
)

SoftConditions could be (probably) also wrapped with PollingContitions
and similar.
I have seen in PollingConditions that assert keyword can be omitted with
the help of @ConditionBlock, so hopefully in SoftConditions it would be
possible as well.


To make basic usage easier there would a method `void
softly(Closure<?>... conditions)` in the Specification class which
creates SoftConditions and passes assertions to it:

then:
softly(
{ assert 1 },
{ assert foo == "Bar" }
)


I predict the misuse with one big Closure and two prevent that I have
two ideas:
1. Detect the situation when only one Closure is passed and throw very
verbose exception explaining that "there is no sense to use
SoftAssertions with just one statement and in case there are many
statements different statements should be passed in a separate Closures.
2. Changes method signature to `softly(Closure<?> condition1, Closure<?>
condition2, Closure<?>... conditions)`, but it will generate only a
warning (not a compilation error) and MissingMethodException at runtime
is much less meaningful.


What do you think about SoftAssertions in Spock in general and the
proposed API?


Marcin

--
http://blog.solidsoft.info/ - Working code is not enough

[Marcin Zajączkowski]

Will be possible, but with the same restrictions as with
PollingCondtitions - only when a type is declared explicit, like:
SoftConditions softly = new SoftConditions()
(will not work also with softly method declared in Specification class -
at AST level (MethodCallExpression/VariableExpression) there is no
information about concrete class type).

Marcin

[Peter Niederwieser]

Marcin,

soft assertions can definitely be useful, and it would be great to have support for them. I think I'd favor a deeper integration that allows something like:

@SoftConditions // can alternatively go on class

def "killing me softly"() {

  expect: "weirdness"

  1 == 2 // evaluated

  2 == 3 // evaluated

}

Or perhaps:

// could additionally require annotation here

def "killing me softly"() {

  expect: "weirdness"

  1 == 2 // evaluated

  2 == 3 // not evaluated

  and: "more weirdness"

  3 == 4 // evaluated

  4 == 5 // not evaluated

}

What I like about the second proposal:

* Each individual block is still evaluated in the usual way.

* Dependent conditions are still possible (in the same block).

* There can still be just one failure per named unit, just that the unit is now a block rather than a method.

Implementing one of these proposals may take more effort than a standard API approach, but feels more in the spirit of Spock to me.

Cheers,

Peter

[Maciej Gawinecki]

What is the current state of this feature request?

[Leonard Brünings]

Soft assertions were introduced in spock 1.1 http://spockframework.org/spock/docs/1.1/all_in_one.html#_1_1_rc_1_released_2016_06_30