Better matcher error messages

[Brian Farrell]

I really like REST assured, but I've noticed that the error messages when a matcher fails sometimes aren't as good as they could be.

For example:

import static com.jayway.restassured.RestAssured.given;
import static org.hamcrest.Matchers.containsInAnyOrder;

import java.util.Collection;

import org.junit.Test;

import com.jayway.restassured.builder.RequestSpecBuilder;
import com.jayway.restassured.specification.RequestSpecification;

public class RATest {
   
    RequestSpecification reqSpec = new RequestSpecBuilder()
            .setBaseUri("http://httpbin.org")
            .setPort(80)
            .addParam("arg1", "val1")
            .addParam("arg2", "val2")
            .addParam("arg3", "val3")
            .build();
   
    @Test
    public void raAssert() {
        given(reqSpec)
        .when() .get("get")
        .then() .body("args.values()", containsInAnyOrder("val1", "valtwo", "val3"));
    }
   
    @Test
    public void junitAssert() {
        Collection<String> argValue =
        given(reqSpec)
        .when() .get("get")
        .then() .extract().path("args.values()");
       
        org.hamcrest.MatcherAssert.assertThat(argValue, containsInAnyOrder("val1", "valtwo", "val3"));
    }
}

The first test fails with the following exception:

java.lang.AssertionError: JSON path args.values() doesn't match.
Expected: iterable over ["val1", "valtwo", "val3"] in any order
  Actual: [val3, val2, val1]

and the second test fails with this exception:

java.lang.AssertionError:
Expected: iterable over ["val1", "valtwo", "val3"] in any order
     but: Not matched: "val2"

The second message is much better and points directly to the issue- imagine if the list was 100 items rather than just three.

There's nothing fancy in Hamcrest's MatcherAssert:

     

public static <T> void assertThat(String reason, T actual, Matcher<? super T> matcher) {

if (!matcher.matches(actual)) {

Description description = new StringDescription();

description.appendText(reason)

.appendText("\nExpected: ")

.appendDescriptionOf(matcher)

.appendText("\n     but: ");

matcher.describeMismatch(actual, description);

throw new AssertionError(description.toString());

}

}

Maybe you can reproduce that logic or call it directly?  I think there are probably a lot of good matcher error messages that we're missing out on.

Thanks,
-Brian

[Johan Haleby]

Interesting findings. I cannot use the code you provide as it is since REST Assured stacks up multiple error messages and displays them all at once when using "except". Unfortunately "describeMismatch" is a void method. If you like to experiment with this and help out please try to make changes in the com.jayway.restassured.assertion.BodyMatcher class. This is where the error messages are generated today. 

--
You received this message because you are subscribed to the Google Groups "REST assured" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rest-assured...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Brian Farrell]

I made a couple of minor changes to BodyMatcher.groovy (attached) and now I'm getting the error messages I was hoping for:

java.lang.AssertionError: JSON path args.values() doesn't match.

Expected: iterable over ["val1", "valtwo", "val3"] in any order

     but: Not matched: "val2"

[Johan Haleby]

Thanks for helping out. However I'm not convinced that the new error messages are always for the better. Before the change it looked like this:

JSON path lotto.lottoId doesn't match.

Expected: a value less than <2>

  Actual: 5

JSON path lotto.winning-numbers doesn't match.

Expected: a collection containing <21>

  Actual: [2, 45, 34, 23, 7, 5, 3]

But now it looks like this:

JSON path lotto.lottoId doesn't match.

Expected: a value less than <2>

  Actual: <5> was greater than <2>

JSON path lotto.winning-numbers doesn't match.

Expected: a collection containing <21>

  Actual: was <2>, was <45>, was <34>, was <23>, was <7>, was <5>, was <3>>

I don't like the was <2>, was <45>, was <34>, ... part.

Regards,

/Johan

[Brian Farrell]

I see your point- the second error message is not an improvement.  

One benefit is that you have more control over the message this way, which is good because the most useful error message is going to vary depending on what you're validating, how large it is, etc.

If you prefer to just see the original object, you can do something like this:

class WithActualObject<T> extends BaseMatcher<T> {

private final Matcher<T> matcher;

public WithActualObject(Matcher<T> matcher) {

super();

this.matcher = matcher;

}

public boolean matches(Object item) {

return matcher.matches(item);

}

public void describeTo(Description description) {

matcher.describeTo(description);

}

public static <T> Matcher<T> withActualObject(Matcher<T> matcher) {

            return new WithActualObject<T>(matcher);

        }

}

Then  body("lotto.winning-numbers", withActualObject(hasItem(21))) gives you an error message that just shows the original object:

JSON path lotto.winning-numbers doesn't match.

Expected: a collection containing <21>

     but: was <[2, 45, 34, 23, 7, 5, 3]>

Obviously this change would fairly significantly change error messages in existing tests, and maybe most users would prefer things as they are today, I'm not sure.  I'd also be happy with a way to configure things one way or the other.

[Johan Haleby]

I've made this configurable now:

given().config(RestAssured.config().matcherConfig(new MatcherConfig(HAMCREST))). ..

So if you specify HAMCREST as ErrorDescriptionType you'll get the error messages you're after. You can configure this for all requests by using a static configuration:

RestAssured.config = config().config().matcherConfig(new MatcherConfig(HAMCREST));

By default REST_ASSURED ErrorDescriptionType is used. 

Please try this out and give me feedback by depending on version 2.3.4-SNAPSHOT after having added the following repo:

<repositories>

        <repository>

            <id>sonatype</id>

            <url>https://oss.sonatype.org/content/repositories/snapshots/</url>

            <snapshots />

        </repository>

</repositories>

/Johan

[Brian Farrell]

Perfect.  Works for me.

One thing I noticed while trying this out that wasn't completely obvious at first: if I create a RequestSpecification, then modify the static RestAssured.config, then make a request with that specification, my static changes are ignored for that request.  After I realized what was going on, I understood why it might work that way.  But originally I expected that static changes would be picked up everywhere automatically.

In any case, I appreciate your quick responses here.  Thanks!

-Brian

[Johan Haleby]

Do you think MatcherConfig and ErrorDescriptionType are good names or do you have other suggestions?

Regarding the specifications I agree that sometimes they can be a bit tricky to get your head around. Especially since they are immutable (which I kind of regret now).

[Brian Farrell]

Yeah I think those are good choices for names.  At least they make sense to me.  :)