[Mockito User Survey] How should the perfect Mockito JUnit5 API look like?

[Christian Schwarz]

Currently the implementation "Mockito for JUnit5" is in progress see https://github.com/mockito/mockito/pull/1221. To provide the best possible API I want to ask all of you Mockito users: How should the perfect Mockito JUnit5 API look like?

Currently we have different proposals that can be discussed:


Proposal [1]
by Tim van der Lippe

1. Nested classes can specify a new runner which overrides the parent runner.
2. The @ExtendWith specifies the strictness as well and this strictness will be used for all tests. E.g. @ExtendWith(MockitoExtension.Silent.class) or @ExtendWith(MockitoExtension.Strict.class). @ExtendWith(MockitoExtension.class) is equivalent to the strict variant.
3. The user can opt-in to a different strictness per method.

The algorithm to retrieve the strictness is then:

1. If Method has annotation, use that strictness
2. Else use strictness specified by runner

As every runner already defines it strictness level, we don't need to fallback to the DEFAULT_STRICTNESS.

@ExtendWith(MockitoExtension.Strict.class)
class Test {
   @Mock Foo foo;

   @Test //strict-stubbing inherited from Test
   void test1(){
      foo.bar();
   }

   @ExtendWith(MockitoExtension.Silent.class)
   class NestedTest{
  
       @Test //silent inherited from NestedClass
       void test2(){
         foo.bar();
       }  

       @Test
       @Strictness(WARN) //warn defined on method level
       void test2(){
         foo.bar();
        
      }  
   }
}

Proposal [2]

by Christian Schwarz inspired by Tsuyoshi Murakami

Use the annotation  @WithMockito to init mocks, spy's and so on like Junit4 @RunWith(MockitoJunitRunner)


Uses the annotation @Strictness( ..) to set /override the strictness for the test-root-class, nested-test-classes and test-methods.


@WithMockito //init mocks, spys and captors, strictness is per default WARN
@Strictness(STRICT_STUBS) //optional annotation to defined strictness
class Test {
   @Mock Foo foo;

   @Test //strict-stubbing inherited from Test
   void test1(){
      foo.bar();
   }

   @Strictness(SILENT) //override the stricness from Test
   class NestedTest{
  
       @Test //silent inherited from NestedClass
       void test2(){
         foo.bar();
       }  

       @Test
       @Strictness(WARN) //warn defined on method level
       void test2(){
         foo.bar();
        
      }  
   }
}

 

[Tim van der Lippe]

To clarify more about my proposal, given our usage with JUnit4 (http://static.javadoc.io/org.mockito/mockito-core/2.11.0/org/mockito/junit/MockitoJUnitRunner.html) I would like the API to be very similar. As such, I would say: just replace `@RunWith` with `@ExtendWith`.

In other words, while we had this with JUnit4:

```java

@RunWith(MockitoJUnitRunner.class)

@RunWith(MockitoJUnitRunner.Silent.class)

@RunWith(MockitoJUnitRunner.Strict.class)

```

And I would like this in JUnit5:

```java

@ExtendWith(MockitoJUnitRunner.class)

@ExtendWith(MockitoJUnitRunner.Silent.class)

@ExtendWith(MockitoJUnitRunner.Strict.class)

```

In this case, the strictness level is incorporated in the runner. This removes the need to specify both `@ExtendWith` and `@Strictness` level.

Op do 9 nov. 2017 om 11:12 schreef Christian Schwarz <chris...@gmail.com>:

--
You received this message because you are subscribed to the Google Groups "mockito" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mockito+u...@googlegroups.com.
To post to this group, send email to moc...@googlegroups.com.
Visit this group at https://groups.google.com/group/mockito.
To view this discussion on the web visit https://groups.google.com/d/msgid/mockito/5d8a5401-692e-41c4-866e-7e53243253d0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Christian Schwarz]

>To clarify more about my proposal, given our usage with JUnit4 (http://static.javadoc.io/org.mockito/mockito-core/2.11.0/org/mockito/junit/MockitoJUnitRunner.html) I would like the API to be very similar.

Why do we need to be similar to JUnit5? Junit5 allows to provide more user friendly / less boilerplate test code.

Defining a strictness-level is from my working experience is a rarely used feature, the most teams tend to move fast forward and use the default strictness. The API of Mockito should provide that with minimal coding effort. This can be achieved by a plain annotation like @WithMockito which implies a default stricness. Setting the stricness-level at the extension annotation level like @WithMockito(strictness=STRICT_STUBS)is suboptimal cause it would introduce a second concept to define the strictness parallel to @Strictness.

Advantages of proposal [2]:
 - we have a single concept to define the strictness at any test level
 - good readability / minimal boilerplate,  the majority of mockito users need only one annotation without additional parameters
-  easy to extend API, new configuration options can  be added with additional annotations or parameters of the @WithMockito annotation

[Tim van der Lippe]

`@ExtendWith(MockitoJUnitRunner.class)` does exactly this right? It has a default strictness, without specifying default strictness. I find `WithMockito` ambigous and non-JUnit5-like.

Op ma 13 nov. 2017 om 14:02 schreef Christian Schwarz <chris...@gmail.com>:

--
You received this message because you are subscribed to the Google Groups "mockito" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mockito+u...@googlegroups.com.
To post to this group, send email to moc...@googlegroups.com.
Visit this group at https://groups.google.com/group/mockito.

To view this discussion on the web visit https://groups.google.com/d/msgid/mockito/6d81ba34-ea0e-4b75-b20b-e2ca4223011b%40googlegroups.com.

[Christian Schwarz]

Hi Tim,

`@ExtendWith(MockitoJUnitRunner.class)` does exactly this right? It has a default strictness, without specifying default strictness.

Thats right!


 

I find `WithMockito` ambigous and non-JUnit5-like.

In the end it is matter of taste, everyone has its own preference. Hopefully some mockito users will post there opinion here.

I would like to hear the opinion of the mockito-core members, too. Once the API is release there is no way back.

Maybe we should leave the strictness definition out in the first version. We could add such API once we know in which

direction mockito-junit5 should evolve.

[Malte Finsterwalder]

I rather like the default JUnit 5 Way best.

It's easier to understand for people that know JUnit 5 than a custom annotation like "@WithMockito" and "@Strictness".

When I come across an "@ExtendWith(...)"-Annotation I immediately know how to read it.

@WithMockito does not tell me anything, unless I already know Mockito.

I don't see any benefit in the custom annotations.

So I would clearly vote for:

@ExtendWith(MockitoJUnitRunner.class)

@ExtendWith(MockitoJUnitRunner.Silent.class)

@ExtendWith(MockitoJUnitRunner.Strict.class)

Greetings,

   Malte

--
You received this message because you are subscribed to the Google Groups "mockito" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mockito+unsubscribe@googlegroups.com.

To post to this group, send email to moc...@googlegroups.com.
Visit this group at https://groups.google.com/group/mockito.

To view this discussion on the web visit https://groups.google.com/d/msgid/mockito/7b568de6-956e-4d4e-8696-206a2c38775d%40googlegroups.com.

[Zagretdinov Arthur]

Let me, put in my two penny worth. As a user who migrate from jUnit 4 to 5 I would prefer the first proposal by Tim van der Lippe. Find and replace will work. 

But if I was a developer who write a new code I would prefer to avoid having something like `@ExtendWith(MockitoJUnitRunner.class)` in code, because `*Runner` it is something that remand me junit4 and hell with runners.  So I would vote for the 2nd proposal. But I don’t like name name `@WithMockito`. Maybe something like ‘@MockitoTest`, `@SilentMockitoTest` and `@StrictMockitoTest`. 

The second way provides easy for extending meaning of annotation and extending annotation by 3rd part libraries (like PowerMock for example :) ) 

So I vote for 2nd approach by Christian Schwarz. 

Best regrads,

Arthur Zagretdinov

To unsubscribe from this group and stop receiving emails from it, send an email to mockito+u...@googlegroups.com.

To post to this group, send email to moc...@googlegroups.com.
Visit this group at https://groups.google.com/group/mockito.

To view this discussion on the web visit https://groups.google.com/d/msgid/mockito/CALS12-Pf4X-V%3DL%3D-GQ_Cn8Lwg%3DH0fCAFszQ5eY9oTd-6i50QLw%40mail.gmail.com.

[Tim van der Lippe]

Oh just to make it complete: it should have been `@ExtendWith(MockitoJUnitExtension.class)`, so not with the runner. But that is a minor detail.

To unsubscribe from this group and stop receiving emails from it, send an email to mockito+u...@googlegroups.com.
To post to this group, send email to moc...@googlegroups.com.
Visit this group at https://groups.google.com/group/mockito.

To view this discussion on the web visit https://groups.google.com/d/msgid/mockito/7b568de6-956e-4d4e-8696-206a2c38775d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mockito" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mockito+u...@googlegroups.com.
To post to this group, send email to moc...@googlegroups.com.
Visit this group at https://groups.google.com/group/mockito.
To view this discussion on the web visit https://groups.google.com/d/msgid/mockito/CALS12-Pf4X-V%3DL%3D-GQ_Cn8Lwg%3DH0fCAFszQ5eY9oTd-6i50QLw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mockito" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mockito+u...@googlegroups.com.
To post to this group, send email to moc...@googlegroups.com.
Visit this group at https://groups.google.com/group/mockito.

To view this discussion on the web visit https://groups.google.com/d/msgid/mockito/VI1P194MB0077084B878A0EA7919DA6EFF6260%40VI1P194MB0077.EURP194.PROD.OUTLOOK.COM.

[Szczepan Faber]

Great discussion, thank you guys! My feedback:

1. I suggest that next time we use GitHub ticket for API discussion - it is easier to vote, it's possible to edit comments (to fix typos).

2. Do we want JUnit5 support in Mockito? There’s an option to keep it out the framework, for a different team to maintain.

3. Strictness in Runners:

MockitoJUnitRunner.Strict depends on a interesting feature offered by JUnit Runners: a callback when all test methods from given test class have completed. We use this to identify unused stubbings in the scope of the entire class. The use case: "default" stubbing in setup method, used in most of test methods but not all. MockitoJUnitRunner.Strict does not have an equivalent MockitoRule / Strictness setting (not possible to implement - rules are per test method).

Given the current API of @ExtendsWith, does it support above use case? Callback when all test methods have finished?

If the question is 'yes', are we interested in implementing MockitoJUnitRunner.Strict equivalent MockitoExtension? I'd say we don't. Let's keep the MockitoExtension consistent with MockitoJUnit.Rule and MockitoSession model. Let's make it operate on Strictness enum.

FYI: Current Strictness enum and our JUnit runners are not consistent:

a) Strictness.LENIENT == MockitoJUnitRunner.Silent

b) Strictness.WARN == no equivalent

c) Strictness.STRICT_STUBS == MockitoJUnitRunner.StrictStubs

d) MockitoJUnitRunner.Strict == combination of Strictness.WARN + Strictness.STRICT_STUBS

There 2 reasons this API is not consistent:

 - Rule and Runner don’t offer the same feature set and we wanted the best possible developer experience with each

 - Incremental design :)

To keep the API clean, simple and consistent, I suggest that the extension implements Strictness in similar way as Rules and MockitoSession. Ideally, the extension uses MockitoSession behind the hood.

4. I suggest to call the extension “MockitoJUnitExtension” to avoid future name collisions and make the intent of the public class clearer.

5. I need to think a bit more about the annotation vs. inner class API. It’s a tough choice and an important one.

Sorry for late feedback! I hope it helps! I will review the PR shortly and also offer feedback about annotation vs. inner class.

Cheers!

To unsubscribe from this group and stop receiving emails from it, send an email to mockito+unsubscribe@googlegroups.com.

To post to this group, send email to moc...@googlegroups.com.
Visit this group at https://groups.google.com/group/mockito.

To view this discussion on the web visit https://groups.google.com/d/msgid/mockito/7b568de6-956e-4d4e-8696-206a2c38775d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mockito" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mockito+unsubscribe@googlegroups.com.

To post to this group, send email to moc...@googlegroups.com.
Visit this group at https://groups.google.com/group/mockito.
To view this discussion on the web visit https://groups.google.com/d/msgid/mockito/CALS12-Pf4X-V%3DL%3D-GQ_Cn8Lwg%3DH0fCAFszQ5eY9oTd-6i50QLw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mockito" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mockito+unsubscribe@googlegroups.com.

To post to this group, send email to moc...@googlegroups.com.
Visit this group at https://groups.google.com/group/mockito.
To view this discussion on the web visit https://groups.google.com/d/msgid/mockito/VI1P194MB0077084B878A0EA7919DA6EFF6260%40VI1P194MB0077.EURP194.PROD.OUTLOOK.COM.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "mockito" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mockito+unsubscribe@googlegroups.com.

To post to this group, send email to moc...@googlegroups.com.
Visit this group at https://groups.google.com/group/mockito.

To view this discussion on the web visit https://groups.google.com/d/msgid/mockito/CAAYcR_J%3DYYniUDrrNqdXsx9qdEH5DZo6yG%3DFcfNqfhDf7XVpwQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Szczepan Faber
Creator of Mockito | author on LinkedIn | @mockitoguy on Twitter | my site on GitHub

Check out how to give great code reviews. You will be glad you did!

[Szczepan Faber]

Hey guys!

FYI: I replied to issue 792 (https://github.com/mockito/mockito/issues/792#issuecomment-347048356), where we're considering making strictness configurable per mock / per stubbing. It is related to this discussion because having strictness per mock/stubbing reduces the value of the proposal 2.

Cheers!

[Christian Schwarz]

Am Montag, 27. November 2017 00:39:34 UTC+1 schrieb Szczepan Faber:

Hey guys!

FYI: I replied to issue 792 (https://github.com/mockito/mockito/issues/792#issuecomment-347048356), where we're considering making strictness configurable per mock / per stubbing. It is related to this discussion because having strictness per mock/stubbing reduces the value of the proposal 2.

Yes it reduces the value at method or class level. We can use the same annotation to define the stricness at the field declaration of the type to mock. We have full flexability here, mockito users can uses them as they or the project requires it.

@Mock
@Strictness(LENIENT)
Foo mock;

It fits to the same concept at method and class level declaration

@Extends(MockitoJunitExtension.class)
@Strictness(STRICT_STUBS) 
class Test {
   @Strictness(LENIENT)
   @Mock Foo foo;

 
   @Strictness(SILENT)
   class NestedTest{ 

       @Test
       @Strictness(WARN)
       void test(){
         foo.bar();
         
      }  
    }
}

[Szczepan Faber]

I suggest we go ahead with proposal 1 (inner class), minus the @Strictness annotation to keep the public API simple and the internal implementation cheap. I don't think we need to add @Strictness annotation given that we have use cases to configure strictness per mock / stubbing (ticket #792).

I put together this table to help with decision making. It helped me, perhaps you will also find it useful :)

BTW. I will put together a design for #792 in the next few days. It can be worked on in parallel. Christian, do want to take a stab at it?

Cheers!

--

You received this message because you are subscribed to the Google Groups "mockito" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mockito+unsubscribe@googlegroups.com.
To post to this group, send email to moc...@googlegroups.com.
Visit this group at https://groups.google.com/group/mockito.

To view this discussion on the web visit https://groups.google.com/d/msgid/mockito/7a1c07a3-1bde-41ee-87ef-b5f0a273b09b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Szczepan Faber]

Ticket about Strictness per mock / method: I described the problem statement and suggested implementation: #792. It is related to this discussion because it removes the need for @Strictness annotation, hopefully making the decision making easier :)

Cheers!