Test grouping and named setups

[José Valim]

Hi everyone,

There are two recurrent complaints about ExUnit when writing tests:

1. It does not support grouping

2. It does not allow developers to name nor compose setup

The only way in ExUnit to declare some code as part of a setup is by declaring a setup block and calling the function inside the block, like this:

setup context do

  {:ok, login_as(context)}

end

I would like to propose improvements for those two issues. I will first write a proposal including code samples and then we can discuss naming and implementation details.

## Groups and named setups

The idea is to introduce the group/2 macro and allow atoms to be given to setup:

defmodule StringTest do

  use ExUnit.Case, async: true

  group "String.capitalize/2" do

    setup :set_test_string

    test "capitalizes the first letter", context do

      assert "T" <> _ = context.test_string

    end

  end

  group "String.bar/2" do

    setup context do

      # Regular setups are not going anywhere and will still work

      # You may return :ok | keyword | map

    end

    test "..." do ...

  end

  def set_test_string(_context) do

    %{test_string: "test_string"}

  end

end

By using groups, we can now better organize the tests and named setups allow us to easily share setup logic between groups without relying on nesting.

Internally, we can consider "setup :some_atom" to be equivalent to:

setup context, do: context |> some_atom()

The group/2 macro will also store the group tag in the test. For example, you will be able to:

mix test --only group:"String.capitalize/2"

Setups defined in group/2 will only apply to the group. We will also support @grouptag to set some tags specific to the current group. setup_all cannot be called inside the group (as setup_all is always per test case). We won't allow group calls to be nested (we want developers to compose at the function level and not on nesting/hierarchies).

## Naming

There is one big open question which is: is "group" a good name? I am personally not a fan. group works fine for grouping at the unit test level but it does not read nicely when you want to group based on a condition (for example, group "when name is an atom").

Here are some alternatives: context, testing, group, describe, scenario, having, etc. I recommend everyone to actually try to write some pseudo-groups in their tests and see what reads nicely. I would love your feedback on what you think works nicely. Please include examples. :)

One of my favorite picks is context but we already use context to mean the data that goes through setup and tests.

## Feedback

Please give us feedback on this proposal. Those features are mostly straight-forward to implement, the most important aspects are the semantics.

Thank you!

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

[Josh Adams]

I love this proposal.  I have always liked reading the `describe`-style tests.  That's my vote.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4JpFn-%3DGkG_Ya-w64Qqt7OoworqUSAUgzqgWtTFU1fyrw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Josh Adams

[Drew Olson]

I'm a big fan of this proposal as well. I implemented ex_spec[1] to support this for my projects and I've found the "describe" and "context" names to be nice (they function as aliases).

I'm excited about composing setups (something I don't support in ex_spec), but I'm confused about how and where the composition happens. If you have a setup at the top-level and then more (named or traditional) setups in the groups, are they composed? Are the setups in the group only specific to that group?

Thanks, looking forward to this!

- Drew

[1] - https://github.com/drewolson/ex_spec

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAA1-O0wmo0T3Bu5CpYrESkgb4z%2B2v_A%2Bu2Y3Tvr8--a8oAmC7Q%40mail.gmail.com.

[Andrea Leopardi]

Personally I really don't like "describe" or "scenario"; the former forces you to group these tests by a description, the second one has a "enterprisy" feel to me :P and I don't think brings any meaningful meaning with it.

Tbh, I think group is the most fitting name for such functionality: it's very direct, you're just creating a group of tests that share something.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAA1-O0wmo0T3Bu5CpYrESkgb4z%2B2v_A%2Bu2Y3Tvr8--a8oAmC7Q%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Andrea Leopardi

an.le...@gmail.com

[Bruce Tate]

This solves two of the biggest problems, as you describe. 

+1 from me as well. We're working on some things to improve composition in tests, and across tests. 

-bt

On Fri, May 27, 2016 at 11:18 AM, José Valim <jose....@plataformatec.com.br> wrote:

--

You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4JpFn-%3DGkG_Ya-w64Qqt7OoworqUSAUgzqgWtTFU1fyrw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Bruce Tate
President, RapidRed, LLC
Phone: 512.772.4312
Fax: 512 857-0415

Author of Seven Languages in Seven Weeks, Deploying Rails Applications, From Java to Ruby, Rails: Up and Running, Beyond Java, 6 others.

[Sergii Boiko]

Maybe a little bit offtopic, but my main issue with ExUnit is that assert and refute match on truthiness and falsiness instead of strict true/false match.

First run into it when my test suddenly passed on empty function, because it expected false, but refute accepts nil as well.

The same issue was with Rspec 2, but they fixed it in RSpec 3. Maybe it's possible to address it with some special option?

[Ben Wilson]

I imagine the behaviour of assert and refute aren't going to change, as that'd be a breaking change. assert foo() == false is pretty clear as a work around though.

[Dmitry Belyaev]

Hi Jose,

The proposal is great, thank you.

I personally like 'group' as it is similar to common_test in Erlang/OTP. But if some other term had to be used I think 'set' or 'category' would also do.

--
Best wishes,
Dmitry Belyaev

[Daniel Perez]

I think `group` is very explicit and easy to understand.

I find it much clearer than describe/context.

> group "when name is an atom"

I think this is because it is the text we would write with `context`,

but dropping the initial `when` would simply read fine IMO: group "name is an atom"

Daniel

[José Valim]

Thanks everyone for the feedback so far.

It is very likely we won't go with the term "group". It is an overloaded term and given the direction we are taking with GenStage and GenBroker, I wouldn't be surprised if we end-up adding Kernel.group/2 or something similar in the future. Group or group by is an essential in many collection handling operations and there is nothing specific to test in "group".

Here are other suggestions that came up: "grouping", "bundle", "test_group".

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

[José Valim]

Both features have been implemented here:

1. https://github.com/elixir-lang/elixir/commit/056536add75accc02542ef84d48c8535127c4171

2. https://github.com/elixir-lang/elixir/commit/58377cdeea091643c002414942dcab4e8b042b13

Notice we used the name "bundle" instead of "group" since it provides the same idea of grouping and can be read as both noun and verb. One of the benefits of the word "bundle" is that it generally does not come with the expectation it can be nested.

In any case, the naming discussion is still open for debate and feedback is welcome.

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

[Drew Olson]

This is looking good, I like "bundle". Any specific reason to not support nesting?

Thanks!

Drew

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4L6iRmHzfjQmXn0vTC0E%2B3gW1%3Drqj_eett1YvCfwOrXAg%40mail.gmail.com.

[José Valim]

Thank you Drew. The reason for not support nesting was mentioned in the initial proposal. We want to push developers to compose setups by building functions (hence the support for setup :some_code) instead of relying on hierarchies.

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAN3B1jdTD3nO6So_YD-aVkN6Hfc7dCBajnpfMbjoj0RSJtpMLQ%40mail.gmail.com.

[Drew Olson]

José -

Thanks for the clarification, I should have read more closely. 

I understand the argument against nested setups and think stopping folks from using them is a good idea. That said, I often find nested groups useful in grouping subsets of functionality within a given function (just from an organizational / naming perspective). 

Either way, this is a huge improvement. 

Thanks. 

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4%2BOCkrn%2Bq588%2Bc8DD%2BuHbAbwFO79BZhoCXczYrXZ9UUrA%40mail.gmail.com.

[Tobias Pfeiffer]

Although a bit late to the party I want to express my immense support
for this proposal. The 2 concerns you mentioned are my chief concerns
and especially among people I show Elixir as well. So there is a big
:+1: for this.

Not a huge fan of group/bundle - still very much like describe/context.
`describe` for `describe "MyModule.fun"` and context for `context "admin"`.

Glad to see ExUnit is taking this direction!
Tobi

On 05/28/2016 01:47 PM, Drew Olson wrote:
> José -
>
> Thanks for the clarification, I should have read more closely.
>
> I understand the argument against nested setups and think stopping folks
> from using them is a good idea. That said, I often find nested groups
> useful in grouping subsets of functionality within a given function
> (just from an organizational / naming perspective).
>
> Either way, this is a huge improvement.
>
> Thanks.
>
> On Saturday, May 28, 2016, José Valim <jose....@plataformatec.com.br

> <mailto:jose....@plataformatec.com.br>> wrote:
>
> Thank you Drew. The reason for not support nesting was mentioned in
> the initial proposal. We want to push developers to compose setups
> by building functions (hence the support for setup :some_code)
> instead of relying on hierarchies.
>
>
>

> *José Valim*
> www.plataformatec.com.br <http://www.plataformatec.com.br/>

> Skype: jv.ptec
> Founder and Director of R&D
>
> On Sat, May 28, 2016 at 1:26 PM, Drew Olson <dr...@drewolson.org
> <javascript:_e(%7B%7D,'cvml','dr...@drewolson.org');>> wrote:
>
> This is looking good, I like "bundle". Any specific reason
> to not support nesting?
>
> Thanks!
> Drew
>
>
> On Saturday, May 28, 2016, José Valim
> <jose....@plataformatec.com.br

> <javascript:_e(%7B%7D,'cvml','jose....@plataformatec.com.br');>>

> wrote:
>
> Both features have been implemented here:
>
> 1. https://github.com/elixir-lang/elixir/commit/056536add75accc02542ef84d48c8535127c4171
> 2. https://github.com/elixir-lang/elixir/commit/58377cdeea091643c002414942dcab4e8b042b13
>
> Notice we used the name "bundle" instead of "group" since it
> provides the same idea of grouping and can be read as both
> noun and verb. One of the benefits of the word "bundle" is
> that it generally does not come with the expectation it can
> be nested.
>
> In any case, the naming discussion is still open for debate
> and feedback is welcome.
>
>
>

> *José Valim*
> www.plataformatec.com.br <http://www.plataformatec.com.br/>

> Skype: jv.ptec
> Founder and Director of R&D
>
> On Sat, May 28, 2016 at 8:53 AM, José Valim
> <jose....@plataformatec.com.br> wrote:
>
> Thanks everyone for the feedback so far.
>
> It is very likely we won't go with the term "group". It
> is an overloaded term and given the direction we are
> taking with GenStage and GenBroker, I wouldn't be
> surprised if we end-up adding Kernel.group/2 or
> something similar in the future. Group or group by is an
> essential in many collection handling operations and
> there is nothing specific to test in "group".
>
> Here are other suggestions that came up: "grouping",
> "bundle", "test_group".
>
>
>

> *José Valim*
> www.plataformatec.com.br <http://www.plataformatec.com.br/>

> *José Valim*
> www.plataformatec.com.br
> <http://www.plataformatec.com.br/>

> Skype: jv.ptec
> Founder and Director of R&D
>
>
>
> --
> You received this message because you are subscribed to the
> Google Groups "elixir-lang-core" group.
> To unsubscribe from this group and stop receiving emails
> from it, send an email to
> elixir-lang-co...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4L6iRmHzfjQmXn0vTC0E%2B3gW1%3Drqj_eett1YvCfwOrXAg%40mail.gmail.com

> <https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4L6iRmHzfjQmXn0vTC0E%2B3gW1%3Drqj_eett1YvCfwOrXAg%40mail.gmail.com?utm_medium=email&utm_source=footer>.

> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the
> Google Groups "elixir-lang-core" group.
> To unsubscribe from this group and stop receiving emails from
> it, send an email to
> elixir-lang-co...@googlegroups.com

> <javascript:_e(%7B%7D,'cvml','elixir-lang-core%2Bunsu...@googlegroups.com');>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/elixir-lang-core/CAN3B1jdTD3nO6So_YD-aVkN6Hfc7dCBajnpfMbjoj0RSJtpMLQ%40mail.gmail.com
> <https://groups.google.com/d/msgid/elixir-lang-core/CAN3B1jdTD3nO6So_YD-aVkN6Hfc7dCBajnpfMbjoj0RSJtpMLQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.

>
> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "elixir-lang-core" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to elixir-lang-co...@googlegroups.com

> <javascript:_e(%7B%7D,'cvml','elixir-lang-core%2Bunsu...@googlegroups.com');>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4%2BOCkrn%2Bq588%2Bc8DD%2BuHbAbwFO79BZhoCXczYrXZ9UUrA%40mail.gmail.com
> <https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4%2BOCkrn%2Bq588%2Bc8DD%2BuHbAbwFO79BZhoCXczYrXZ9UUrA%40mail.gmail.com?utm_medium=email&utm_source=footer>.

> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google
> Groups "elixir-lang-core" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to elixir-lang-co...@googlegroups.com

> <mailto:elixir-lang-co...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/elixir-lang-core/CAN3B1jdZbRKYRUSYzfNPvZJnyvr0HyqfrgFNHp6CeHnFDhbN8g%40mail.gmail.com
> <https://groups.google.com/d/msgid/elixir-lang-core/CAN3B1jdZbRKYRUSYzfNPvZJnyvr0HyqfrgFNHp6CeHnFDhbN8g%40mail.gmail.com?utm_medium=email&utm_source=footer>.

[José Valim]

Not a huge fan of group/bundle - still very much like describe/context.
`describe` for `describe "MyModule.fun"` and context for `context "admin"`.

After following my own advice and using "bundle" for a couple days I have really struggled to write good "bundle" names and make them read cleanly alongside my test definitions.

I will soon push a commit to master that gives "describe" a try. We don't plan to support both describe and context though and context is a no-go for the other reasons stated earlier.

Thank you Tobi,

[Robert Virding]

Are there any plans to move towards adapting and using Common Test? It is much more oriented towards testing systems with many ways of grouping (:-)) and controlling tests.

Robert

[Gleb Arshinov]

test_group is not elegant. But it is scalable if you consistently
prefix all these keywords with test_ e.g. test_setup This creates a
separate namespace for test-related words. Could be the way to go if
you want to have a lot of them.

Another word that comes to mind is "collection"

Gleb

> --
> You received this message because you are subscribed to the Google Groups
> "elixir-lang-core" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elixir-lang-co...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4J-DbM9Nt6M743mGxBkKhE5i9Y3QajtBFaGoogCXNPa0Q%40mail.gmail.com.

[Nathan Hessler]

 any thoughts on terms such as given/when. I'm not a fan of cucumber, but I do think these could easily replace describe/context and even read a bit better.

given "String.capitalize/2" do

    setup :set_test_string

 

    when "whatever state/flag/condition we want to test" do 

        test "capitalizes the first letter", context do

              assert "T" <> _ = context.test_string

        end

               end 

  end

[Chris Keele]

I think 'having' scans well, but entirely because it's a past participle which English uses for pretty much every verb tense, which makes it easy to write spec-like test names. However, it doesn't play well with tests groups like having "Registry.register/3". Maybe 'concerning' which works well with non-sentence test names and is a little more descriptive?

• Setup: concerning "a User with no name"
• Edge case: concerning "inputs with extra spaces"
• Function namespaces: concerning "Registry.register/3"
• Verb lead-ins: concerning "parsing invalid data"
• Noun lead-ins: concerning "an invalid token"

[Jamu Kakar]

To throw another suggestion into the mix, about might work:

about "parsing invalid data" do

  test "negative integers are rejected" do: ...

end

--

You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/e04100f5-d1b5-45b5-a110-8a4c1d06c911%40googlegroups.com.

[Ciarán Walsh]

I was thinking `scene`:

  scene "parsing invalid data" do

    test "negative integers are rejected" do: ...

  end

[Bruce Tate]

scene is very nice. 

-bt

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/d5d51651-2ea6-4a13-b58c-074da80ee703%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Onorio Catenacci]

Or even scenario:

scenario "parsing invalid data" do

   test "negative integers are rejected" do: ...

end

--

Onorio

[Greg Vaughn]

My contribution to the bikeshed ;-)

"regarding" and/or its abbreviation "re"

-Greg Vaughn

> To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/4b48930c-ef8d-4537-8492-4cf1bfb20d9f%40googlegroups.com.