Named setups in ExUnit

[José Valim]

Hello everyone,

Today there is no easy to share setups functions in ExUnit.

For example, let's suppose I have a setup that receives the testing metadata, spawns a couple GenServer, and return their pids:

setup context do

  server1 = MyServer1.start_link(name: context.test)

  server2 = MyServer2.start_link(name: context.test)

  if context[:server3] do

    server3 = MyServer3.start_link(name: context.test)

  end

  {:ok, server1: server1, server2: server2, server3: server3}

end

Now, imagine that I want to break those setups apart, I could write:

def setup_server_1_and_2(context) do

  server1 = MyServer1.start_link(name: context.test)

  server2 = MyServer2.start_link(name: context.test)

  {:ok, server1: server1, server2: server2}

end

def setup_sever3(context) do

  if context[:server3] do

    server3 = MyServer3.start_link(name: context.test)

  end

  {:ok, server3: server3}

end

Now I can merge them in my test:

setup context do

  {:ok, part1} = setup_server1_and_2(context)

  {:ok, part2} = setup_server3(context)

  {:ok, part1 ++ part2}

end

Which is very annoying and repetitive. Another alternative is to have two setups:

setup context do

  {:ok, setup_server1_and_2(context)}

end

setup context do

  {:ok, setup_server3(context)}

end

Which is still too verbose.

The proposal is to allow the following syntax:

setup context |> setup_server1_and_2() |> setup_server3()

This automatically pass the context through multiple setups, merging the returned results and ensure the different setups compose neatly.

A more concrete example could be setting up the connection in Phoenix to login users or setting up accept headers:

setup context |> login_as(:user) |> accepts_json()

Thoughts? Feedback?

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

[Bruce Tate]

One of the biggest problems I see in exunit is the duplication of setup code. Another is that frameworks can't provide testing helpers that allow users to do setup steps *in a composable way*. 

This proposal is a pretty good start to the most important needs I see for the testing ecosystem. 

-bt

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4%2BYkhO6x9O%3DxKcGtbp5iax9GSkj4LPUyTsj8vX8TY6M0Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Bruce Tate
President, RapidRed, LLC
Phone: 512.772.4312
Fax: 512 857-0415

Author of Seven Languages in Seven Weeks, Deploying Rails Applications, From Java to Ruby, Rails: Up and Running, Beyond Java, 6 others.

[Gabriel Jaldon]

I think having the pipes in `setup context |> setup_server1_and_2() |> setup_server3()` work differently in `setup` than it does everywhere else is confusing. How 'bout something like:

         

          def setup_server3(context) do

            if get_tag(context, :server3) do

               put_assign(context, :server3, MyServer3.start_link(name: context.test))

            else

               context

            end

          end

          def setup_server1_and_2 do(context)

            server1 = MyServer1.start_link(name: context.test)

            server2 = MyServer2.start_link(name: context.test)

            context

            |> put_assign(:server1, server1)

            |> put_assign(:server2, server2)

          end

          setup context do

            context

            |> setup_server1_and_2

            |> setup_server3

          end

In the above example, we only work on `context` just like we do with `conn` in Plug. This makes it easy for us to make helper functions to be used in `setup` that compose nicely because they would all just accept and return `context`. In the tests, we could also have a `get_assign/2` to access fields in the context rather than directly pattern-matching on `context`. This way, we could make changes to the structure of context without affecting users. 

I think this would be more familiar to Elixir devs. What do you guys think? 

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAMp4_ijmKAC6Da1MTjd67_tu6wiEOvDSiWdMQCn%2BTAaMbovCTA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Gabe Jaldon

[Aleksei Magusev]

I agree with Gabriel that pipes will look confusing, and a bit alien I guess.

[Bruce Tate]

I think a little context for this conversation is important here. 

There's a little bit of urgency to me because we need some way to control 

1) all of the ceremony inherent in setup

2) the duplication in setup 

We're using Elixir in production now, and one of the first things we did was to build a solution that allowed us to solve this problem. I think others are starting to do the same. If we don't start to address the pain points inherent in exunit, everyone is going to be building and promoting their own testing framework. 

Personally, this is where I am. If we get an Elixir alternative relatively soon, we'll use it. If we have to wait too long, we'll just evolve what we have, and others will do the same, and it will be a shame. 

Given that backdrop, José and I have been talking about alternatives. 

We use, and are happy with, a test case that looks like this:

with "a connection" do 

  setup(context)...

  end

  test...

  test...

  with "a logged in user" do

    setup(context)...

    test...

    test...

  end

end

and we're comfortable. It allows us to group and name logical tests, and it lets us nest contexts through the setups. José didn't like that as much because people can take nested contexts too far. So we brainstormed some and came up with the idea of named setup functions. That lets me have many first level setups, like this: 

setup context |> log_in(:joe) |> create_post |> ...

... tests with a logged in user

setup context |> log_in(:admin) |> create_post |> ...

... tests with a logged in admin

setup context |> create_post |> ...

... tests with a logged out user 

defsetup login(context, username) do...      #(or maybe just def)

...

It lets me quickly organize my tests, and thought the pipe doesn't work exactly like Elixir does it is a useful visual representation. It is slightly better than nested contexts, because I don't have to organize my setups in a strict tree. 

In general,  that code is beautiful to read; I intuitively understand what it does, and I have no problems with macros that allow us to maintain a mental model without maintaining precise semantics if there are very good reasons for doing so.

You can also see potential here. Since things compose, we could eventually use a similar strategy for integration tests. We could have module attributes (or just additional string arguments to setups) that further describe what a group of tests is doing so you'd be able to tell what failed without consulting the tests. 

The cost is that the macro makes use of the pipe symbol to communicate something that is almost the same as composition, but not quite. As far as my team is concerned, we read and write tests as often as we do any other development task. 

But we have to have some common place to start. It also will show that exunit is evolving, and it will be less likely that the Elixir testing situation fragments and degenerates like testing communities just about everywhere else. 

All this is to say that I am not married to the pipe syntax, but I am married to *something*, in release 1.1, that helps me control duplicate setup code and the surrounding ceremony that makes my tests much harder to read., write, and maintain.

Hope this provides a little context to this proposal. 

-bt

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/179b2da7-2b7d-4b48-8d9a-de5b745abece%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Johnny Winn]

Not sure if this helps but let me start with how we recently implemented context for individual tests. 

Here is an example setup:

setup context do

  cond do

     context[:valid] -> 

          {token, expires} = context[:valid]

          Our.Cache.put(token, epoch_now + expires)

          {:ok, [token: token, expires: expires]}

     context[:expired] ->

          {token, expires} = context[:valid]

          Our.Cache.put(token, epoch_now - expires)

          {:ok, [token: token, expires: expires]}

     context[:missing] ->

         {token} = context[:missing]

         {:ok, [token: token]}

     true ->

         {:ok}

  end

end

Then here is one of our tests:

 @tag valid: { "e72e16c7e42f292c6912e7710c838347ae178b4a", 7200 }

 test "validate a stored token", %{token: token} do

   assert true == Our.Cache.active_token?(token)

 end

Now, is this my favorite, no but it does allow me to create a "state" for my tests to run in. Where I could see this breaking down is when a complex state is needed to run a test but I think that is actually a different problem.

I could easily see that transformed into reusable functions for state. Something like:

describe "valid thing" do

  setup context do 

  end

  test "this thing is valid" do

   

  end

end

but I would almost prefer being able to reuse that through functions:

 setup context do

    describe context

  end

  # this would need the fu

  def describe(%{valid: context}) do

    {token, expires} = context

    {:ok, [token: token, expires: expires]}

  end

  @tag valid: { "12345", 1234 }

  test "the truth", %{token: token} do

    assert "12345" == token

  end

Sorry, it's not complete. I know that `context` would need worked in there too.

Thanks,

Johnny

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAMp4_ij_4_Frvc0RRLfya_1xuQ_ShKHtP1_RUmENH6HFtD3A_Q%40mail.gmail.com.

[Booker Bense]

Maybe just a different operator symbol? 

This is kind of a emacs/vi debate, i.e. context dependent behavior of exactly the same symbol. 

You either hate it or love it. As an emacs user, I hate it. 

But there are vi users and we all have to co-exist. 

I vaguely recall that there are a couple "open" operators that can be used.

I can't find documentation on these <~> <|>

   setup context <|> log_in(:admin) <|> create_post ... 

Or you could use a word.  

   setup context with log_in(:admin) with create_post with ...

not as visually pleasing. I did a google image search of merge

and <|> seems close to some of the common images that came

up.

- Booker C. Bense 

[Saša Jurić]

I wonder if this is really a test-specific problem, or something that needs to be tackled on the language level. 

In particular, the problem seems to be in this code:

setup context do

  {:ok, part1} = setup_server1_and_2(context)

  {:ok, part2} = setup_server3(context)

  {:ok, part1 ++ part2}

end

As Gabriel mentions, with a plain pipeline we can compose and reuse functions:

context

|> my_setup_1

|> my_setup_2

|> ...

However, the problem here is that functions return {:ok, result} or {:error, reason}. There was recently a related SO question (http://stackoverflow.com/questions/32239919/return-statement-in-elixir) where various solutions were proposed, such as ad-hoc composition, 3rd party monad libraries or Bruce's elixir_pipes.

If we have a standard solution for this pattern out of the box, perhaps we could avoid introducing special sugar to test framework only.

[José Valim]

If the main concern is that the pipeline does not work as in Elixir code, would it be solved if all named setups are required to receive the context and return a context? This makes them slightly different from setup but it would have a more straight-forward translation:

setup context |> login_as(:user) |> accepts_json()

Which translates to:

setup context do

  {:ok, context |> login_as(:user) |> accepts_json()}

end

What do you think?

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

[Booker Bense]

I kind of really like the idea of the <|> "space pipe" operator, mostly because I like saying

"space pipe". 

This would take any { :ok, foo } tuple and work just like 'foo |>' as long as elem(0) == :ok , 

if not it would raise an error. 

Again we seem to be wandering into the dataflow terrain. 

On a serious note, what happens when one of the setup functions does not return 

{:ok, context} ? 

- Booker C. Bense

P.S. You know you really want to work in a language with a "space pipe" operator. Just

admit it... 

P.P.S. SPACE PIPE ERROR, come on. %-)

[José Valim]

I don't think it is worth introducing another operator due to the conceptual overhead as I would prefer the approach in Bruce's pipe library.

In any case, you may as well send a proposal here in core identifying the cases and arguing for the new operator. It should not be part of this discussion though because we would like, if possible, to introduce names setups for 1.1.

--

[Booker Bense]

Sorry, I was just trying to be silly on a Friday after a long week. 

However I did have a real question about failure modes. Or is it just

obvious that you must raise an excption in the setup code? 

- Booker C. Bense 

P.S. Writing SpacePipe was far easier than I expected. I encourage everyone 

to try it, I learned a lot in an hour.