PROPOSAL: Improvements to doctests with pattern matching

[Devon Estes]

I think doctests are extremely helpful and powerful since they encourage developers to write testable, accurate documentation for their code. But the lack of some of the niceties that exist in ExUnit might make it less likely for developers to choose doctests instead of ExUnit tests, so I've been thinking of ways to narrow this gap.

Currently if we are writing doctests and want to assert that the output of the function we're testing matches a pattern instead of testing the output for strict equality, we need to do one of the following two things:

```

iex> match?(%{c: _}, Map.put(%{a: :b}, :c, :d))

true

iex> %{c: _} = Map.put(%{a: :b}, :c, :d)

%{a: :b, c: :d}

```

There are a few problems I see with this at the moment. First is the failure messages - they're not nearly as good as the failure messages in ExUnit at the moment for pattern matching.

```

iex> %{f: _} = Map.put(%{a: :b}, :c, :d)

%{a: :b, c: :d}

```

  1) doctest Benchee.Configuration.init/1 (1) (Benchee.ConfigurationTest)

     test/benchee/configuration_test.exs:3

     ** (MatchError) no match of right hand side value: %{a: :b, c: :d}

     stacktrace:

       (for doctest at) lib/benchee/configuration.ex:167: (test)

And then for the other one we get an even less helpful error message:

```

iex> match?(%{f: _}, Map.put(%{a: :b}, :c, :d))

true

```

  1) doctest Benchee.Configuration.init/1 (1) (Benchee.ConfigurationTest)

     test/benchee/configuration_test.exs:3

     Doctest failed

     doctest:

       iex> match?(%{f: _}, Map.put(%{a: :b}, :c, :d))

       true

     code:  match?(%{f: _}, Map.put(%{a: :b}, :c, :d)) === true

     left:  false

     right: true

     stacktrace:

       lib/benchee/configuration.ex:167: Benchee.Configuration (module)

To get the very nice new colored diff for matches available in 1.10 in our failure message we can do the following:

```

iex> assert %{f: _} = Map.put(%{a: :b}, :c, :d)

%{a: :b, c: :d}

```

  1) doctest Benchee.Configuration.init/1 (1) (Benchee.ConfigurationTest)

     test/benchee/configuration_test.exs:3

     match (=) failed

     code:  assert %{f: _} = Map.put(%{a: :b}, :c, :d)

     left:  %{f: _}

     right: %{a: :b, c: :d}

     stacktrace:

       (for doctest at) lib/benchee/configuration.ex:167: (test)

But putting this assertion in a doctest takes away from value of the example as documentation since the example is no longer really able to be cut and pasted - it's now more test than documentation and doesn't really belong here.

Also, If you want to pattern match in the final line of a doctest, you still need to put the return value of the match (which is always the value being matched against) at the end of the test or else it's not parsed as a valid test, which makes the pattern match useless since you're already making the more strict comparison with `===`. Basically, we can't do this today:

```

iex> %{f: _} = Map.put(%{a: :b}, :c, :d)

iex> %{e: _} = Map.put(%{a: :b}, :e, :f)

iex> %{f: _} = Map.put(%{a: :b}, :f, :e)

```

My proposal is that we allow this to be a valid doctest, and that we use it to give really good failure messages. Anytime the last line of a doctest is a pattern match, we convert that into `assert pattern = expr` when parsing and running the test.

The other thing I came up with (that I don't particularly like but figured I'd put it out there just in case someone builds something good off of it) is to use the anonymous function shorthand syntax to represent the output of the previous line and do something like this:

```

iex> Map.put(%{a: :b}, :c, :d)

%{f: _} = &1

iex> Map.put(%{a: :b}, :e, :f)

%{e: _} = &1

```

This gets a little further away from the intended purpose of documentation and towards more of a test, but it would still give a better failure message for developers and keeps the requirement that the last line of a doctest includes something to test against.

If folks have any better ideas for this I'd love to hear them!

[José Valim]

Good call!

What if we simply convert any line where = is the root expression and the left-side is not a variable to an assertion?

The rationale is that this is an ok doctest:

    iex> {:error, _} = do_something(var, 0)

    iex> :ok = do_something(var, 1)

And we would most likely want to raise on these cases too.

So we can just convert everything automatically whenever we can.

José Valim

www.plataformatec.com.br

Founder and Director of R&D

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/b93ea3e4-9ebe-4090-b14c-bae138c60f4f%40googlegroups.com.