[Proposal] Improve supported syntax in Doctest

[eksperimental]

I have been working on a few improvements for doctests.
Everything that I'm proposing here has already been implemented.
The proposals started because some modules in Elixir are not being tested with doctests, and this
is because there are certain limitation in doctest. These improvements aim to eliminate those.

My proposal consist of two main points:

1) Streamline Doctests with regular examples.
Currently some examples need to be expressed in regular code, not in IEx examples, because some
features are not supported.

-- a) Include comments in doctest syntax.
We should allow full-line comments in code (currently only comments after evaluated expressions
are allowed). My proposal is to have three different kind of comments.

---- I) Regular comments, such as:
# this is a regular comment
iex> true
true

---- II) Return value comments.
These are useful when we will not get the same value everytime the function is run.

iex> spawn(fn -> true end)
#=> PID<0.97.0>

---- III) Print comments.
Currently it is not clear when a value is returned or printed on screen (stdout/stderr), so my
proposal is to create a new convention for this. My suggestion is to use "#>> ", but it could
be anything.

iex> IO.puts "Elixir"
#>> Elixir
:ok

2) Mimic IEx by allowing user to import IEx.Helpers, which are automatically imported when IEx is
loaded.
I came across this when I was trying to run doctests in the Port module. In the examples the
flush/0 function is used which is imported from IEx.Helpers. Currently it is impossible simulate a
doctest if the whole IEx environment is required, without explicitly importing IEx.Helpers.
So my proposal is to add an option :import_iex_helpers, which works the same way as :import
option, taking :true, :false, :only, and :except values (default value is :false).


doctest ModuleToDocTest, import_iex_helpers: true

or
doctest ModuleToDocTest, import_iex_helpers: [only: [flush: 0]]


this way the first example in the Port module could be doctested as:

iex> port = Port.open({:spawn, "cat"}, [:binary])
#=> #Port<0.1444>
iex> send port, {self(), {:command, "hello"}}
#=> {#PID<0.80.0>, {:command, "hello"}}
iex> send port, {self(), {:command, "world"}}
#=> {#PID<0.80.0>, {:command, "world"}}
iex> flush()
#>> {#Port<0.1444>, {:data, "hello"}}
#>> {#Port<0.1444>, {:data, "world"}}
:ok
iex> send port, {self(), :close}
#=> {#PID<0.80.0>, :close}
iex> flush()
#>> {#Port<0.1444>, :closed}
:ok


The implemented and tested code can be found here:
https://github.com/eksperimental/elixir/tree/doctest_improvements3

Thank you for reading and looking forward for your feedback.

[José Valim]

We should support regular comments (a-I). We should not support (a-II) and (a-III) because they are not valid in IEx.

import_iex_helpers sounds like an interesting idea. Is there any helper we would use besides flush()?

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/20170508215910.65ffd36a.eksperimental%40autistici.org.
For more options, visit https://groups.google.com/d/optout.

[eksperimental]

José Valim <jose....@plataformatec.com.br> wrote:
> import_iex_helpers sounds like an interesting idea. Is there any helper we
> would use besides flush()?

probably yes,
such as pid/1,3 pwd/0, maybe ls/1


>
> *José Valim*

> > email to elixir-lang-co...@googlegroups.com.

[eksperimental]

José Valim <jose....@plataformatec.com.br> wrote:
> We should support regular comments (a-I). We should not support (a-II) and
> (a-III) because they are not valid in IEx.

How come, since they are all treated as comments by IEx.

[José Valim]

pwd and ls are not going to be effective in doctests as the results will be highly dependent of the user directory and what is on their machine.

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/20170508221600.66851fc4.eksperimental%40autistici.org.

[eksperimental]

They could be useful for doctests such when dealing with file and dirs creation and removal,
you cd/1 a dir, and then mkdir_p, and `touch` a file and `ls` the contents of the specific dir,

On Mon, 8 May 2017 18:17:33 +0200
José Valim <jose....@plataformatec.com.br> wrote:

> pwd and ls are not going to be effective in doctests as the results will be
> highly dependent of the user directory and what is on their machine.
>
>
>

> *José Valim*

> www.plataformatec.com.br
> Skype: jv.ptec
> Founder and Director of R&D
>
> On Mon, May 8, 2017 at 5:16 PM, eksperimental <eksper...@autistici.org>
> wrote:
>
> > José Valim <jose....@plataformatec.com.br> wrote:
> > > We should support regular comments (a-I). We should not support (a-II)
> > and
> > > (a-III) because they are not valid in IEx.
> >
> > How come, since they are all treated as comments by IEx.
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elixir-lang-core" group.
> > To unsubscribe from this group and stop receiving emails from it, send an

> > email to elixir-lang-co...@googlegroups.com.

[José Valim]

Then you can use the File API to do exactly that. Such as File.ls! and File.cwd!.

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

On Mon, May 8, 2017 at 6:26 PM, eksperimental <eksper...@autistici.org> wrote:

They could be useful for doctests such when dealing with file and dirs creation and removal,
you cd/1 a dir, and then mkdir_p, and `touch` a file and `ls` the contents of the specific dir,



On Mon, 8 May 2017 18:17:33 +0200
José Valim <jose....@plataformatec.com.br> wrote:

> pwd and ls are not going to be effective in doctests as the results will be
> highly dependent of the user directory and what is on their machine.
>
>
>
> *José Valim*
> www.plataformatec.com.br
> Skype: jv.ptec
> Founder and Director of R&D
>
> On Mon, May 8, 2017 at 5:16 PM, eksperimental <eksper...@autistici.org>
> wrote:
>
> > José Valim <jose....@plataformatec.com.br> wrote:
> > > We should support regular comments (a-I). We should not support (a-II)
> > and
> > > (a-III) because they are not valid in IEx.
> >
> > How come, since they are all treated as comments by IEx.
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elixir-lang-core" group.
> > To unsubscribe from this group and stop receiving emails from it, send an

> > email to elixir-lang-core+unsubscribe@googlegroups.com.

> > To view this discussion on the web visit https://groups.google.com/d/
> > msgid/elixir-lang-core/20170508221600.66851fc4.
> > eksperimental%40autistici.org.
> > For more options, visit https://groups.google.com/d/optout.
> >
>

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/20170508232653.0388c5f1.eksperimental%40autistici.org.

[eksperimental]

I see your point. But I think it would simplify things if what's available in IEx is available
in doctests, since that's the environment it is supposed to be running in.

On Mon, 8 May 2017 18:28:32 +0200
José Valim <jose....@plataformatec.com.br> wrote:

> Then you can use the File API to do exactly that. Such as File.ls! and
> File.cwd!.
>
>
>

> > > > email to elixir-lang-co...@googlegroups.com.

> > > > To view this discussion on the web visit https://groups.google.com/d/
> > > > msgid/elixir-lang-core/20170508221600.66851fc4.
> > > > eksperimental%40autistici.org.
> > > > For more options, visit https://groups.google.com/d/optout.
> > > >
> > >
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elixir-lang-core" group.
> > To unsubscribe from this group and stop receiving emails from it, send an

> > email to elixir-lang-co...@googlegroups.com.

> > To view this discussion on the web visit https://groups.google.com/d/

> > msgid/elixir-lang-core/20170508232653.0388c5f1.

[Andrea Leopardi]

FWIW, when I need to test messages today I use `receive` in doctests as well.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/20170508233503.5b997d54.eksperimental%40autistici.org.

[eksperimental]

So, where should we move from here?

[José Valim]

We should support the first bullet, which is the comments syntax if we don't support it yet.

I don't think we should support IEx helpers because although users will often try them in IEx, they may also try in their apps and any IEx helper will fail. And it will be confusing especially for new users which are not quite familiar with what is part of IEx and what is Kernel.

--

José Valim

[eksperimental]

> I don't think we should support IEx helpers because although users will
> often try them in IEx, they may also try in their apps and any IEx helper
> will fail. And it will be confusing especially for new users which are not
> quite familiar with what is part of IEx and what is Kernel.

That's a really good point.

Should we only import `flush`?

Could you also care to exlain why not support "#=> ". It is not clear yo me yet why they are not
supported by IEx when they are just comments, no different than what we will support in bullet 1.

[José Valim]

If they fit under the first case, then sure. But no special case should be added and we should not promote using comments for return types. If you write this:

    IEx> 1

    #=> :oops

We won't compare 1 with :oops. And there is no purpose in writing in the style above as nothing will be tested anyway.

[José Valim]

We should not import flush for all of the reasons said before. Prefer to use receive.

[eksperimental]

Well, after working out the regular comments a bit, I can tell you the hard part is implementing
the inclusion of regular comments, because they may not always make a valid doctest.
Result comments (#=> ) and print comments (#>> ) are super easy to implement,
literally 8 lines of code each.

But they work different, because the implementation that I made for regular comment only take a
regular comment as a line starting with a hash followed by a white-space, otherwise it is not
possible to differentiate it from "#PID<0.94.0>" or mapsets.

Result and print comments (#=>, #>>) cannot be treated as regular comments, because doctests will
always validate with them. And yes, they are not evaluated (that's the whole point of having them),
otherwise we would just use regular code.

Please have a look at the final implementation.
https://github.com/eksperimental/elixir/tree/doctest_comments
I think I have covered all the cases with invalid tests when comments are used.

This commits
https://github.com/eksperimental/elixir/commit/d79e7aea4e07e15550c97ed67487d0b2e019ff3f
details all the changes, I can split them at least in two commits if we decide to go with them