Should this result in a warning?

[thibaut...@gmail.com]

Hello,

While debugging some doctests I haven't written initially myself, I realised that one can create doctests that are bogus and will be skipped quite easily, just like this:

## Examples

iex > SomeModule.some_method("param")

"expected_output"

The culprit is the extra space between "iex" and ">", which prevents the test from being executed (and the developer is not aware of that at all!).

I wonder if some form of protection against that could be made, either in Elixir or in Credo maybe?

Not a huge issue, but it could result in some head-banging for some :-)

-- Thibaut

[José Valim]

Hi Thibaut,

There are two questions I asked myself:

1. What are the broadcast cases we can warn on to help people running into situations like these?

2. Can we emit false positives?

One thing to keep in mind is that looking for "iex>" is already a bit lax today as it doesn't consider indentation. So having any line starting with "iex>" will be considered a doctest. Adding rules like "iex >" may make false positives more possible then, especially if we try to detect other typos.

So at the moment, I am not sure if it is worth it. It feels like this should be solved somehow (maybe change editors to highlight it), but I am not convinced.

What are everyone's thoughts?

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/bf7c0071-5540-409c-8e63-bdcf31fca68bn%40googlegroups.com.

[Valter Sundström]

Personally I feel the risk of a false positives is very low for this: 

^ *iex *>

 In how many other situations would you want a line starting with this, before a test, and not mean for it to be a doctest? I really struggle to come up with an example, but I might definitely have blind spots.

Accidentally dropping tests silently seems a lot worse, even if I think that the situation shouldn't happen often if tests are properly run when being made (e.g watching them fail at least once, specifically to make sure your test is actually doing what you think).

In the worst case, it also seems a change like this could be reverted in a following minor release if it really turns out to give too many false positives.

Just my casual input

// Valter

[Adam Lancaster]

Is it something the formatter could handle at all? Perhaps with a best effort if it ever sees `iex >` it makes it `iex>`

Best

Adam

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAPkwuK%3DbGV7yWuvW5Xo_gThGo3_XKbcZX0Tzeh-%2BphHJh5iMxg%40mail.gmail.com.

[José Valim]

As a general principle the formatter does not change the AST and changing the contents of a string would definitely change the AST. :)

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/5028A1A0-7302-4586-A7CD-F03EC26089E3%40a-corp.co.uk.

[Thibaut Barrère]

After sleeping on it, I realise that I've come across this specific problem only once, ever, since working with Elixir.

Also it reminds me of another case where the doctests were written, but not registered in the test case, causing similar consequences (if I recall well).

Maybe there is a broader theme of "how to make sure doctests cannot catch people off guard", but for the specific problem I've raised here, I believe some time is probably needed to see clearer, because the drawbacks of "managing" this (false positives / how to whitelist them etc) are very real too.

I'll comment back if I meet those issues in a later codebase :-)

-- Thibaut

You received this message because you are subscribed to a topic in the Google Groups "elixir-lang-core" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elixir-lang-core/kqal5g87_EE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4KAb5hHvvmuwrdFaUhbeFdauEghrOg0Tqmd1Vj4-V%3DofQ%40mail.gmail.com.