Bisect tests question

[Paulo D Gonzalez]

In my previous life ( :) ) I was able to use something that would be equivalent to `mix test --bisect --seed 776` and it used to be very helpful when dealing with a big codebase with crazy test failures.

I was wondering what are your thoughts on this feature in Elixir? I tried looking at existing discussions on this and didn't find any. If there are, could you please point me to it?

Thank you.

Paulo

[Andrea Leopardi]

I can't recall if this has been discussed before but I don't think it has. Could you expand in more detail what --bisect does?

Andrea Leopardi

an.le...@gmail.com

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/7bf0827d-1174-4aec-ba08-58e88de6c2de%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Paulo D Gonzalez]

Hi Andrea! Thanks for the quick reply.

Here is some info on bisect: 

https://relishapp.com/rspec/rspec-core/docs/command-line/bisect

Here is where the implementation of the lib I used to use:

https://github.com/rspec/rspec-core/tree/master/lib/rspec/core/bisect

This was useful when tests would fail in a mysterious way. This would give us some added introspection when it succeeded. It felt like a `git bisect` that was done automatically for you.

You received this message because you are subscribed to a topic in the Google Groups "elixir-lang-core" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elixir-lang-core/rv1A8x4DBvo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAM9Rf%2BKwSfrMgwM%2B42J3bNK-fpY3%3D0Ls5yRZVN%2BCJkaWYBUobg%40mail.gmail.com.

[Arjan Scherpenisse]

So the idea here is that some tests "pollute" other test cases because they have side effects; leave traces of their run in the system.

My initial reaction would be that in the BEAM and Elixir this issue is not so prominent as in other more dynamic languages, because we (in general) don't do monkeypatching or other crazy runtime stuff, and database tests are already isolated from each other through the sandbox.

However I have been bitten by this in some cases, for instance depending on `Application.get_env`, or on which OTP applications are started or stopped.

In general, having issues like this is a symptom of having test cases that do not clean up after themselves like they should, I believe. I'm not sure that this should be a part of the core.

Arjan

On Wednesday, April 17, 2019 at 8:03:03 PM UTC+2, Paulo D Gonzalez wrote:

Hi Andrea! Thanks for the quick reply.

Here is some info on bisect: 

https://relishapp.com/rspec/rspec-core/docs/command-line/bisect

Here is where the implementation of the lib I used to use:

https://github.com/rspec/rspec-core/tree/master/lib/rspec/core/bisect

This was useful when tests would fail in a mysterious way. This would give us some added introspection when it succeeded. It felt like a `git bisect` that was done automatically for you.

On Wed, Apr 17, 2019 at 12:52 PM Andrea Leopardi <an.le...@gmail.com> wrote:

I can't recall if this has been discussed before but I don't think it has. Could you expand in more detail what --bisect does?

Andrea Leopardi

an.le...@gmail.com

On Wed, Apr 17, 2019 at 7:38 PM Paulo D Gonzalez <pdgonz...@gmail.com> wrote:

In my previous life ( :) ) I was able to use something that would be equivalent to `mix test --bisect --seed 776` and it used to be very helpful when dealing with a big codebase with crazy test failures.

I was wondering what are your thoughts on this feature in Elixir? I tried looking at existing discussions on this and didn't find any. If there are, could you please point me to it?

Thank you.

Paulo

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-l...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/7bf0827d-1174-4aec-ba08-58e88de6c2de%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elixir-lang-core" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elixir-lang-core/rv1A8x4DBvo/unsubscribe.

To unsubscribe from this group and all its topics, send an email to elixir-l...@googlegroups.com.

[Andrea Leopardi]

Yep I am not sure about this being in core either. From a wild guess, I think this would not be trivial to implement so it might be something to maintain in a separate library.

Andrea

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/e006ad91-8794-48e9-a969-9d53a2ecf788%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Andrea Leopardi

an.le...@gmail.com

[Paulo D Gonzalez]

Thanks Arjan and Andrea.

To both your points, I agree, this is not something to be in `core`. I thought I'd ask since `ExUnit` is most likely the most developed testing lib we have and this was the best channel I found to ask questions about it.

Arjan: Completely agree with it being symptoms of a bigger problem. But, when you don't have control of the start of a project and are faced with responsibilities coming into a project, you tend to reach for all tools available to make sense of things. I'm sure you've been there before :) I do think the starting and stopping applications explicitly may positive results. I'll investigate further.

Anyways, thanks for the reply. I just thought I'd ask because it was the `coolest` feature I remember from the old testing framework even though it was never used if things were set up properly.

Thanks again!

PG

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/e006ad91-8794-48e9-a969-9d53a2ecf788%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Andrea Leopardi

an.le...@gmail.com

[Myron Marston]

I authored the `--bisect` feature in RSpec.  It's a useful and well-loved feature, but I don't think it is nearly as useful in Elixir.  One of the big problems is ExUnit's async tests: bisection works well when your tests run sequentially, but as soon as you have any concurrency you've thrown out the determinism on which bisection depends.  Given how common `async: true` tests are in Elixir, I don't think that a bisection tool is going to useful very often for Elixir.

[Paulo D Gonzalez]

That's a very fair and strong point. Completely agree. Thanks!

 By the way, I loved `--bisect`! Was definitely one of my favorite tools when we had to put a test suite in order a couple of jobs ago. Thanks for writing it! 

To unsubscribe from this group and all its topics, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/a00c3942-129e-4bca-b8b0-43584cc5cb30%40googlegroups.com.

[Rich Morin]

> On May 19, 2019, at 20:24, Myron Marston <myron....@gmail.com> wrote:
>
> Given how common `async: true` tests are in Elixir, I don't think that a

> bisection tool is going to [be] useful very often for Elixir.

If I'm not mistaken, bisection is generally used to track down problems
that have been discovered in some other manner. In this situation, the
speed of the tests may not be as critical as determining the source of
the problem. So, perhaps Elixir's bisection code could override the
`async: true` mode when it is used ...

-r

[Norbert Melzer]

On Mon, May 20, 2019 at 6:00 AM Rich Morin <r.d....@gmail.com> wrote: 

So, perhaps Elixir's bisection code could override the
`async: true` mode when it is used ...

Which in turn might cause the bug not to happen anymore. 

[Myron Marston]

Exactly.  The normal workflow for using --bisect in RSpec is:

• Your test suite is setup to run in random order, and prints out a seed on each run
• If a test fails due to an ordering dependency for a particular ordering, you can reproduce it with `rspec --seed <the same seed>`.
• However, this runs the entire test suite; what is even more useful is a command to run the minimal subset of the suite that reproduces the issue (e.g. typically one passing test that leaks a state change that in turn causes a later test to fail).  To get this you can run `rspec --seed <the same seed> --bisect`.  It gives you that minimal command after repeatedly running subsets of the test suite using the same ordering.

This works great for a test framework that does not run tests concurrently.  With ExUnit, if you have any `async: true` tests, and you get a surprise test failure during a normal run that you suspect is due to an ordering dependency, a tool like `--bisect` wouldn't be likely to help.  There is no deterministic order that the async tests ran in that `--bisect` could use to reproduce the issue as it runs smaller and smaller subsets.  If you wanted a tool like `--bisect`, I think you'd have to decide not to ever run your tests concurrently, which is a trade off that I suspect most Elixir programmers would not be willing to make.

Myron

--

You received this message because you are subscribed to a topic in the Google Groups "elixir-lang-core" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elixir-lang-core/rv1A8x4DBvo/unsubscribe.

To unsubscribe from this group and all its topics, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CA%2BbCVsvoaE6LWYQsgBj%3Dwx3q9%2BQ-QGPb2XZLz5qTHkeqFo_aiA%40mail.gmail.com.