ExUnit becomes slow

[miaout17]

I found that ExUnit becomes very slow since this commit:
https://github.com/josevalim/elixir/commit/4b3255823fbf4acb0cfc

Erlang.init.stop is the correct behavior, but it seems make the Erlang
VM terminate slowly.
The key problem is in the code_test.exs. It has quite a lot OS.cmd
call that runs Erlang VM.
However, it is not run in parallel because they are all in the same
testcase class.

I think there are several to improve this:

1. Let code_test.exs be a special case. Write spawn_link inside it and
let it run in parallel.

2. Make ExUnit::Runner parallel in test function level, not test
class.
And let every test function has only one OS.cmd call.

Which one do you prefer?

[José Valim]

Yes, I have noticed it. :)

I don't think the first alternative is going to work, because, even if we spawn, we would need to wait the spawned process to finish, which would cause the same slowdown that we have today. Or am I missing something?

Anyway, what if we simply break the current tests in different test cases?

object Code1Test

  proto ExUnit::Case

  def one_test

    # ...

  end

end

This will look a bit lame now, but once we are able to define new objects behind the scene (as in Ruby's Class.new), we could have something like below and the result would be the same:

  context do

    def one_test

      # ...

    end

  end

What do you think?

Glad to hear back from you!

José Valim

www.plataformatec.com.br

Founder and Lead Developer

[Ling YC]

The first idea is write multiple OS.cmd in a test function

Process.spawn -> OS.cmd(...)

Process.spawn -> OS.cmd(...)

Process.spawn -> OS.cmd(...)

I think the process will run in parallel, but it's not a good design. 

I think there are 2 level of parallelism we can do: 

(1). Parallel in "TestCase class level". Spawn a process for each TestCase class. 

(2). Parallel in "test function level". Spawn a process for each test function. 

If we chose (2), we just need to put one OS.cmd in one test function, and it automatically runs in parallel. 

If we chose (1), we need to break the OS.cmd calls into different TestCase classes. 

Am I missed something?

I'm not sure if Code1Test, Code2Test...make sense. 

In my imagination, we should collect related test cases into a TestCase (TestSuite?) class. 

Anyway, the "context" syntax sounds good.  

[José Valim]

The first idea is write multiple OS.cmd in a test function

Process.spawn -> OS.cmd(...)

Process.spawn -> OS.cmd(...)

Process.spawn -> OS.cmd(...)

Yes, but after running OS.cmd() we need to make assertions, if we write the tests like this (using this hypothetical method):

"result" = Process.spawn_and_return_result -> OS.cmd()

It wouldn't help a lot because we would have to wait for the result anyway. If we make the assertion inside Process.spawn, the assertion could fail but it would never notify the ExUnit. And if we use spawn_link and make the assertion inside, it would likely kill the running process, causing ExUnit to crash completely.

I think there are 2 level of parallelism we can do: 

(1). Parallel in "TestCase class level". Spawn a process for each TestCase class. 

(2). Parallel in "test function level". Spawn a process for each test function. 

If we chose (2), we just need to put one OS.cmd in one test function, and it automatically runs in parallel. 

If we chose (1), we need to break the OS.cmd calls into different TestCase classes. 

Yes, this is right. I have chosen to have TestCase in parallel (1) because it is much less likely to have conflicts. If we start to run several tests in the same TestCase in parallel (2), it is very likely to increase the chance of conflicts as they are testing very similar things. For example, if you are testing a named process, it is likely you will start this named process in the setup method and then all tests will conflict because they are starting the same named process.

 

I'm not sure if Code1Test, Code2Test...make sense. 

In my imagination, we should collect related test cases into a TestCase (TestSuite?) class. 

Yup, it does not make sense. :) It would be a hack until Elixir provides a functionality that allows us to define objects behind the scenes.

Anyway, the "context" syntax sounds good.  

With this in mind, do you think it is still worth to give a shot to parallel tests (2)?

[miaout17]

On 4月30日, 下午3時10分, José Valim <jose.va...@gmail.com> wrote:
> With this in mind, do you think it is still worth to give a shot to parallel tests (2)?

I considered again. Now I think (2) is not very needed.
In most case class-level parallel is enough, and "context" can handle
special cases like long task in CodeTest.

[José Valim]

Fixed then:

https://github.com/josevalim/elixir/commit/de39c45d26d82376db032e045e0e35aac9b059d3

José Valim

www.plataformatec.com.br

Founder and Lead Developer

[Grant West]

Is there a modern solution so this? If a Case necessarily has several unavoidably slow tests, is there any way to run them in parallel? For TDD purposes, adding even 5 seconds to total test duration is very undesirable. Did the "context" solution ever come to pass? What is the modern solution? Would you be open to something like "use ExUnit.Case, parallel: true"?

I understand that defaulting to parallel would break many things. But there are many, if not the majority of cases were parallel tests would be fine. Most test suites don't access global state. Having the OPTION to enable parallel tests where it just isn't feasible to speed tests up would be a good tool.

[Aleksei Matiushkin]

Use `use ExUnit.Case, async: true` https://hexdocs.pm/ex_unit/ExUnit.html

Also, for TDD one might run the currently-under-development tests with `mix test test/foo_test.exs:5` or using tags https://hexdocs.pm/mix/Mix.Tasks.Test.html

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/157d0acc-e8fe-4573-9715-4ba014b58e9dn%40googlegroups.com.

--

Aleksei Matiushkin, Software Engineer - R&D

 

Office (+34) 935 679 834

 

8 Devonshire Square, London, EC2M 4PL, United Kingdom

Torre Mapfre, Planta 22, Marina, 16-18, 08005 Barcelona, Spain

kantox.com

  

        

 

Kantox Limited is a UK private company with registered company number 07657495 and registered address at 8 Devonshire Square, London EC2M 4PL, United Kingdom. We are authorised with the UK Financial Conduct Authority (FCA) under the Payment Service Regulation 2017 as a Payments Institution (FRN 580343) for the provision of payment services and with HMRC as a Money Service Business Registration No.12641987.

Kantox European Union, S.L.  is a Spanish private company with tax ID number B67369371 and registered address at Torre Mapfre, Planta 22, Marina, 16-18, 08005 Barcelona, Spain. Kantox is authorized by the Bank of Spain, with registration number 6890, which is the supervisor of the Spanish banking system along with the European Central Bank. Additionally, we are supervised by SEPBLAC, the Supervisory Authority for the prevention of money laundering and terrorist financing in Spain.

KANTOX is the Controller for the processing of data in accordance with the GDPR and LOPDGDD for the purpose of maintaining a commercial relationship. You may exercise your rights of access and rectification, portability, restriction and opposition by writing to KANTOX to the email: gd...@kantox.com. You have your right to make a complaint at www.aepd.es.  

[Sabiwara Yukichi]

I think the point raised here is to run tests in parallel within a `ExUnit.Case`. As mentioned in the docs:

  The individual tests within each test case are still run serially.

This article uses a workaround to make this happen by generating one module per test, but it shows some nice speedups in their example (from 460s (async) to 70s).

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGF5_6eS62H_EQ7vEj_OyaJqUBuYT4dsFSL%3D_uPGNR2N4%2BtUxQ%40mail.gmail.com.

[José Valim]

Yes, we are interested in incorporating those improvements. Discussion here: https://github.com/elixir-lang/elixir/pull/11949

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CANnyohZhjR_SV6OED3%2BeyzS_26ES5CAiR_eP6JhjEU472edyxg%40mail.gmail.com.