Proposal/discussion: Umbrella app lazy loading applications when running tests

[Dylan Chong]

I’ll start off by stating the problem that I’m having.

Basically, unit tests take too long to start up for a ideally superfast TDD cycle. This is because, in our umbrella app, we have many micro services , Each being their own app, and each standing up various other applications like Phoenix endpoints, Prometheus metrics, Various libraries doing things as their app starts up.

I have already done my best to reduce the amount of work being done at applications that up time, but there is still a delay of about six seconds (plus compilation times) for a single hello world test to run.

I have noticed that these two ways of running the test are not equal, especially in terms of speed.

mix test apps/utils/test/hello.exs

cd apps; mix test test/hello.exs

In my umbrella app at work, the first will take about six seconds to run (after repeated runs, to ignore compilation times) but the second will take 1.5. This is because the second option does not start up other umbrella apps

So why not always just do option two? Firstly, umbrella app dependencies Will not be seen unless mix is run from the top level umbrella - dependency overrides for Erlang libraries will appear to be unresolved, and mix test.watch will be unavailable if defined in the top level umbrella mix file.

possible solution: When running tests in an umbrella app, don’t start up other umbrella apps until running a test that is either inside it or depends on it via the test app’s mix file. If this is possible, it would still optimise test start up times when running mix test with —stale or —failed options

Looking forward to hearing thoughts!

[Adam Lancaster]

If the tests are unit tests I wonder if you need to start the app at all?

Could you run them with `mix test —no-start` ?

You could also explore having a dedicated unit_test env to aid this perhaps

Best

Adam

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/2e3515d8-055b-4dbe-9a2e-2c1e4cfd2a1fn%40googlegroups.com.

[Dylan Chong]

Hey cheers for the quick response

I’ve done my best to reduce the bumber of apps starting up.

—no-start prevents mimic from starting up so i cant actually run any tests

A hacky `Application.ensure_all_started(:mimic)` in the test_helper.exs got it running

Many tests often touch the DB so even if mimic could start up, I’d have to start up the app to start up the ecto repo, which would trigger a cascade start of the app anyway.

Makes me wonder what the intention was for ExUnit and what tests should be written using it.

But being restricted to pure unit only is very restrictive. Low key integration tests are really needed in a lot of cases.

> You could also explore having a dedicated unit_test env to aid this perhaps

How are you thinking this should be approached? As in custom MIX_ENV value? Or ExUnit tags? Is this documented anywhere?

You received this message because you are subscribed to a topic in the Google Groups "elixir-lang-core" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elixir-lang-core/oaEMVC1uFtU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/0F010B35-5F78-4F4B-AC85-F25384F3766C%40a-corp.co.uk.

[Adam Lancaster]

Originally you said 

> Basically, unit tests take too long to start up

But the tests you are talking about sound like a mix of both integration tests (as they hit the db for example) and unit tests.

ExUnit is great for both kinds of tests and both kinds of tests are valuable to have but my suggestion was talking unit tests only as they are what you mentioned.

You can speed them up easily because by virtue of them being unit tests you’ll know they won’t interact with any external services like a db and are unlikely to need the app to actually start in order to run.

As you discovered you can add to the test_helper any deps you did need started (like Mox for example). So my suggestion for speeding up unit tests only was run them without actually starting the app.

In that world you could make a unit_test folder to put them in, add a new MIX_ENV and a mix unit_test command that runs them. But again this would only help speed up unit tests.

For integration tests you won’t be able to do that as your app will have to start.

I’m curious do you see a speed difference if you run your tests like this:

` mix cmd mix test --color`

Best

Adam

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/53B886FC-9D0C-418C-AC9B-3FB105740101%40gmail.com.

[Dylan Chong]

Originally you said 

> Basically, unit tests take too long to start up

Sorry yep, I was not precise with my words (unit and integration).

To clarify, we do have unit tests, but we have a large amount of component tests (probably a better term for what I meant when I said integration, although I guess component tests include some level of integration).

These component tests are often things like EntityCRUD stuff.

Or tests for background jobs that touch the DB.

> For integration tests you won’t be able to do that as your app will have to start.

Yeah, this is the bit were lazy loading umbrella apps (or rather, being more selective about what apps start based on which tests are running) would be handy.

` mix cmd mix test —color`

Yeah running an individual app prevents running starting up other apps, so yeah, it is faster.

The downside to this method is not being able to combine it with `mix test —stale` or `—failed` so that tests from multiple apps can run

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/25A24A28-6689-4C81-9246-F71248D22BBA%40a-corp.co.uk.