Unit testing applications that use Ecto

[Adrian]

Hello,

For an application having multiple Ecto based models, what would be the best approach for isolating the Ecto dependency for unit testing. My initial approach was to create a mock based on Ecto.Repo behavior and in my App.Repo call "use" dynamically based upon configuration. But i'm not sure if it would be better maybe to make the split higher up on the model level.

[José Valim]

This has been discussed in the past but my advice is simply don't.

Ecto already decouples the model from the repository. You can manipulate changesets, perform validations and so on without ever touching the repository. Therefore, you can test all of those things directly and concurrently. On the other hand, there is no way to effectively test things like queries without going through the repository so although you do need the repository in this case, mocking makes no sense.

My advice here is: keep all "pure" code, code that does not depend on the Repo, in the model and have one place where you are going to do all side-effects and database operations. In a Phoenix application, the place for side-effects would be the controller that would invoke many "services" like the repository, sending e-mails and so on. You will write controller tests as integration tests because the job of the controller is *to integrate layers*. Mocking the repo in controller tests make no sense because, every time you mock your own component, you still need an integration test that guarantees everything works, so what is the mock giving you anyway?

To sum up:

* models - is about data transformation. You will build changesets and queries but never reach a service, like the repo or e-mails, from your model

* controllers - is about integrating layers. It gets data from models (like changesets, queries) and send them to services (talking to the database, sending e-mails, etc). Controllers should be kept simple because it should be only about integrating layers

* views - is also about data transformation. In this case, transforming the controller assigns (current user, fetched models) into templates/strings

Which means that, in terms of testing we have (let's use User as an example):

* test/models/user_test.exs - you will test your changeset and others. Again, only data transformation, tests can be "async: true" because it doesn't talk to the database.

* test/models/user_repo_test.exs - you will test anything the model returns that needs the repository to be tested, like complex queries. Here it makes no sense to mock because you *need* the repo, testing a complex query against a fake repo makes no purpose. Also, since it depends on the Repo, tests cannot be concurrent (until Ecto 1.1 where we solve this problem)

* test/views/user_test.exs - you will test how your views are rendered. Again, it is only data transformation, so tests can be "async: true" because it doesn't talk to the database.

* test/controllers/user_controller_test.exs - integration layer. You'll test a simple pass through the different layers and the expected result. Important: you are not going to test "models" and "views" combinations here. For example, you won't test that when you POST attributes A and B, field C will be added to the model. That's is going to be in the model test. In the same way you are not going to test how users with different attributes are rendered. That's in the view test.

 

We hope to share more on this soon, specially in the Programming Phoenix book. Also read http://blog.plataformatec.com.br/2015/10/mocks-and-explicit-contracts/ for more info on mocks.

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Director of R&D

On Tue, Oct 20, 2015 at 3:46 AM, Adrian <adr...@releasequeue.com> wrote:

Hello,

For an application having multiple Ecto based models, what would be the best approach for isolating the Ecto dependency for unit testing. My initial approach was to create a mock based on Ecto.Repo behavior and in my App.Repo call "use" dynamically based upon configuration. But i'm not sure if it would be better maybe to make the split higher up on the model level.

--
You received this message because you are subscribed to the Google Groups "elixir-ecto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-ecto...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-ecto/f1a39826-753c-4676-baef-c4b272eb95a3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ringo De Smet]

José,

I work with Adrian on this one. We want to have as much as possible unit tests which don't touch the database and run super fast.

In your answer, I follow your reasoning. It's just not clear how to do this in practice. Probably, it's more of a "best practices" problem. Are there good code samples available? How do you structure everything so I can tag all the tests which do go to the repository? Similar to how you tag the tests which invoke the real Twitter api in the blog article you mention.

Is this setup described more thoroughly in the beta ebook already? If so, I buy it instantly. :-)

https://pragprog.com/book/phoenix/programming-phoenix

Ringo

[Drew Olson]

I want to whole-heartedly agree with everything José recommended. This is how I've tested my Phoenix applications that use Ecto and it works great in practice. If you're curious what these "full stack" tests look like, I've got an example[1] project with a few samples.

[1] - https://github.com/drewolson/blox/blob/master/test/controllers/post_controller_test.exs#L8-L20

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-ecto/3ca2b35b-b81b-41a9-8b33-9aabf3eb96a7%40googlegroups.com.

[José Valim]

Hi Ringo!

 

I work with Adrian on this one. We want to have as much as possible unit tests which don't touch the database and run super fast.

Right. That's what my e-mail was trying to make clear: Ecto already guides you towards what can be unit tested and what cannot. Changesets, models and so on are just data so you can unit test them without the repository. However, it makes no sense to test a complex query without a repository. Similarly, it makes no sense to test a controller with a mock repository because the controller is meant to be an integration layer, it integrates complex things like the conn and repository, while we push the simple data-oriented things to models and views.

As with any code, writing good tests need to juggle many things, including correctness, performance and readability. If you are writing more tests than you should or you have poor coverage, in favor of performance, you are going to be in trouble at some point anyway.

Of course, nobody writes slow and fragile suites on purpose, but it is a side-effect that shows up when we excessively focus on one aspect over the other. For example, in Rails we see a lot of slow test suites for a bunch of different reasons:

1. too many integration tests due to lack of confidence in the over mocked unit tests

2. over reliance on factories and database state

3. lack of concurrent tests (possibly but far from easy in Rails)

4. following blind rules like "one assertion per test" which may run an expensive setup over and over again

All I am saying is: don't push towards unit tests because you want your tests to be faster. Use unit tests because they are the proper tests to write for a particular boundary.

In your answer, I follow your reasoning. It's just not clear how to do this in practice. Probably, it's more of a "best practices" problem. Are there good code samples available? How do you structure everything so I can tag all the tests which do go to the repository? Similar to how you tag the tests which invoke the real Twitter api in the blog article you mention.

I tried to answer this question in my original e-mail when it comes to how to organize the code in MVC and how I would test each part. Unfortunately I don't have any public code samples yet maybe someone has some to share.

Regarding the tag, you can simply tag each test case that uses the repository with "@moduletag :repo". But they should be easy to spot regardless if you follow the best practices in my previous e-mail: they will be the controller tests and all tests in model with the _repo_test.exs prefix.

 

Is this setup described more thoroughly in the beta ebook already? If so, I buy it instantly. :-)

https://pragprog.com/book/phoenix/programming-phoenix

We are going to cover some of those topics on the testing chapter which is not out yet.