Speeding up Rails specs

[Scott Harvey]

Over the years I've gotten into the habit of saving all the associated objects I need in order to test a specific method.

This means I would end up with a situation something like this:

class Order < AR

  has_many :line_items

  

  def total_cost

    line_items.paid.sum(&:cost)

  end

end

class LineItem < AR

  belongs_to :order

  

  named_scope :paid, :conditions => { :paid => true }

end

it 'total_cost returns the total of all the line items' do

  order = Order.create

  order.line_items.create!(:cost => 10, :paid => true)

  order.line_items.create!(:cost => 20, :paid => true)

  order.line_items.create!(:cost => 50, :paid => false)

  order.total_cost.should == 30

end

In an effort to get my tests running quicker I would like to be able to test this type of situation without saving to the database.

I've heard it said that you should never stub the object under test but instead mock and stub associated objects as needed. I'm just not sure how to go about stubbing the LineItem class in this case without rendering the test useless.

Anyone have any thoughts on how this could be done or any more general tips on speeding up Rails tests?

Scott

[Gareth Townsend]

The real problem here is that you're testing two things. You're testing that the named_scope returns paid items, and that you can sum the cost of each of those items.

If your total_cost method continues to use the named scope, then you won't be able to get away from using the database, without stubbing out the named_scope itself, which would be stubbing the object under test.

--
You received this message because you are subscribed to the Google Groups "Ruby or Rails Oceania" group.
To view this discussion on the web visit https://groups.google.com/d/msg/rails-oceania/-/Do3fM2YDVc0J.
To post to this group, send email to rails-...@googlegroups.com.
To unsubscribe from this group, send email to rails-oceani...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/rails-oceania?hl=en.

[Scott Harvey]

Gareth,

The paid named_scope in the example is on the LineItem model so stubbing that out would not be stubbing out the object under test.

The test that checks that the paid named_scope works will be in the LineItem spec folder so I'm happy to stub the named_scope out if there is a nice way to do that.

Scott

[Gareth Townsend]

class Order < AR
has_many :line_items

def total_cost
line_items.paid.sum(&:cost)
end
end

class LineItem < AR
belongs_to :order

named_scope :paid, :conditions => { :paid => true }
end

it 'total_cost returns the total of all the line items' do
order = Order.create
order.line_items.create!(:cost => 10, :paid => true)
order.line_items.create!(:cost => 20, :paid => true)
order.line_items.create!(:cost => 50, :paid => false)
order.total_cost.should == 30
end

On 26/08/2011, at 2:09 PM, Scott Harvey wrote:

> The paid named_scope in the example is on the LineItem model so stubbing that out would not be stubbing out the object under test.

I clearly need more coffee! (or is it less coffee?)

> The test that checks that the paid named_scope works will be in the LineItem spec folder so I'm happy to stub the named_scope out if there is a nice way to do that.

In that case, something like this would completely avoid the database: (I haven't checked the syntax at all)

it 'total_cost returns the total of all the line items' do

order = Order.new
line_items = [stub(:cost => 10), stub(:cost => 20)]
order.stub(:line_items).and_return(stub(:paid => line_items))
order.total_cost.should == 30
end

You only need to stub the bits you're using, hence no paid methods on the line_item stubs. You also don't need to save the order object, you can just use the in memory representation.

[Dmytrii Nagirniak]

 def total_cost
   line_items.paid.sum(&:cost)
 end
end

Why would you calculate the sum in-memory instead of the database? So that you can stub it?

[Malcolm Locke]

It looks like stub_chain might help:
http://apidock.com/rspec/Spec/Mocks/Methods/stub_chain

I.e. (untested):

mock_line_item = mock("line_item")
mock_line_item.stub(:cost).and_return(10)
Order.stub_chain(:line_items, :paid).and_return([mock_line_item])

Still feels wrong though ...

Malc

[Malcolm Locke]

On Fri, Aug 26, 2011 at 04:39:09PM +1200, Malcolm Locke wrote:
> mock_line_item = mock("line_item")
> mock_line_item.stub(:cost).and_return(10)

> #Order.stub_chain(:line_items, :paid).and_return([mock_line_item])
order = Order.build
order.stub_chain(:line_items, :paid).and_return([mock_line_item])

Is more likely to work

Malc

[Scott Harvey]

>      order.stub_chain(:line_items, :paid).and_return([mock_line_item])
>
> Is more likely to work
>
> Malc

As you mentioned it still feels a bit wrong as you end up testing the
implementation of the method instead of the behaviour.

It seems to be a choice between hitting the database and having slow
tests or stubbing out the method chain and having brittle tests.

Either way I'm not completely happy with the resulting test suite.

I would have thought this situation would be fairly common in a rails
application, how are people handling this currently?

Scott

[Dmytrii Nagirniak]

I would have thought this situation would be fairly common in a rails
application, how are people handling this currently?

I would just hit the database. Correct behavior here would be more important than couple of milliseconds saved.

It reminds me 2 things:

1. When rails 3  was released it could not boot even though all tests passed. The reason was that boot-up process was stubbed (don't remember exact details though).

2. you can't stub one of the most powerful things - plain SQL. I saw people stubbing it and checking SQL that would be executed. It might work, but it is something that is too much detached from real situation.

I might be in minority, but I prefer to stub as little as possible. Most of it would be something without consistent state (database has consistent state), availability etc. Not necessarily external services (http requests), but internal as well (Time).

[Andrew Grimm]

Named scope is newer than Rails 1.2, so forgive me asking:

Is named scope incapable of working without hitting the database, or should it work fine in theory unless you've got SQL in it?

Andrew

--
You received this message because you are subscribed to the Google Groups "Ruby or Rails Oceania" group.

[Scott Harvey]

> I would just hit the database.

That's the conclusion that I reached a while back as well which is why
I've been doing it for so long.

The thing that annoys me is I read through a book like Continuous
Testing (http://pragprog.com/book/rcctr/continuous-testing) and they
say you should be aiming for your tests to run fast enough to have
hundreds running per second.

That's all well and good for code that isn't hitting external services
(database, file system) but within a rails application it just doesn't
seem at all feasible and still keep a solid test suite.

If anyone has any more tips on when to mock/stub within a rails
application or how to speed up a test suite I would love to hear it.

Scott

[Richard McGain]

You could use an in-memory database if all you require is speed, but it is an all or nothing solution, not one that is isolated to these specific tests. If you have loads of fixtures / dump a lot of stuff into your testing db it also might not be appropriate for memory usage reasons.

Not sure which db you are using so I won't point out any specifics(I just googled for them anyway).

Richard McGain

--
You received this message because you are subscribed to the Google Groups "Ruby or Rails Oceania" group.

[Dmytrii Nagirniak]

Is named scope incapable of working without hitting the database

The Arel is all about SQL (relational algebra), so I can't think how it can work without DB.

But you can stub the scopes, connection or something else. Then it won't be a scope anymore. It is something totally different.

 

[fresh...@gmail.com]

On 26 August 2011 15:03, Dmytrii Nagirniak <dna...@gmail.com> wrote:

1. When rails 3  was released it could not boot even though all tests passed. The reason was that boot-up process was stubbed (don't remember exact details though).

[citation needed]

(My main objection is with the word 'released', as if the core Rails team would release a new version without even using it themselves!)

--
James

[Dmytrii Nagirniak]

The thing that annoys me is I read through a book like Continuous
Testing (http://pragprog.com/book/rcctr/continuous-testing) and they
say you should be aiming for your tests to run fast enough to have
hundreds running per second.

I don't take it all too literally.

The FIRE acronym is advocated in the book. Where
F=Fast

I=Informative

R=Reliable

E=Exhaustive

To me, stubbing contradicts the R (you replace part of YOUR system; how can the testing be reliable as it never runs your code).

FAST means that "the tests in our suite need to be fast, so that we can run them after every change".

And to me the main barrier currently is Rails start-up time rather than time to run a test.

With Autotest/Guard you can run your test after you save the file. It is fast enough.

The book itself describes how to break dependency doing HTTP request to Twitter. This is totally fine.

But I would love to hear how people really break dependency on database in Rails.

You probably need to explicitly separate data access layer and the logic.
It would at least double the amount of code and overcomplicate it.

In turn you would need to write more test (by factor of 2-3?).

So you would end-up with the system of 5000 tests that run for 60 seconds.

Or you can have less complex system with same test coverage and 2000 tests that run for 60 seconds.

I would probably choose 2nd.

(P.S. I am exaggerate a bit, but you get the point).

[Gareth Townsend]

> To me, stubbing contradicts the R (you replace part of YOUR system; how can the testing be reliable as it never runs your code).

And in certain circumstances this is exactly what you want to do, because you don't care about that part of your system.

Lately I've been doing the following:

Monkey patch ActiveRecord in spec_helper to raise if you try and write to the database, this forces you not to use it. It changes the way you structure some of your code, usually for the better, but ultimately it means you have a very fast unit test suite. These tests are fast, hundreds of tests per second fast.

Then in Cucumber I do everything through the user interface. If we need a user, we run the sign up process, then test whatever it was that required the user. Nothing gets into the database unless it came through the front end. This is slow, but exhaustively tests the full software stack. I tag every cucumber test with the model names and/or feature name involved, so that I can run the tests that I think might break locally before pushing, then I let a build server handle the full suite.

So far it's been working out well.

[Dmytrii Nagirniak]

As I said I don't remember the all the details. Now I doubt it was Rails 3, probably 2. But I agree that 'release' word is inappropriate. Beside that, this is what meant:

"Although the Rails initializer tests covered a fair amount of area, successfully getting the tests to pass did not guarantee that Rails booted"

http://yehudakatz.com/2009/07/01/new-rails-isolation-testing/

[fresh...@gmail.com]

Stubbing does not have to contradict the 'R'. Stubbing isolates of the code under test. This is great for unit tests, where a *unit' of code is the thing being tested. *Not* every layer under it. If one of my unit tests fails, it's because the code is broken *at the level that I'm testing*, not deeper in the stack.

That does not mean never running automated tests with the full stack - those are *integration* tests.

However, due to Rail's tight coupling of the models and the database layer I will concede that there are times when you have no choice but to hit the database (e.g. for testing CRUD operatios and testing the result of a named scope that relies on an aggregation performed in a database query - you need to test that named scope, and if the work is happing in the DB then you'll need it for the unit test)

--

You received this message because you are subscribed to the Google Groups "Ruby or Rails Oceania" group.

To post to this group, send email to rails-...@googlegroups.com.
To unsubscribe from this group, send email to rails-oceani...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/rails-oceania?hl=en.

--
James

[Scott Harvey]

> Monkey patch ActiveRecord in spec_helper to raise if you try and write to the database, this forces you not to use it. It changes the way you structure some of your code, usually for the better, but ultimately it means you have a very fast unit test suite. These tests are fast, hundreds of tests per second fast.

Any chance you can share that patch so I can try it out for myself?

Scott

[Scott Harvey]

> However, due to Rail's tight coupling of the models and the database layer I
> will concede that there are times when you have no choice but to hit the
> database (e.g. for testing CRUD operatios and testing the result of a named
> scope that relies on an aggregation performed in a database query - you need
> to test that named scope, and if the work is happing in the DB then you'll
> need it for the unit test)

I guess this might be an argument for restructuring your Rails
application so your models stick to the Single Responsibility
Principle.

An approach to this was outlined by Yehuda in this Stack Overflow answer.

http://stackoverflow.com/questions/1068558/oo-design-in-rails-where-to-put-stuff/1071510#1071510

I haven't tried this myself or seen any other applications using this
pattern but it sounds good in principle and would decouple a lot of
classes from the database layer.

Scott

[Gareth Townsend]

Sure.

https://gist.github.com/1172821

There might be a nicer way to do this, but this does the trick for postgresql.

> --
> You received this message because you are subscribed to the Google Groups "Ruby or Rails Oceania" group.

[Brian Guthrie]

I've seen teams handle the issue of test suite speed and database
connectivity in ActiveRecord a number of different ways. Here are four
of them, and some tradeoffs.

- Stub the object under test. Benefits: Fast, simple. Risks: Stubbing
the object under test means that you're no longer testing the object
you think you're testing. This is a bad idea. It also
allows/encourages you to test implementation rather than behavior,
which is also a bad idea, because it makes your tests brittle.

- Stub database interactions at the driver level. Benefits: It's
fast. It allows for "true" unit tests. You don't have to stub the
object under test. Risks: It's a pain in the ass. See unit_record and
more modern friends. I've been on a project that did this with a large
test suite and I didn't much value the feedback those tests gave me.
ActiveRecord is too tied to the underlying database.

- Hit the database, but use a faster in-memory database (for example,
sqlite instead of mysql). Benefits: Speed. Risks: Slight differences
in the underlying database can ruin your tests, and your day. But if
you're not doing anything too implementation-specific then this can
work well.

- Hit the database as normal but parallelize your tests across
multiple processors and/or machines. This is my preferred approach
because it has the least impact on the test suite. Benefits: Speed, no
need to modify existing tests. Scales well as you add hardware. Helps
more than just database tests. Risks: Setup is annoying, potentially
difficult to configure, test runs may occasionally fail for funky
reasons (network I/O, e.g.).

HTH,

Brian

> --
> You received this message because you are subscribed to the Google Groups "Ruby or Rails Oceania" group.

[fresh...@gmail.com]

So you keep your Rails models as simply a data-access later, and move all domain logic to a mirrored set of classes that are easy to unit test in isolation?

--
You received this message because you are subscribed to the Google Groups "Ruby or Rails Oceania" group.
To post to this group, send email to rails-...@googlegroups.com.
To unsubscribe from this group, send email to rails-oceani...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/rails-oceania?hl=en.

--
James

[Scott Harvey]

> So you keep your Rails models as simply a data-access later, and move all
> domain logic to a mirrored set of classes that are easy to unit test in
> isolation?

Yeah, that's how I understand it.

I think naming the mirrored classes might get annoying but I'm sure
you could come up with some system that would work.

The other benefit I can see it that your 'mirrored' classes will only
implement the methods you need for your application so you would have
a clearly defined DSL for the application.

Again, it all sounds good in theory but I'm yet to see it in practice.

Scott

[Scott Harvey]

Brian,

I could probably see a situation where you could use a fast database
locally like sqlite but then have your integration server (if you have
one) running your production database setup.

Scott

[fresh...@gmail.com]

On 26 August 2011 16:28, Scott Harvey <scottand...@gmail.com> wrote:

> So you keep your Rails models as simply a data-access later, and move all
> domain logic to a mirrored set of classes that are easy to unit test in
> isolation?

Yeah, that's how I understand it.

I think naming the mirrored classes might get annoying but I'm sure
you could come up with some system that would work.

The other benefit I can see it that your 'mirrored' classes will only
implement the methods you need for your application so you would have
a clearly defined DSL for the application.

This is how a lot of Java apps are architected. I don't mean that in a bad way at all. It's a valid idea.  I might give it a try on my next project.

 

Again, it all sounds good in theory but I'm yet to see it in practice.

Scott

--
You received this message because you are subscribed to the Google Groups "Ruby or Rails Oceania" group.
To post to this group, send email to rails-...@googlegroups.com.
To unsubscribe from this group, send email to rails-oceani...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/rails-oceania?hl=en.

--
James

[Scott Harvey]

> This is how a lot of Java apps are architected. I don't mean that in a bad
> way at all. It's a valid idea.  I might give it a try on my next project.

I finally managed to find the article I read that suggested this
pattern, worth a read.

http://solnic.eu/2011/08/01/making-activerecord-models-thin.html

Scott

[Gregory McIntyre]

I second Gareth's approach although I've never managed to be that
"pure". Next project, I am totes going to do that RSpec
raise-on-db-access thing.

I've come to think of RSpecs as unit tests and I don't expect them to
test the "seams" or integration - their responsibility is to be fast
and test as much as is *practical*. My Cucumber scenarios act as
integration tests - their responsibility is to be exhaustive and
involve the whole stack working together in a way that is as close to
real life as possible. I've been burnt by prepackaged pickle setup
steps, etc. and have come to create all my data the "long way"
nowadays as Gareth describes.

I have certainly discovered tests that test more than one thing and
cause 100% rcov, and when I stub/mock them properly, code that should
have had its own unit tests but didn't has been "uncovered" (double
entendre?). Therefore, I feel that isolating unit tests better leads
to more thorough testing.

Also, I like the principle that I shouldn't be testing simple named
scopes - that is functionality you get from Rails and is tested there
already. In the same way that a unit test stubs out its collaborators,
I am happy to stub out something that is unit tested inside the Rails
framework.

--
Gregory McIntyre

[Scott Harvey]

> Also, I like the principle that I shouldn't be testing simple named
> scopes - that is functionality you get from Rails and is tested there
> already. In the same way that a unit test stubs out its collaborators,
> I am happy to stub out something that is unit tested inside the Rails
> framework.

So with that in mind how would you go about writing a unit test for
the example that I gave at the top of this thread?

Scott

[Gregory McIntyre]

On 26 August 2011 17:03, Scott Harvey <scottand...@gmail.com> wrote:
> So with that in mind how would you go about writing a unit test for
> the example that I gave at the top of this thread?

Oh you want me to do more than mouth off? ;-X

I'd do something along these lines:

https://gist.github.com/1172955

--
Gregory McIntyre

[Scott]

Oh you want me to do more than mouth off? ;-X

I'd do something along these lines:

https://gist.github.com/1172955

right, so that ends up being similar to what Malc suggested by using stub_chain.

It still feels a bit too close to testing the implementation of the method than I would like but as long as you have a sensible method that isn't crossing multiple associations it shouldn't be too brittle in the long run.

Scott

[Gregory McIntyre]

On 26 August 2011 18:45, Scott <scottand...@gmail.com> wrote:
> It still feels a bit too close to testing the implementation of the method
> than I would like

Yep. I'm not sure of a better way to work around the paid scope.

I have been finding that liberal use of let allows me to put any
"required but you'd rather not have to" stubs that are a little bit
implementation aware (such as the paid stub in my gist) into a before
step very early on in the spec and what is left in each case is just

1) varying inputs (via let) and
2) outputs (via it { should })

that can be expanded rapidly and mixed together in combinations of
context blocks, without much extra stubbing/mocking.

I hope that makes sense. Happy to explain with more examples if it helps.

--
Gregory McIntyre

[Scott]

I hope that makes sense. Happy to explain with more examples if it helps.

I think what you have shown makes sense. You are stubbing out the "paid" method on a LineItem to return a predefined value and the model under test is responding as expected.

Whether the "paid" call executes a named_scope or instance method is really inconsequential to the object under test.

As long as the LineItem model has an appropriate test that "paid" is returning the correct value we should be all good.

Having an integration test on top of all this would ensure that everything is wired up as expected.

Scott