Making tests run faster
Frederick Cheung
I don't know how everyone else is doing but we've taken to having the
fixtures for all our models declared in test_helper. While this may
be contrary to the ideals, in practice this ends up being necessary,
as once you start travelling through the associations a lot of models
end up being touched and maintaining the set of fixtures used by each
test file was an increasing hassle.
This does however lead to an inefficiency: fixtures are reloaded once
per test case (they are cached in @@already_loaded_fixtures
[self.class]), which makes sense if they are defined once per test
case, but not if you used them like we do. With fairly minor changes,
we've made fixtures (optionally) cached for the duration of all tests
ran. On our setups this has cut the time it takes to run tests by
anywhere between 30 and 50%, depending on the number of fixtures used.
Anyway what are people's thoughts on this. Worth writing a patch, or
better in a plugin ? (or is everyone too busy with travel plans for
railsconf europe :-) )
Fred
Tobias Lütke
failed to get it working without regressions. If it works i'll most
certainly merge it into trunk.
--
Tobi
http://shopify.com - modern e-commerce software
http://typo.leetsoft.com - Open source weblog engine
http://blog.leetsoft.com - Technical weblog
Frederick Cheung
>
> Please send me this patch. I have attempted this in the past put
> failed to get it working without regressions. If it works i'll most
> certainly merge it into trunk.
>
I've attached what we've been using, as long as you declare all your
fixtures in advance it has worked fine in our apps (this isn't
however the case of the unit tests that come with activerecord - for
example method_scoping.rb has several subclasses of
Test::Unit::TestCase, with different fixture sets). In it's current
state it's very much an 'all or nothing' setting. It also needs mocha
Caveats aside here's what I did (to rails 1.2.3) :
class_inheritable_accessor :use_transactional_fixtures
class_inheritable_accessor :use_instantiated_fixtures #
true, false, or :no_instances
class_inheritable_accessor :pre_loaded_fixtures
-
+ class_inheritable_accessor :only_load_fixtures_once
+
self.fixture_table_names = []
self.use_transactional_fixtures = false
self.use_instantiated_fixtures = true
self.pre_loaded_fixtures = false
+ self.only_load_fixtures_once = false
- self.fixture_class_names = {}
-
@@already_loaded_fixtures = {}
self.fixture_class_names = {}
@@ -528,15 +528,31 @@
raise RuntimeError, 'pre_loaded_fixtures requires
use_transactional_fixtures'
end
+ if only_load_fixtures_once &&!use_transactional_fixtures
+ raise RuntimeError, 'only_load_fixtures_once requires
use_transactional_fixtures'
+ end
+
@fixture_cache = Hash.new
# Load fixtures once and begin transaction.
if use_transactional_fixtures?
- if @@already_loaded_fixtures[self.class]
- @loaded_fixtures = @@already_loaded_fixtures[self.class]
+ if only_load_fixtures_once
+ if @@already_loaded_fixtures[self.class.superclass]
+ @loaded_fixtures = @@already_loaded_fixtures
[self.class.superclass]
+ freeze_now @@freeze_time
+ else
+ @@freeze_time = Time.now.change :usec => 0
+ freeze_now @@freeze_time
+ load_fixtures
+ @@already_loaded_fixtures[self.class.superclass] =
@loaded_fixtures
+ end
else
- load_fixtures
- @@already_loaded_fixtures[self.class] = @loaded_fixtures
+ if @@already_loaded_fixtures[self.class]
+ @loaded_fixtures = @@already_loaded_fixtures[self.class]
+ else
+ load_fixtures
+ @@already_loaded_fixtures[self.class] = @loaded_fixtures
+ end
end
ActiveRecord::Base.send :increment_open_transactions
ActiveRecord::Base.connection.begin_db_transaction
Having done that, set only_load_fixtures_once to true and you should
be ready to roll. You'll also need the following method defined on
Test::Unit::TestCase
def freeze_now(value=Time.now)
Time.stubs(:now).returns(value)
end
The one change we did have to make is that some of our tests assumed
that Time.now > some_fixture.created_at, where you had
some_fixture:
id: 1
created_at: <%= Time.now %>
Fred
Frederick Cheung
have all of your fixtures declarations in test_helper: ie the first
time we see fixtures :customers we load it, but other times not. This
would require a little more work, but i think would be doable (but
you'd still need to stub out Time.now or else you'd run into
consistency issues (eg fixtures for different tables with (eg)
created_at's set to <%= Time.now.to_s(:db) %>, ie identical .yml
files, but different rows in the database if the 2 files were loaded
at significantly different times.
Fred
Frederick Cheung
Longer version of this is on rails-talk (http://www.ruby-forum.com/
topic/125118)
Short version: get plugin from https://svn1.hosted-projects.com/
fcheung/faster_fixtures/trunk
Ensure all fixtures are defined in test_helper.rb
add require 'faster_fixtures' to test_helper.rb
Thoughts appreciated (in particular I'd love to know a better way
than the horrible thing I had to do to be able to (effectively)
alias_method_chain on setup_with_fixtures (is this horribleness to
cope with the case that there may or may not be an end user setup
method that we don't want to clobber the setup_with_fixtures or is
there some other reason?)
Fred
> --~--~---------~--~----~------------~-------~--~----~
> You received this message because you are subscribed to the Google
> Groups "Ruby on Rails: Core" group.
> To post to this group, send email to rubyonra...@googlegroups.com
> To unsubscribe from this group, send email to rubyonrails-core-
> unsub...@googlegroups.com
> For more options, visit this group at http://groups.google.com/
> group/rubyonrails-core?hl=en
> -~----------~----~----~----~------~----~------~--~---
>
Frederick Cheung
fixtures are in test_helper.rb or not - it simply ensures any given
fixture is not loaded more that once
So now you just install the plugin (http://svn1.hosted-projects.com/
fcheung/faster_fixtures/trunk) and add require 'faster_fixtures' to
test_helper.rb
Fred
On 19 Sep 2007, at 08:27, Frederick Cheung wrote:
> I've wrapped/rewrote this up as a plugin if people want a poke.
> Longer version of this is on rails-talk (http://www.ruby-forum.com/
> topic/125118)
>
> Short version: get plugin from http://svn1.hosted-projects.com/
DHH
> fcheung/faster_fixtures/trunk) and add require 'faster_fixtures' to
> test_helper.rb
This is great. Highrise's unit tests went from 18 seconds to 10
seconds. Let's definitely get this in for Rails 2.0.
Frederick Cheung
Coolio, I'll put a patch together & some stuff to make it enabled/
disabled.
Fred
Jack Danger Canty
I've improved this so that it doesn't care whether or not your
fixtures are in test_helper.rb or not - it simply ensures any given
fixture is not loaded more that once
So now you just install the plugin ( http://svn1.hosted-projects.com/
fcheung/faster_fixtures/trunk) and add require 'faster_fixtures' to
test_helper.rb
I love the plugin and I saw a speedup as well but it broke a few tests. 9/250 tests failed - several had something to do with testing whether model attributes had been set internally via a before_save or similar callback. I can only guess that there's some extra caching going on.
I don't have time to debug it just yet but I recommend we get some more folks to try it out on their app before we fold this into core.
::Jack Danger
DHH
> folks to try it out on their app before we fold this into core.
Definitely. Could you send the test cases that failed to Frederick?
Then he can probably discern what's up.
Jack Danger Canty
Will do.
::Jack
Frederick Cheung
On 21 Sep 2007, at 21:07, Jack Danger Canty wrote:
>
> I love the plugin and I saw a speedup as well but it broke a few
> tests. 9/250 tests failed - several had something to do with
> testing whether model attributes had been set internally via a
> before_save or similar callback. I can only guess that there's
> some extra caching going on.
>
> I don't have time to debug it just yet but I recommend we get some
> more folks to try it out on their app before we fold this into core.
I'd be interested in seeing what's going on. It does introduce extra
caching (indeed that's the whole point), but there always was some
caching: it's just that instead of per testcase caching it's common
across all test cases.
Fred
Jack Danger Canty
It looks like all the errors I'm seeing relate to the fact that the plugin freezes time during test cases. My app has a few tests that do things like:
) time = Time.now
) record = Model.wacky_find_or_create
) assert record.created_at > time
I took just a few minutes to refactor my tests and they all pass now but this is definitely a gotcha for the plugin.
I'm seeing a 55% speedup in unit tests and 25% in functional tests. Wicked.
::Jack
Mislav Marohnić
The hack for chaining the "setup" method you declared ugly in the comments is really not bad. I've been thinking and I can't really think of a way to make it happen in Ruby except monkeypatching TestCase run method to allow for custom, chainable callbacks.
Frederick Cheung
It just irks me that I had to essentially reproduce the method_added
from fixtures.rb along with the messy business of first redefining
the method_added to be a no-op etc... Good to know that I haven't
missed any tricks.
Fred
Frederick Cheung
On 21 Sep 2007, at 23:20, Jack Danger Canty wrote:
> It looks like all the errors I'm seeing relate to the fact that the
> plugin freezes time during test cases. My app has a few tests that
> do things like:
>
> ) time = Time.now
> ) record = Model.wacky_find_or_create
> ) assert record.created_at > time
>
> I took just a few minutes to refactor my tests and they all pass
> now but this is definitely a gotcha for the plugin.
for a while) we had a few cases like this. I think we just changed
all of our > in those cases to be >=.
I think internally my justification for that was that morally my
tests shouldn't care whether I'm running on a fast machine or a slow
machine and if my computer was infinitely fast then it could execute
tests faster than the granularity for Time.now
It would be nice not to have to freeze time, but you pretty much have
to for stuff to be consistent
> I'm seeing a 55% speedup in unit tests and 25% in functional
> tests. Wicked.
>
Sweet
Fred
> ::Jack
Jonathan Viney
http://svn.viney.net.nz/things/rails/plugins/testcase_setup_and_teardown_with_blocks/lib/testcase_setup_and_teardown_with_blocks.rb
-Jonathan.
DHH
Rails 2.0. I've added a ticket for it: http://dev.rubyonrails.org/ticket/9682
On Sep 22, 6:31 am, Frederick Cheung <frederick.che...@gmail.com>
wrote:
> smime.p7s
> 3KDownload
Frederick Cheung
>
> What's the progress on this? I'd really like to see it make it into
> Rails 2.0. I've added a ticket for it: http://dev.rubyonrails.org/
> ticket/9682
>
I've got a patch just about ready (all the tests in AR pass with it,
just need to add some tests specifically for the faster fixtures stuff)
I did have 2 queries:
- Should there be a flag to turn this on and off? I've already set
things up so that encountering a non transactional test jettisons the
cache (since the test could be junking the test data, requiring
fixtures to be reloaded)
- I've used mocha as my time fixing mechanism, which would mean an
added dependency. activerecord/test/mixin_test.rb shows a different
way of fixing the value of Time.now. Would it be preferable to use
something similar instead ?
Fred
Mislav Marohnić
- I've used mocha as my time fixing mechanism, which would mean an
added dependency. activerecord/test/mixin_test.rb shows a different
way of fixing the value of Time.now. Would it be preferable to use
something similar instead ?
Frederick Cheung
Sounds sensible. Just wanted to check that was the general
community's vibe.
Fred
julik
Sounds sensible. Just wanted to check that was the general
community's vibe.
Frederick Cheung
You could probably get away with it quite a lot of time. I've waffled
a bit about this http://www.texperts.com/2007/09/22/loading-fixtures-
with-parsimony/
The short version is that any time a test does something somehow
connected to Time.now you can get failures (depending on how much
time passed between the point at which you loaded fixtures and the
point at which the test run), because previously it was true that
(basically) Time.now = Time when fixtures were loaded = created of an
object with created_at: <%= Time.now.to_s(:db) %>. I think (and
certainly my experience when i wrote this with our apps) that
maintaining that assumption is helpful (and certainly outweighs any
disadvantages I found).
Fred
Tarmo Tänav
> You could probably get away with it quite a lot of time. I've waffled
> a bit about this http://www.texperts.com/2007/09/22/loading-fixtures-
> with-parsimony/
> The short version is that any time a test does something somehow
> connected to Time.now you can get failures (depending on how much
> time passed between the point at which you loaded fixtures and the
> point at which the test run), because previously it was true that
> (basically) Time.now = Time when fixtures were loaded = created of an
> object with created_at: <%= Time.now.to_s(:db) %>. I think (and
> certainly my experience when i wrote this with our apps) that
> maintaining that assumption is helpful (and certainly outweighs any
> disadvantages I found).
I've been using the same technique for speeding up my tests,
independently developed. I don't think freezing time is something
that should be enforced by rails, if some of your tests really
need it then those tests should do it.
For me freezing time has not really been a problem because the
timestamps that I do test are usually specified with a fixed
hour/minute part but a changing date (fixed offset against today)
date part.
So I think this patch in general is great, but the time freezing thing
should not be included.
John Barnette
On 9/25/07, Frederick Cheung <frederic...@gmail.com> wrote:
> You could probably get away with it quite a lot of time. I've waffled
> a bit about this http://www.texperts.com/2007/09/22/loading-fixtures-
> with-parsimony/
The fact that this assertion:
assert_not_equal(Time.now.to_i, (sleep 1; Time.now.to_i))
will always fail is a little bit disconcerting. Our codebase has a
fair amount of time-based tests. While most of them are fairly chunky,
e.g.,
Time.travel_to(2.weeks.from_now) do
assert_something_date_sensitive
end
...some of them do depend on time continuing to run. In fact, we've
carefully made sure that time continues to run even when we've
temporarily changed the definition of 'now'.
It seems like the window of danger if time continues to run during a
test is relatively small. Additionally, I'm not convinced that a test
which depends on the timestamps of fixture data to a granularity of
less than, say, five minutes is suitably robust.
~ j.
Frederick Cheung
On 25 Sep 2007, at 22:27, John Barnette wrote:
> ...some of them do depend on time continuing to run. In fact, we've
> carefully made sure that time continues to run even when we've
> temporarily changed the definition of 'now'.
>
Would your tests still pass if say you replaced your test machine
with one that ran 10 times faster (or say if you switched to a faster
implementation of ruby)
> It seems like the window of danger if time continues to run during a
> test is relatively small. Additionally, I'm not convinced that a test
> which depends on the timestamps of fixture data to a granularity of
> less than, say, five minutes is suitably robust.
>
Rather than window of danger what I was really gunning for is
consistency. We've also got our share of time-based tests, but they
do tend to be on the shorter timescale of things.
It certainly wouldn't be hard to only freeze time for the purpose of
loading the tests, providing the option to freeze it permanently if
desired. The other question is what will break more tests out there:
time that doesn't pass or fixtures data that is older than existing
tests expect ? The apps I've worked on couldn't care less about the
former, but I've no idea what the general pixture is
Fred
julik
I've been using the same technique for speeding up my tests,
independently developed. I don't think freezing time is something
that should be enforced by rails, if some of your tests really
need it then those tests should do it.
julik
The other question is what will break more tests out there:
time that doesn't pass or fixtures data that is older than existing
tests expect ?
julik
I'm not convinced that a test
which depends on the timestamps of fixture data to a granularity of
less than, say, five minutes is suitably robust.
Frederick Cheung
Sure that's the best practice. However (as far as I can tell) if you
have been following best practices then freezing doesn't add
problems, but if you haven't then the whole only loading fixtures
thing once might break some tests. So do you go a little out of your
way to accommodate these people, do you ignore them and let them fix
their tests when they upgrade or do you toss a little vinegar in
their eyes by requiring them to set some setting ?
Fred
DHH
> have been following best practices then freezing doesn't add
> problems, but if you haven't then the whole only loading fixtures
> thing once might break some tests.
I think we should leave freezing time out as a separate concern and
just focus on the fixture speedups for now. No need to marry the two
issues, imo.
Frederick Cheung
I think you still want to do enough freezing so that it looks like
your fixtures were loaded at the same time. Otherwise you end up with
weird stuff like blog posts created before the user that created them
& other oddities.
Fred
Jack Danger Canty
Frederick Cheung
On 26 Sep 2007, at 01:06, Jack Danger Canty wrote:
> On 9/25/07, Frederick Cheung <frederic...@gmail.com> wrote: On
> 26 Sep 2007, at 00:35, DHH wrote:
> >
> >> Sure that's the best practice. However (as far as I can tell) if
> you
> >> have been following best practices then freezing doesn't add
> >> problems, but if you haven't then the whole only loading fixtures
> >> thing once might break some tests.
> >
> > I think we should leave freezing time out as a separate concern and
> > just focus on the fixture speedups for now. No need to marry the two
> > issues, imo.
>
> I think you still want to do enough freezing so that it looks like
> your fixtures were loaded at the same time. Otherwise you end up with
> weird stuff like blog posts created before the user that created them
> & other oddities.
>
> I don't think making sure that fixtures loading at the same time is
> the role of the faster_fixtures code. If there is a fixture
> created at Time.now that depends on another fixture created at
> Time.now then clearly the developer doesn't care about comparing
> the creation dates.
Just talking about my code for a second, that's not quite true: I
don't care about the specific values of any of the created_at's in
the database, I only care about the relations between them: what
happens before/after/at the same time at other times. Hard coding
'2007-04-08 19:00:00' and '2007-04-08 19:02:00' in fixed dates seemed
a lot nastier then having one thing created_at 2.minutes.ago and the
other at Time.now.
> Better ways to avoid those 'oddities' would be to load fixtures in
> a reasonable order or to set more plausible created_at datestamps.
> Or we could have a separate discussion about freezing time.
Is there such a thing as reasonable order? I'm not convinced things
couldn't be a bit circular. But at the end of the day I think it
holds that the 2 are orthogonal
Fred
>
> Aside from the mocha dependency and the time freezing I think this
> is golden.
>
I scrapped the mochaness anyway
>
julik
I think you still want to do enough freezing so that it looks like
your fixtures were loaded at the same time. Otherwise you end up with
weird stuff like blog posts created before the user that created them
& other oddities.
Jay Levitt
> Otherwise you end up with
> weird stuff like blog posts created before the user that created them
> & other oddities.
Couldn't that happen in production as well, though? Obviously not this
particular case, because creating a user generally happens a few steps
before a post is made. But for any two related records, unless your test
assumes which order they're written in, one could be written during the
last jiffy of 11:22:33, and the other could be written during the first
jiffy of 11:22:34.
In fact, let's say this is a blog that allows anonymous posts, but for
recordkeeping, it automatically creates a user at post time if one doesn't
already exist. That could easily lead to the above situation.
The problem with freezing time is that it assumes that nobody else is doing
anything with it. I often test expirations by mocking Time.now; will your
time-freeze interfere with my mocking, no matter whether I'm mocking in
FlexMock or Mocha or RSpec?
--
Jay Levitt |
Boston, MA | My character doesn't like it when they
Faster: jay at jay dot fm | cry or shout or hit.
http://www.jay.fm | - Kristoffer
Frederick Cheung
On 26 Sep 2007, at 06:24, Jay Levitt wrote:
>
> On Wed, 26 Sep 2007 00:43:16 +0100, Frederick Cheung wrote:
>
>> Otherwise you end up with
>> weird stuff like blog posts created before the user that created them
>> & other oddities.
>
>
> In fact, let's say this is a blog that allows anonymous posts, but for
> recordkeeping, it automatically creates a user at post time if one
> doesn't
> already exist. That could easily lead to the above situation.
Sure, but that;s not quite the point of my example: the point was
that if you have a situation where logically x.created_at must be >=
y.created_at then that no longer necessarily holds (i'll give an
example from my app: we have questions and answers (that arrive and
are sent back over sms) It's just plain nonsensical for the answer to
be sent before the question it's answering arrived). To an extent
this could already happen in rails: fixtures are all loaded in one
go, but obviously they're not all guarenteed to be loaded within the
same secon.
Fred
> The problem with freezing time is that it assumes that nobody else
> is doing
> anything with it. I often test expirations by mocking Time.now;
> will your
> time-freeze interfere with my mocking, no matter whether I'm
> mocking in
> FlexMock or Mocha or RSpec?
>
I'm not sure about that one. I've often changed time more than once
in a test, once for the fixtures stuff and then refrooze it again in
the tests themselves. That was with mocha. With this code' i'd
changed this so that it reopens Time and alias_method_chain's now. I
think that makes it orthogonal to mocha et al but I don't actually know.
Anyway, the time changing stuff looks to be a separate problem for now.
Fred
James Mead
--
James.
http://blog.floehopper.org
http://tumble.floehopper.org
Frederick Cheung
> I'm sorry I'm coming to this conversation rather late in the day,
> but I wonder whether there has been any thought given to moving
> fixtures from core Rails into a plugin? I know that a lot of
> developers (including me) don't use them in their tests any more -
> preferring instead to use mocking in unit tests that don't touch
> the database in combination with integration tests that do touch
> the database and explicitly call ActiveRecord create!, etc, to
> insert data. I only mention the idea because seems to fit with the
> philosophy of moving stuff out into plugins. Anyway, just a thought.
Well apart from anything else the activerecord tests themselves use
fixtures quite a lot and obviously mocking out the database
interactions for those tests is probably not appropriate since that's
what is being tested. We still use fixtures a lot (but then on
average our database interactions are quite a bit beyond your typical
CRUD, so eg we want test coverage for our hand rolled SQL etc...).
Fred
Jack Danger Canty
I'm sorry I'm coming to this conversation rather late in the day, but I wonder whether there has been any thought given to moving fixtures from core Rails into a plugin? I know that a lot of developers (including me) don't use them in their tests any more - preferring instead to use mocking in unit tests that don't touch the database in combination with integration tests that do touch the database and explicitly call ActiveRecord create!, etc, to insert data. I only mention the idea because seems to fit with the philosophy of moving stuff out into plugins. Anyway, just a thought.
Michael Koziarski
> wonder whether there has been any thought given to moving fixtures from core
> Rails into a plugin? I know that a lot of developers (including me) don't
> use them in their tests any more - preferring instead to use mocking in unit
> tests that don't touch the database in combination with integration tests
> that do touch the database and explicitly call ActiveRecord create!, etc, to
> insert data. I only mention the idea because seems to fit with the
> philosophy of moving stuff out into plugins. Anyway, just a thought.
As someone who uses fixtures each and every day, I have no interest in
seeing this happen :)
Having said that, if fixtures are somehow getting in the way for those
of you who *don't* use them, we could take patches to clean that up.
> --
> James.
> http://blog.floehopper.org
> http://tumble.floehopper.org
>
> >
>
--
Cheers
Koz
Michael Koziarski
> > just focus on the fixture speedups for now. No need to marry the two
> > issues, imo.
>
> I think you still want to do enough freezing so that it looks like
> your fixtures were loaded at the same time. Otherwise you end up with
> weird stuff like blog posts created before the user that created them
> & other oddities.
I'm with David on this one, for the cases where it matters you can
still freeze time yourself. We can address the other oddities if and
when a large number of people report them.
--
Cheers
Koz
Frederick Cheung
Agreed. The patch currently attached to the ticket already reflects
this.
Fred