Simple benchmark between Shoulda and RSpec

[Lucas Húngaro]

I've been using Shoulda for a long time, and I like it. But I always
believed it was way faster than RSpec.

Today, a friend of mine did a simple benchmark, here it is:
http://github.com/fnando/benchmark-rspec-shoulda/tree/master

The post where he explains the benchmark is in portuguese, here:
http://simplesideias.com.br/benchmark-entre-rspec-e-shoulda/

Any opinions?

[Matt Jankowski]

It doesn't look like the test/benchmark code referenced in that blog
post is actually in that git repo?...

-Matt

[Lucas Húngaro]

Matt, check the repository branches. For exemple, here's the Post
specs with factory_girl and RSpec -
http://github.com/fnando/benchmark-rspec-shoulda/blob/b8132e4892991ec2faaf5643cdf7677b4ce9b846/spec/models/post_spec.rb

What surprised me is that, on Ruby 1.9.1, the Shoulda samples are way
faster than on 1.8.7. With RSpec the improvement is minimum. I think
it suggests that Shoulda's code is using some unoptimized constructs.

[Lucas Húngaro]

Well, this can be caused by Test::Unit and its assertions too.

On Aug 13, 5:57 pm, Lucas Húngaro <lucashung...@gmail.com> wrote:
> Matt, check the repository branches. For exemple, here's the Post

> specs with factory_girl and RSpec -http://github.com/fnando/benchmark-rspec-shoulda/blob/b8132e4892991ec...

[Ryan McGeary]

While I don't read Portuguese, I don't think this tells the whole story.  The results intrigued me, and I started to look into it.  There's definitely a problem with the way Shoulda handles tons of tests in one class, but this benchmark doesn't have much to do with the real world.  For instance, I took Nando's Shoulda tests and ran them using different numbers of should blocks (Nando benchmarked 20,000 tests in one class).

1000 tests, 1000 assertions, 0 failures, 0 errors

Finished in 3.091728 seconds.  (real 0m7.856s)

2000 tests, 2000 assertions, 0 failures, 0 errors

Finished in 5.712364 seconds.  (real 0m11.867s)

3000 tests, 3000 assertions, 0 failures, 0 errors

Finished in 9.460874 seconds.  (real 0m18.493s)

...

...

20000 tests, 20000 assertions, 0 failures, 0 errors

Finished in 81.982342 seconds.  (real 6m47.071s)

Here's all my data and a graph.  

http://img.skitch.com/20090813-p4x8f2rj5mbu1tdrrdp3r7h1a5.png

Notice that Shoulda's total time to completion is somewhat exponential in relation to the number of tests.  The actual test time is linear to the number of tests.  I think this is due to how Shoulda dynamically creates Test::Unit tests up front before running those tests.  So, in summary, yes, Shoulda is much slower for 20,000 tests in one class and we might benefit from looking into Shoulda's initialization routines, but if you have 20,000 tests in one class, you're doing it wrong.

Additional thoughts?

-Ryan

[Dan Croak]

Ryan,

On Thu, Aug 13, 2009 at 5:31 PM, Ryan McGeary<ryan.m...@gmail.com> wrote:

> I took Nando's Shoulda tests and ran them
> using different numbers of should blocks (Nando benchmarked 20,000 tests in
> one class).

How do these tests compare to RSpec?

> So, in summary, yes,
> Shoulda is much slower for 20,000 tests in one class and we might benefit
> from looking into Shoulda's initialization routines, but if you have 20,000
> tests in one class, you're doing it wrong.

Agreed.

--
Dan Croak
@Croaky

[Lucas Húngaro]

I agree that this benchmark doesn't simulate a real situation, but it
surely suggests an issue and I think you have a good guess on what it
is.

On Aug 13, 5:31 pm, Ryan McGeary <ryan.mcge...@gmail.com> wrote:
> While I don't read Portuguese, I don't think this tells the whole story.
>  The results intrigued me, and I started to look into it.  There's
> definitely a problem with the way Shoulda handles tons of tests in one
> class, but this benchmark doesn't have much to do with the real world.  For
> instance, I took Nando's Shoulda tests and ran them
> using different numbers of should blocks (Nando benchmarked 20,000 tests in
> one class).
> 1000 tests, 1000 assertions, 0 failures, 0 errors
> Finished in 3.091728 seconds.  (real 0m7.856s)
>
> 2000 tests, 2000 assertions, 0 failures, 0 errors
> Finished in 5.712364 seconds.  (real 0m11.867s)
>
> 3000 tests, 3000 assertions, 0 failures, 0 errors
> Finished in 9.460874 seconds.  (real 0m18.493s)
> ...
> ...
> 20000 tests, 20000 assertions, 0 failures, 0 errors
> Finished in 81.982342 seconds.  (real 6m47.071s)
>

> Here's all my data and a graph.http://img.skitch.com/20090813-p4x8f2rj5mbu1tdrrdp3r7h1a5.png

>
> Notice that Shoulda's total time to completion is somewhat exponential in
> relation to the number of tests.  The actual test time is linear to the
> number of tests.  I think this is due to how Shoulda dynamically creates
> Test::Unit tests up front before running those tests.  So, in summary, yes,
> Shoulda is much slower for 20,000 tests in one class and we might benefit
> from looking into Shoulda's initialization routines, but if you have 20,000
> tests in one class, you're doing it wrong.
>
> Additional thoughts?
>
> -Ryan
>