Benchmark Accuracy

23 views
Skip to first unread message

Nathan Weizenbaum

unread,
Oct 11, 2009, 6:54:39 AM10/11/09
to erector
Nathan Weizenbaum from Haml-land here. I just wanted to point out that
your benchmarks don't accurately reflect the rendering speed in any
framework but Sinatra. The issue is that Sinatra doesn't support any
sort of template caching; the ERB and Haml templates are being re-
parsed every request, and the Ruby code they generate is being re-
evaluated. This is in contrast to a framework like Rails, where the
evaluated Ruby code is kept around after the first request - which is
how ERB and Haml are expected to be run for maximum performance.

In addition, for benchmarking, Haml should be run with the :ugly flag
enabled, which triggers various performance enhancements at the
expense of output formatting. :ugly is set to true by default in
production mode for Rails.

Until Sinatra supports proper caching, it's a good idea to avoid it
for benchmarking - especially when you're reporting Erector's speed
relative to ERB and Haml based on those benchmark numbers.

Alex Chaffee

unread,
Oct 11, 2009, 1:37:06 PM10/11/09
to nex...@gmail.com, erector
Fair enough! I'll rerun the benchmarks in Rails, with maximum ugliness :-)

I don't want to besmirch the good name of either Haml or ERB,
especially since Erector is meant as an alternative for design's sake,
not speed. And as I wrote in the FAQ, "We should point out, of course,
that the choice of templating engines by itself is not what will make
your application scalable. The effectiveness of your caching policy
will dwarf that of your rendering engine in nearly all cases."

(To be perfectly honest, the reason I did the original off-the-cuff
benchmarks in Sinatra is because I don't understand Rails template
caching and other under-the-hood machinery very well, and I wanted to
make sure I personally understood exactly what was going on. I guess
it's time to crack open the Rails source code again... sigh.)

---
Alex Chaffee - al...@cohuman.com - http://alexch.github.com
Stalk me: http://friendfeed.com/alexch | http://twitter.com/alexch |
http://alexch.tumblr.com

Alex Chaffee

unread,
Oct 11, 2009, 4:03:12 PM10/11/09
to Nathan Weizenbaum, erector
Oh, I did that first! :-)

http://gist.github.com/115155

I switched to Sinatra because I wanted a more real-world environment. I wasn't thinking about template caching. I should take down that "2x faster" claim, huh? >blush< 

I would like to run Haml vs. Erector with precompiled templates + nested partials to see if Haml's dispatching can keep up with plain Ruby.

Here's today's results of that gist.

# of iterations = 100000
                         user     system      total        real
erector              56.630000   2.340000  58.970000 ( 72.280892)
erector_mixin        55.870000   2.310000  58.180000 ( 69.633833)
nokogiri             67.940000   1.630000  69.570000 ( 83.682278)
haml                 43.840000   1.220000  45.060000 ( 53.608881)
erubis                3.070000   0.070000   3.140000 (  3.787638)
tagz                123.200000   5.030000 128.230000 (154.505187)
builder             143.780000   3.970000 147.750000 (182.832440)
markaby             255.990000   5.540000 261.530000 (319.555178)
On Sun, Oct 11, 2009 at 11:53 AM, Nathan Weizenbaum <nex...@gmail.com> wrote:
> You might want to just run the benchmarks without the muss and fuss of a web
> framework at all. That's what we do. Check out
> http://github.com/nex3/haml/blob/master/test/benchmark.rb (the "Cached"
> benchmark, not either of the ActionView benchmarks).

Nathan Weizenbaum

unread,
Oct 11, 2009, 2:53:27 PM10/11/09
to Alex Chaffee, erector
You might want to just run the benchmarks without the muss and fuss of a web framework at all. That's what we do. Check out http://github.com/nex3/haml/blob/master/test/benchmark.rb (the "Cached" benchmark, not either of the ActionView benchmarks).

On Sun, Oct 11, 2009 at 10:37 AM, Alex Chaffee <ale...@gmail.com> wrote:

Nathan Weizenbaum

unread,
Oct 11, 2009, 4:39:40 PM10/11/09
to Alex Chaffee, erector
I think the issue is that it's highlighting a particular source of slowness in Haml more than would be highlighted in real-world code. Haml does quite a bit of processing on tags with dynamic attributes (more than it really should - this presents a good opportunity for future optimization :-) ), which is a lot slower than Erubis' "just concatenate the thing" approach. It would be more fair if the Erubis code HTML-escaped the URL, which Haml does automatically, but even then it would probably still be slanted towards Erubis. The better solution would probably be to make the templates look more like normal pages, rather than having so much space taken up by a tag with dynamic attributes.

I'll get to work on fixing that speed issue, too.

On Sun, Oct 11, 2009 at 1:15 PM, Nathan Weizenbaum <nex...@gmail.com> wrote:
Something's going weird with that benchmark... Haml shouldn't be much slower than Erubis when :ugly is enabled. I'll look into it and get back to you.

Nathan Weizenbaum

unread,
Oct 11, 2009, 4:15:09 PM10/11/09
to Alex Chaffee, erector
Something's going weird with that benchmark... Haml shouldn't be much slower than Erubis when :ugly is enabled. I'll look into it and get back to you.

On Sun, Oct 11, 2009 at 1:03 PM, Alex Chaffee <ale...@gmail.com> wrote:

Alex Chaffee

unread,
Oct 11, 2009, 8:03:09 PM10/11/09
to Nathan Weizenbaum, erector
Looks like healthy competition is making both of us get better ;-)

I'm curious how Haml and Erubis do with deeply-nested partials. If your partials are small enough then the advantage from glomming together massive strings with little executable code interspersed diminishes relative to Erector's one-tag-after-another approach. You probably know off the top of your head how to call a partial via render in a non-Rails world... can you maybe help improve that gist? And if you like, in return, I'll help add Erector to your benchmark suite...

Nathan Weizenbaum

unread,
Oct 11, 2009, 8:15:11 PM10/11/09
to Alex Chaffee, erector
Haml should do pretty well on deeply nested partials as long as :ugly is enabled. It just gloms them onto the stream that's being concatenated, although the strings are built up separately before being joined, which may take more time. Haml leaves partials to the host framework, though, so I consider that basically not my problem. That does mean it's going to be difficult to benchmark partials outside of Rails, though.

It would be interesting to see a plugin for Rails or something that would add partial functionality more like Erector's: using the same buffer in partials that's used in the main page.

Alex Chaffee

unread,
Oct 11, 2009, 9:47:59 PM10/11/09
to Nathan Weizenbaum, erector
It would be interesting to see a plugin for Rails or something that would add partial functionality more like Erector's: using the same buffer in partials that's used in the main page.

I was chatting with Yehuda at RailsConf about all this and we agreed this would be a good thing to make work in Rails 3. I'm looking forward to digging into the new template plugin API (whenever Brian Takita and I can get together) and seeing if this made it in.

 - A

Reply all
Reply to author
Forward
0 new messages