Nathan Weizenbaum from Haml-land here. I just wanted to point out that
your benchmarks don't accurately reflect the rendering speed in any
framework but Sinatra. The issue is that Sinatra doesn't support any
sort of template caching; the ERB and Haml templates are being re-
parsed every request, and the Ruby code they generate is being re-
evaluated. This is in contrast to a framework like Rails, where the
evaluated Ruby code is kept around after the first request - which is
how ERB and Haml are expected to be run for maximum performance.
In addition, for benchmarking, Haml should be run with the :ugly flag
enabled, which triggers various performance enhancements at the
expense of output formatting. :ugly is set to true by default in
production mode for Rails.
Until Sinatra supports proper caching, it's a good idea to avoid it
for benchmarking - especially when you're reporting Erector's speed
relative to ERB and Haml based on those benchmark numbers.
Fair enough! I'll rerun the benchmarks in Rails, with maximum ugliness :-)
I don't want to besmirch the good name of either Haml or ERB,
especially since Erector is meant as an alternative for design's sake,
not speed. And as I wrote in the FAQ, "We should point out, of course,
that the choice of templating engines by itself is not what will make
your application scalable. The effectiveness of your caching policy
will dwarf that of your rendering engine in nearly all cases."
(To be perfectly honest, the reason I did the original off-the-cuff
benchmarks in Sinatra is because I don't understand Rails template
caching and other under-the-hood machinery very well, and I wanted to
make sure I personally understood exactly what was going on. I guess
it's time to crack open the Rails source code again... sigh.)
On Sun, Oct 11, 2009 at 3:54 AM, Nathan Weizenbaum <nex...@gmail.com> wrote:
> Nathan Weizenbaum from Haml-land here. I just wanted to point out that
> your benchmarks don't accurately reflect the rendering speed in any
> framework but Sinatra. The issue is that Sinatra doesn't support any
> sort of template caching; the ERB and Haml templates are being re-
> parsed every request, and the Ruby code they generate is being re-
> evaluated. This is in contrast to a framework like Rails, where the
> evaluated Ruby code is kept around after the first request - which is
> how ERB and Haml are expected to be run for maximum performance.
> In addition, for benchmarking, Haml should be run with the :ugly flag
> enabled, which triggers various performance enhancements at the
> expense of output formatting. :ugly is set to true by default in
> production mode for Rails.
> Until Sinatra supports proper caching, it's a good idea to avoid it
> for benchmarking - especially when you're reporting Erector's speed
> relative to ERB and Haml based on those benchmark numbers.
I switched to Sinatra because I wanted a more real-world environment. I
wasn't thinking about template caching. I should take down that "2x faster"
claim, huh? >blush<
I would like to run Haml vs. Erector with precompiled templates + nested
partials to see if Haml's dispatching can keep up with plain Ruby.
> You might want to just run the benchmarks without the muss and fuss of a
web
> framework at all. That's what we do. Check out
> http://github.com/nex3/haml/blob/master/test/benchmark.rb (the "Cached"
> benchmark, not either of the ActionView benchmarks).
> On Sun, Oct 11, 2009 at 10:37 AM, Alex Chaffee <ale...@gmail.com> wrote:
>> Fair enough! I'll rerun the benchmarks in Rails, with maximum ugliness
:-)
>> I don't want to besmirch the good name of either Haml or ERB,
>> especially since Erector is meant as an alternative for design's sake,
>> not speed. And as I wrote in the FAQ, "We should point out, of course,
>> that the choice of templating engines by itself is not what will make
>> your application scalable. The effectiveness of your caching policy
>> will dwarf that of your rendering engine in nearly all cases."
>> (To be perfectly honest, the reason I did the original off-the-cuff
>> benchmarks in Sinatra is because I don't understand Rails template
>> caching and other under-the-hood machinery very well, and I wanted to
>> make sure I personally understood exactly what was going on. I guess
>> it's time to crack open the Rails source code again... sigh.)
>> On Sun, Oct 11, 2009 at 3:54 AM, Nathan Weizenbaum <nex...@gmail.com>
>> wrote:
>> > Nathan Weizenbaum from Haml-land here. I just wanted to point out that
>> > your benchmarks don't accurately reflect the rendering speed in any
>> > framework but Sinatra. The issue is that Sinatra doesn't support any
>> > sort of template caching; the ERB and Haml templates are being re-
>> > parsed every request, and the Ruby code they generate is being re-
>> > evaluated. This is in contrast to a framework like Rails, where the
>> > evaluated Ruby code is kept around after the first request - which is
>> > how ERB and Haml are expected to be run for maximum performance.
>> > In addition, for benchmarking, Haml should be run with the :ugly flag
>> > enabled, which triggers various performance enhancements at the
>> > expense of output formatting. :ugly is set to true by default in
>> > production mode for Rails.
>> > Until Sinatra supports proper caching, it's a good idea to avoid it
>> > for benchmarking - especially when you're reporting Erector's speed
>> > relative to ERB and Haml based on those benchmark numbers.
You might want to just run the benchmarks without the muss and fuss of a web
framework at all. That's what we do. Check out
http://github.com/nex3/haml/blob/master/test/benchmark.rb (the "Cached"
benchmark, not either of the ActionView benchmarks).
On Sun, Oct 11, 2009 at 10:37 AM, Alex Chaffee <ale...@gmail.com> wrote:
> Fair enough! I'll rerun the benchmarks in Rails, with maximum ugliness :-)
> I don't want to besmirch the good name of either Haml or ERB,
> especially since Erector is meant as an alternative for design's sake,
> not speed. And as I wrote in the FAQ, "We should point out, of course,
> that the choice of templating engines by itself is not what will make
> your application scalable. The effectiveness of your caching policy
> will dwarf that of your rendering engine in nearly all cases."
> (To be perfectly honest, the reason I did the original off-the-cuff
> benchmarks in Sinatra is because I don't understand Rails template
> caching and other under-the-hood machinery very well, and I wanted to
> make sure I personally understood exactly what was going on. I guess
> it's time to crack open the Rails source code again... sigh.)
> On Sun, Oct 11, 2009 at 3:54 AM, Nathan Weizenbaum <nex...@gmail.com>
> wrote:
> > Nathan Weizenbaum from Haml-land here. I just wanted to point out that
> > your benchmarks don't accurately reflect the rendering speed in any
> > framework but Sinatra. The issue is that Sinatra doesn't support any
> > sort of template caching; the ERB and Haml templates are being re-
> > parsed every request, and the Ruby code they generate is being re-
> > evaluated. This is in contrast to a framework like Rails, where the
> > evaluated Ruby code is kept around after the first request - which is
> > how ERB and Haml are expected to be run for maximum performance.
> > In addition, for benchmarking, Haml should be run with the :ugly flag
> > enabled, which triggers various performance enhancements at the
> > expense of output formatting. :ugly is set to true by default in
> > production mode for Rails.
> > Until Sinatra supports proper caching, it's a good idea to avoid it
> > for benchmarking - especially when you're reporting Erector's speed
> > relative to ERB and Haml based on those benchmark numbers.
I think the issue is that it's highlighting a particular source of slowness
in Haml more than would be highlighted in real-world code. Haml does quite a
bit of processing on tags with dynamic attributes (more than it really
should - this presents a good opportunity for future optimization :-) ),
which is a lot slower than Erubis' "just concatenate the thing" approach. It
would be more fair if the Erubis code HTML-escaped the URL, which Haml does
automatically, but even then it would probably still be slanted towards
Erubis. The better solution would probably be to make the templates look
more like normal pages, rather than having so much space taken up by a tag
with dynamic attributes.
On Sun, Oct 11, 2009 at 1:15 PM, Nathan Weizenbaum <nex...@gmail.com> wrote:
> Something's going weird with that benchmark... Haml shouldn't be much
> slower than Erubis when :ugly is enabled. I'll look into it and get back to
> you.
> On Sun, Oct 11, 2009 at 1:03 PM, Alex Chaffee <ale...@gmail.com> wrote:
>> I switched to Sinatra because I wanted a more real-world environment. I
>> wasn't thinking about template caching. I should take down that "2x faster"
>> claim, huh? >blush<
>> I would like to run Haml vs. Erector with precompiled templates + nested
>> partials to see if Haml's dispatching can keep up with plain Ruby.
>> On Sun, Oct 11, 2009 at 11:53 AM, Nathan Weizenbaum <nex...@gmail.com>
>> wrote:
>> > You might want to just run the benchmarks without the muss and fuss of a
>> web
>> > framework at all. That's what we do. Check out
>> > http://github.com/nex3/haml/blob/master/test/benchmark.rb (the "Cached"
>> > benchmark, not either of the ActionView benchmarks).
>> > On Sun, Oct 11, 2009 at 10:37 AM, Alex Chaffee <ale...@gmail.com>
>> wrote:
>> >> Fair enough! I'll rerun the benchmarks in Rails, with maximum ugliness
>> :-)
>> >> I don't want to besmirch the good name of either Haml or ERB,
>> >> especially since Erector is meant as an alternative for design's sake,
>> >> not speed. And as I wrote in the FAQ, "We should point out, of course,
>> >> that the choice of templating engines by itself is not what will make
>> >> your application scalable. The effectiveness of your caching policy
>> >> will dwarf that of your rendering engine in nearly all cases."
>> >> (To be perfectly honest, the reason I did the original off-the-cuff
>> >> benchmarks in Sinatra is because I don't understand Rails template
>> >> caching and other under-the-hood machinery very well, and I wanted to
>> >> make sure I personally understood exactly what was going on. I guess
>> >> it's time to crack open the Rails source code again... sigh.)
>> >> On Sun, Oct 11, 2009 at 3:54 AM, Nathan Weizenbaum <nex...@gmail.com>
>> >> wrote:
>> >> > Nathan Weizenbaum from Haml-land here. I just wanted to point out
>> that
>> >> > your benchmarks don't accurately reflect the rendering speed in any
>> >> > framework but Sinatra. The issue is that Sinatra doesn't support any
>> >> > sort of template caching; the ERB and Haml templates are being re-
>> >> > parsed every request, and the Ruby code they generate is being re-
>> >> > evaluated. This is in contrast to a framework like Rails, where the
>> >> > evaluated Ruby code is kept around after the first request - which is
>> >> > how ERB and Haml are expected to be run for maximum performance.
>> >> > In addition, for benchmarking, Haml should be run with the :ugly flag
>> >> > enabled, which triggers various performance enhancements at the
>> >> > expense of output formatting. :ugly is set to true by default in
>> >> > production mode for Rails.
>> >> > Until Sinatra supports proper caching, it's a good idea to avoid it
>> >> > for benchmarking - especially when you're reporting Erector's speed
>> >> > relative to ERB and Haml based on those benchmark numbers.
> I switched to Sinatra because I wanted a more real-world environment. I
> wasn't thinking about template caching. I should take down that "2x faster"
> claim, huh? >blush<
> I would like to run Haml vs. Erector with precompiled templates + nested
> partials to see if Haml's dispatching can keep up with plain Ruby.
> On Sun, Oct 11, 2009 at 11:53 AM, Nathan Weizenbaum <nex...@gmail.com>
> wrote:
> > You might want to just run the benchmarks without the muss and fuss of a
> web
> > framework at all. That's what we do. Check out
> > http://github.com/nex3/haml/blob/master/test/benchmark.rb (the "Cached"
> > benchmark, not either of the ActionView benchmarks).
> > On Sun, Oct 11, 2009 at 10:37 AM, Alex Chaffee <ale...@gmail.com> wrote:
> >> Fair enough! I'll rerun the benchmarks in Rails, with maximum ugliness
> :-)
> >> I don't want to besmirch the good name of either Haml or ERB,
> >> especially since Erector is meant as an alternative for design's sake,
> >> not speed. And as I wrote in the FAQ, "We should point out, of course,
> >> that the choice of templating engines by itself is not what will make
> >> your application scalable. The effectiveness of your caching policy
> >> will dwarf that of your rendering engine in nearly all cases."
> >> (To be perfectly honest, the reason I did the original off-the-cuff
> >> benchmarks in Sinatra is because I don't understand Rails template
> >> caching and other under-the-hood machinery very well, and I wanted to
> >> make sure I personally understood exactly what was going on. I guess
> >> it's time to crack open the Rails source code again... sigh.)
> >> On Sun, Oct 11, 2009 at 3:54 AM, Nathan Weizenbaum <nex...@gmail.com>
> >> wrote:
> >> > Nathan Weizenbaum from Haml-land here. I just wanted to point out that
> >> > your benchmarks don't accurately reflect the rendering speed in any
> >> > framework but Sinatra. The issue is that Sinatra doesn't support any
> >> > sort of template caching; the ERB and Haml templates are being re-
> >> > parsed every request, and the Ruby code they generate is being re-
> >> > evaluated. This is in contrast to a framework like Rails, where the
> >> > evaluated Ruby code is kept around after the first request - which is
> >> > how ERB and Haml are expected to be run for maximum performance.
> >> > In addition, for benchmarking, Haml should be run with the :ugly flag
> >> > enabled, which triggers various performance enhancements at the
> >> > expense of output formatting. :ugly is set to true by default in
> >> > production mode for Rails.
> >> > Until Sinatra supports proper caching, it's a good idea to avoid it
> >> > for benchmarking - especially when you're reporting Erector's speed
> >> > relative to ERB and Haml based on those benchmark numbers.
Looks like healthy competition is making both of us get better ;-)
I'm curious how Haml and Erubis do with deeply-nested partials. If your
partials are small enough then the advantage from glomming together massive
strings with little executable code interspersed diminishes relative to
Erector's one-tag-after-another approach. You probably know off the top of
your head how to call a partial via render in a non-Rails world... can you
maybe help improve that gist? And if you like, in return, I'll help add
Erector to your benchmark suite...
On Sun, Oct 11, 2009 at 1:39 PM, Nathan Weizenbaum <nex...@gmail.com> wrote:
> I think the issue is that it's highlighting a particular source of slowness
> in Haml more than would be highlighted in real-world code. Haml does quite a
> bit of processing on tags with dynamic attributes (more than it really
> should - this presents a good opportunity for future optimization :-) ),
> which is a lot slower than Erubis' "just concatenate the thing" approach. It
> would be more fair if the Erubis code HTML-escaped the URL, which Haml does
> automatically, but even then it would probably still be slanted towards
> Erubis. The better solution would probably be to make the templates look
> more like normal pages, rather than having so much space taken up by a tag
> with dynamic attributes.
> I'll get to work on fixing that speed issue, too.
> On Sun, Oct 11, 2009 at 1:15 PM, Nathan Weizenbaum <nex...@gmail.com>wrote:
>> Something's going weird with that benchmark... Haml shouldn't be much
>> slower than Erubis when :ugly is enabled. I'll look into it and get back to
>> you.
>> On Sun, Oct 11, 2009 at 1:03 PM, Alex Chaffee <ale...@gmail.com> wrote:
>>> I switched to Sinatra because I wanted a more real-world environment. I
>>> wasn't thinking about template caching. I should take down that "2x faster"
>>> claim, huh? >blush<
>>> I would like to run Haml vs. Erector with precompiled templates + nested
>>> partials to see if Haml's dispatching can keep up with plain Ruby.
>>> On Sun, Oct 11, 2009 at 11:53 AM, Nathan Weizenbaum <nex...@gmail.com>
>>> wrote:
>>> > You might want to just run the benchmarks without the muss and fuss of
>>> a web
>>> > framework at all. That's what we do. Check out
>>> > http://github.com/nex3/haml/blob/master/test/benchmark.rb (the
>>> "Cached"
>>> > benchmark, not either of the ActionView benchmarks).
>>> > On Sun, Oct 11, 2009 at 10:37 AM, Alex Chaffee <ale...@gmail.com>
>>> wrote:
>>> >> Fair enough! I'll rerun the benchmarks in Rails, with maximum ugliness
>>> :-)
>>> >> I don't want to besmirch the good name of either Haml or ERB,
>>> >> especially since Erector is meant as an alternative for design's sake,
>>> >> not speed. And as I wrote in the FAQ, "We should point out, of course,
>>> >> that the choice of templating engines by itself is not what will make
>>> >> your application scalable. The effectiveness of your caching policy
>>> >> will dwarf that of your rendering engine in nearly all cases."
>>> >> (To be perfectly honest, the reason I did the original off-the-cuff
>>> >> benchmarks in Sinatra is because I don't understand Rails template
>>> >> caching and other under-the-hood machinery very well, and I wanted to
>>> >> make sure I personally understood exactly what was going on. I guess
>>> >> it's time to crack open the Rails source code again... sigh.)
>>> >> On Sun, Oct 11, 2009 at 3:54 AM, Nathan Weizenbaum <nex...@gmail.com>
>>> >> wrote:
>>> >> > Nathan Weizenbaum from Haml-land here. I just wanted to point out
>>> that
>>> >> > your benchmarks don't accurately reflect the rendering speed in any
>>> >> > framework but Sinatra. The issue is that Sinatra doesn't support any
>>> >> > sort of template caching; the ERB and Haml templates are being re-
>>> >> > parsed every request, and the Ruby code they generate is being re-
>>> >> > evaluated. This is in contrast to a framework like Rails, where the
>>> >> > evaluated Ruby code is kept around after the first request - which
>>> is
>>> >> > how ERB and Haml are expected to be run for maximum performance.
>>> >> > In addition, for benchmarking, Haml should be run with the :ugly
>>> flag
>>> >> > enabled, which triggers various performance enhancements at the
>>> >> > expense of output formatting. :ugly is set to true by default in
>>> >> > production mode for Rails.
>>> >> > Until Sinatra supports proper caching, it's a good idea to avoid it
>>> >> > for benchmarking - especially when you're reporting Erector's speed
>>> >> > relative to ERB and Haml based on those benchmark numbers.
Haml should do pretty well on deeply nested partials as long as :ugly is
enabled. It just gloms them onto the stream that's being concatenated,
although the strings are built up separately before being joined, which may
take more time. Haml leaves partials to the host framework, though, so I
consider that basically not my problem. That does mean it's going to be
difficult to benchmark partials outside of Rails, though.
It would be interesting to see a plugin for Rails or something that would
add partial functionality more like Erector's: using the same buffer in
partials that's used in the main page.
On Sun, Oct 11, 2009 at 5:03 PM, Alex Chaffee <ale...@gmail.com> wrote:
> Looks like healthy competition is making both of us get better ;-)
> I'm curious how Haml and Erubis do with deeply-nested partials. If your
> partials are small enough then the advantage from glomming together massive
> strings with little executable code interspersed diminishes relative to
> Erector's one-tag-after-another approach. You probably know off the top of
> your head how to call a partial via render in a non-Rails world... can you
> maybe help improve that gist? And if you like, in return, I'll help add
> Erector to your benchmark suite...
> On Sun, Oct 11, 2009 at 1:39 PM, Nathan Weizenbaum <nex...@gmail.com>wrote:
>> I think the issue is that it's highlighting a particular source of
>> slowness in Haml more than would be highlighted in real-world code. Haml
>> does quite a bit of processing on tags with dynamic attributes (more than it
>> really should - this presents a good opportunity for future optimization :-)
>> ), which is a lot slower than Erubis' "just concatenate the thing" approach.
>> It would be more fair if the Erubis code HTML-escaped the URL, which Haml
>> does automatically, but even then it would probably still be slanted towards
>> Erubis. The better solution would probably be to make the templates look
>> more like normal pages, rather than having so much space taken up by a tag
>> with dynamic attributes.
>> I'll get to work on fixing that speed issue, too.
>> On Sun, Oct 11, 2009 at 1:15 PM, Nathan Weizenbaum <nex...@gmail.com>wrote:
>>> Something's going weird with that benchmark... Haml shouldn't be much
>>> slower than Erubis when :ugly is enabled. I'll look into it and get back to
>>> you.
>>> On Sun, Oct 11, 2009 at 1:03 PM, Alex Chaffee <ale...@gmail.com> wrote:
>>>> I switched to Sinatra because I wanted a more real-world environment. I
>>>> wasn't thinking about template caching. I should take down that "2x faster"
>>>> claim, huh? >blush<
>>>> I would like to run Haml vs. Erector with precompiled templates + nested
>>>> partials to see if Haml's dispatching can keep up with plain Ruby.
>>>> On Sun, Oct 11, 2009 at 11:53 AM, Nathan Weizenbaum <nex...@gmail.com>
>>>> wrote:
>>>> > You might want to just run the benchmarks without the muss and fuss of
>>>> a web
>>>> > framework at all. That's what we do. Check out
>>>> > http://github.com/nex3/haml/blob/master/test/benchmark.rb (the
>>>> "Cached"
>>>> > benchmark, not either of the ActionView benchmarks).
>>>> > On Sun, Oct 11, 2009 at 10:37 AM, Alex Chaffee <ale...@gmail.com>
>>>> wrote:
>>>> >> Fair enough! I'll rerun the benchmarks in Rails, with maximum
>>>> ugliness :-)
>>>> >> I don't want to besmirch the good name of either Haml or ERB,
>>>> >> especially since Erector is meant as an alternative for design's
>>>> sake,
>>>> >> not speed. And as I wrote in the FAQ, "We should point out, of
>>>> course,
>>>> >> that the choice of templating engines by itself is not what will make
>>>> >> your application scalable. The effectiveness of your caching policy
>>>> >> will dwarf that of your rendering engine in nearly all cases."
>>>> >> (To be perfectly honest, the reason I did the original off-the-cuff
>>>> >> benchmarks in Sinatra is because I don't understand Rails template
>>>> >> caching and other under-the-hood machinery very well, and I wanted to
>>>> >> make sure I personally understood exactly what was going on. I guess
>>>> >> it's time to crack open the Rails source code again... sigh.)
>>>> >> On Sun, Oct 11, 2009 at 3:54 AM, Nathan Weizenbaum <nex...@gmail.com
>>>> >> wrote:
>>>> >> > Nathan Weizenbaum from Haml-land here. I just wanted to point out
>>>> that
>>>> >> > your benchmarks don't accurately reflect the rendering speed in any
>>>> >> > framework but Sinatra. The issue is that Sinatra doesn't support
>>>> any
>>>> >> > sort of template caching; the ERB and Haml templates are being re-
>>>> >> > parsed every request, and the Ruby code they generate is being re-
>>>> >> > evaluated. This is in contrast to a framework like Rails, where the
>>>> >> > evaluated Ruby code is kept around after the first request - which
>>>> is
>>>> >> > how ERB and Haml are expected to be run for maximum performance.
>>>> >> > In addition, for benchmarking, Haml should be run with the :ugly
>>>> flag
>>>> >> > enabled, which triggers various performance enhancements at the
>>>> >> > expense of output formatting. :ugly is set to true by default in
>>>> >> > production mode for Rails.
>>>> >> > Until Sinatra supports proper caching, it's a good idea to avoid it
>>>> >> > for benchmarking - especially when you're reporting Erector's speed
>>>> >> > relative to ERB and Haml based on those benchmark numbers.
> It would be interesting to see a plugin for Rails or something that would > add partial functionality more like Erector's: using the same buffer in > partials that's used in the main page.
I was chatting with Yehuda at RailsConf about all this and we agreed this would be a good thing to make work in Rails 3. I'm looking forward to digging into the new template plugin API (whenever Brian Takita and I can get together) and seeing if this made it in.
