Quick benchmarks (dramatic results)

[Matt Aimonetti]

As promised I did some quick benchmarking of the i18n gem (not the
Rails implementation yet) and I'm quite concerned by the results:

http://gist.github.com/7471

(note that you need http://github.com/somebee/rbench to run the
benchmarks)

These are just very simple benchmarks, but the results are quite
alarming. Simple Backend is 60 times slower than using a standard
hash! I didn't look at the Rails implementation but I'm afraid that it
will seriously slow down Rails 2.2!

I'll try to investigate a little bit more, but I think some
optimization needs to happen before Rails 2.2 gets tagged. Feel free
to play with the benchmarks branch: http://github.com/mattetti/i18n/tree/benchmarks

-Matt

[jos...@gmail.com]

Hi Matt,

Thanks for the important work on the benchmarks. The first release of
i18n in Rails is pretty much frozen, so we don't want to get into more
optimization at this point, but it's definitely something we can look
at more closely for the next release.

According to your benchmarks, calling @backend.translate 100,000 times
takes 3.56s, which means you can call it over 28,000 times a second. I
ran a similar benchmark with ActiveSupport#camelize, and that shows
that camelize is 20-25 times slower than the hash lookup. I think this
can be explained by the fact that the hash lookup is built-in C code,
while the ActiveSupport stuff and the i18n stuff have Ruby code in
them. I don't think this is a cause for alarm, and shouldn't be
noticable for virtually any use of Rails.

-- Josh

On Aug 27, 4:08 pm, Matt Aimonetti <mattAimone...@gmail.com> wrote:
> As promised I did some quick benchmarking of the i18n gem (not the
> Rails implementation yet) and I'm quite concerned by the results:
>
> http://gist.github.com/7471
>

> (note that you needhttp://github.com/somebee/rbenchto run the

[Matt Aimonetti]

Hi Josh,

 Thanks for taking the performance report quite seriously. I understand you want to release Rails 2.2 for Rails Conf Europe and that won't give anybody enough time to fix the speed issues.

 I also realize that it's party my fault since I raised this concern a while back and promised some benchmarks which I only delivered today :(

Note that the benchmarks were done on a powerful machine with more RAM than most of us use on their servers. Also, let's not forget that I didn't even load Rails and therefore AS doesn't do it's multibyte magic which seriously slows down any string processing.

However, I do agree that it's not a real show stopper since template rendering isn't the bottleneck of current Rails apps. My concern is that this is the fastest/default i18n processing for Rails and once again we are affecting the overall performances.

As you said, this shouldn't be noticeable in most cases but I believe that we can profile the code and optimize it to make it at least twice as fast.

-Matt

[Jeremy Kemper]

On Wed, Aug 27, 2008 at 8:15 AM, Matt Aimonetti <mattai...@gmail.com> wrote:
> Hi Josh,
>
> Thanks for taking the performance report quite seriously. I understand you
> want to release Rails 2.2 for Rails Conf Europe and that won't give anybody
> enough time to fix the speed issues.
>
> I also realize that it's party my fault since I raised this concern a while
> back and promised some benchmarks which I only delivered today :(

It's ok, as long as speed is at least on par with the other existing
i18n plugins, then we're on the right track and can incrementally
improve.

> Note that the benchmarks were done on a powerful machine with more RAM than
> most of us use on their servers. Also, let's not forget that I didn't even
> load Rails and therefore AS doesn't do it's multibyte magic which seriously
> slows down any string processing.

(I don't know why you keep saying this.)

> However, I do agree that it's not a real show stopper since template
> rendering isn't the bottleneck of current Rails apps. My concern is that
> this is the fastest/default i18n processing for Rails and once again we are
> affecting the overall performances.

Template rendering is, actually, the bottleneck for most Rails apps.

> As you said, this shouldn't be noticeable in most cases but I believe that
> we can profile the code and optimize it to make it at least twice as fast.

By all means :) Anyone have the time and inclination to do a bit of
performance work before (or at) RailsConf.eu?

Best,
jeremy

[Matt Aimonetti]

On Wed, Aug 27, 2008 at 9:55 PM, Jeremy Kemper <jer...@bitsweat.net> wrote:

On Wed, Aug 27, 2008 at 8:15 AM, Matt Aimonetti <mattai...@gmail.com> wrote:
> Hi Josh,
>
>  Thanks for taking the performance report quite seriously. I understand you
> want to release Rails 2.2 for Rails Conf Europe and that won't give anybody
> enough time to fix the speed issues.
>
>  I also realize that it's party my fault since I raised this concern a while
> back and promised some benchmarks which I only delivered today :(

It's ok, as long as speed is at least on par with the other existing
i18n plugins, then we're on the right track and can incrementally
improve.

> Note that the benchmarks were done on a powerful machine with more RAM than
> most of us use on their servers. Also, let's not forget that I didn't even
> load Rails and therefore AS doesn't do it's multibyte magic which seriously
> slows down any string processing.

(I don't know why you keep saying this.)

I base that off of benchmarks I did with Wycats when we investigated multibyte support for Merb. I can look back at my logs and get you more details if you want. This is mainly because Ruby multibyte support is not fully implemented in1.8 and AS has to check on the encoding and parse the string to handle all the chars.
 

> However, I do agree that it's not a real show stopper since template
> rendering isn't the bottleneck of current Rails apps. My concern is that
> this is the fastest/default i18n processing for Rails and once again we are
> affecting the overall performances.

Template rendering is, actually, the bottleneck for most Rails apps.

Sorry, I didn't express myself correctly, what I meant was that at this level it won't really affect the template rendering (other stuff are way slower) but that's still a performance hit.
 

> As you said, this shouldn't be noticeable in most cases but I believe that
> we can profile the code and optimize it to make it at least twice as fast.

By all means :) Anyone have the time and inclination to do a bit of
performance work before (or at) RailsConf.eu?

I won't be in Berlin, but I'll certainly my best to benchmark a bit more and see how we could improve the results. (I'll also try to benchmark Rails directly)

-Matt
 

Best,
jeremy

[Jeremy Kemper]

Right, we discussed it earlier: Multibyte is slow, especially compared
to Yehuda's superfast extension. Multibyte is not baked into String,
though, and doesn't slow down any string processing.

>> > However, I do agree that it's not a real show stopper since template
>> > rendering isn't the bottleneck of current Rails apps. My concern is that
>> > this is the fastest/default i18n processing for Rails and once again we
>> > are
>> > affecting the overall performances.
>>
>> Template rendering is, actually, the bottleneck for most Rails apps.
>
> Sorry, I didn't express myself correctly, what I meant was that at this
> level it won't really affect the template rendering (other stuff are way
> slower) but that's still a performance hit.
>
>>
>> > As you said, this shouldn't be noticeable in most cases but I believe
>> > that
>> > we can profile the code and optimize it to make it at least twice as
>> > fast.
>>
>> By all means :) Anyone have the time and inclination to do a bit of
>> performance work before (or at) RailsConf.eu?
>
> I won't be in Berlin, but I'll certainly my best to benchmark a bit more and
> see how we could improve the results. (I'll also try to benchmark Rails
> directly)

Cool. I'm sure we can get this blazing fast.

Best,
jeremy

[Joshua Harvey]

Thanks for weighing in, Jeremy. I haven't seen any indications yet that there's a bottleneck in the i18n code that will affect anyone. It's always good to profile code and see if anything obvious jumps out, but until this gets some real world use I don't think we should be doing too much premature optimization.

That being said, we should definitely have our ears open for reports of slowness once people start using this, and if anybody does have time to profile the code and does find something obvious to change that will really improve performance, that would be great. Otherwise, we'll look into it ourselves for the second release.

-- Josh

[Sven Fuchs]

I still feel we should not invest in early optimization.

I can't see how the benchmark results are "dramatic" in any way and
we're just not as far in the overall process that it would make sense
to heavily optimize the implementation.

There are just a couple of calls in rather rare situations that Rails
itself does when the user doesn't explicitely request to do them (i.e.
calls that actually add load to Rails). These are things like
rendering ActiveRecord error messages or looking up formats in rarely
used helpers. IIRC each of these does just one call to the api.

That said, as far as I can see there are two things we could do:

- Provide a Caching backend that just delegates to Simple backend and
caches results to memory
- Improve performance of the #interpolate method which maybe could be
done as a regular Regexp instead of using StringScanner. This is just
something nobody figured out, yet.

I'd rather object to optimizations that make the implementation more
complicated at this point though. We'd want to push these back to a
point where we're sure enough that the API survives the storms. Of
course we could have a place for experimenting with this like
OptimizedSimple backend or even an Inline-C implementation but I'd
have issues with making this the default backend at this point.

Personally I'd love to work on this but I think I'll barely find
enough time before RailsConf. Maybe the Code Community Drive thing is
something where some hacking could be done?

--
sven fuchs sven...@artweb-design.de
artweb design http://www.artweb-design.de
grünberger 65 + 49 (0) 30 - 47 98 69 96 (phone)
d-10245 berlin + 49 (0) 171 - 35 20 38 4 (mobile)

[Sven Fuchs]

Forgot to include the gist link.

By "Caching backend" i mean something like this: https://gist.github.com/29ec378f9a766b34eb11

That would be completely unobtrusive to the implementation, wouldn't
tie the gem to Rails and might be something someone could do until
RailsConf?

[Vladimir]

Isn't it better to just use standard rails caching mechanisms like page caching, or fragment caching whenever you encounter dramatic slow down by I18n? Moreover, they are used anyway in most heavy loaded applications.

> By "Caching backend" i mean something like this: https://gist.github.com/29ec378f9a766b34eb11

It does some computations (map,join,inspect,to_sym), that are much heavier than a simple hash lookup, so it still would be tenfold slower, imho.

[Sven Fuchs]

On 28.08.2008, at 11:12, Vladimir wrote:
> Isn't it better to just use standard rails caching mechanisms like
> page caching, or fragment caching whenever you encounter dramatic
> slow down by I18n? Moreover, they are used anyway in most heavy
> loaded applications.

Yeah, that's why we thought we could push code optimization back.

> > By "Caching backend" i mean something like this: https://gist.github.com/29ec378f9a766b34eb11
> It does some computations (map,join,inspect,to_sym), that are much
> heavier than a simple hash lookup, so it still would be tenfold
> slower, imho.

That's true. I should have had another coffee first ;)



>
>
> On Thu, Aug 28, 2008 at 13:03, Sven Fuchs <svenfuchs@artweb-

[Luca Guidi]

Sven Fuchs wrote:
> That said, as far as I can see there are two things we could do:
>
> - Provide a Caching backend that just delegates to Simple backend and
> caches results to memory
> - Improve performance of the #interpolate method which maybe could be
> done as a regular Regexp instead of using StringScanner. This is just
> something nobody figured out, yet.

I tried to change a bit #interpolate, instead of use StringScan I used
regexps in this way:

values.inject(string) do |memo, (pattern, value)|
memo.gsub(/\{\{#{pattern}\}\}/, value.to_s)
end

This portion of code is just a bit faster than actual implementation
(1.16x), but the whole method increase performance of 1.05x.

It's just a quick implementation, surely not good, but it could give
some hints to you, guys.

I posted the code, benchmark and results as gist:
http://gist.github.com/7775

Best,
-luca
--
blog: www.lucaguidi.com

[Jeremy Kemper]

Nice, Luca. You can speed this up further by doing a single gsub:

MATCH = /(\\\\)?\{\{([^\}]+)\}\}/

result = string.gsub(MATCH) do
escaped, pattern, key = $1, $2, $2.to_sym

if escaped
pattern
elsif INTERPOLATION_RESERVED_KEYS.include?(pattern)
raise ReservedInterpolationKey.new(pattern, string)
elsif !values.include?(key)
raise MissingInterpolationArgument.new(pattern, string)
else
value[key].to_s
end
end

# untested ;)

Best,
jeremy

[Luca Guidi]

Hi Jeremy,
Your algorithm is faster than StringScan version (1.80x), and gives a
1.55x boost to #interpolate.

I updated the code and benchmark gist results: http://gist.github.com/7775

[Karel Minarik]

I second Vladimir. Any performance-concerned Rails application should
use default Rails caching anyway.

Karel

[Luca Guidi]

Karel Minarik wrote:
> I second Vladimir. Any performance-concerned Rails application should
> use default Rails caching anyway.

I agree, but it's not bad to speedup time-expensive algorithms.

PS: I attached a patch for Backend::Simple#interpolate:
http://gist.github.com/7775

Best,
-luca
--
blog: www.lucaguidi.com

Pro-Netics: www.pro-netics.com
Sourcesense - making sense of Open Source: www.sourcesense.com

[Sven Fuchs]

On 29.08.2008, at 16:14, Luca Guidi wrote:
> Karel Minarik wrote:
>> I second Vladimir. Any performance-concerned Rails application should
>> use default Rails caching anyway.
> I agree, but it's not bad to speedup time-expensive algorithms.
>
> PS: I attached a patch for Backend::Simple#interpolate:
> http://gist.github.com/7775

Cool :)

I've applied this and created a ticket for Rails

http://rails.lighthouseapp.com/projects/8994-ruby-on-rails/tickets/943-i18n-interpolate-performance-improvement

I think #interpolate even reads better with StringScanner removed.