Re: Desert and Passenger/mod_rails

[Andrei Erdoss]

Does anybody have any opinions on this? Chad? Anybody?

On Wed, Mar 25, 2009 at 3:44 PM, Andrei Erdoss <erd...@gmail.com> wrote:

Hello,

I am a big fan of the Desert concept and I would like to congratulate you for releasing such a great gem.

I am trying to use Desert in a project with Passenger/mod_rails. After reading Brian's post, I was wondering if there would be any performance problems with Desert due lazy loading of files and Passenger forking.

This is what I gathered from one of the authors (Hongli Lai) of Passenger:

"Yes, lazy loaded files are loaded once in each forked process. Though this is no different from Mongrel/Thin, where the same thing happens. You can optimize this by preloading the files in environment.rb,
so that the forked worker processes don't have to load them again. This also saves memory if you're using REE."

If preloading the files in environment.rb is the best solution, what is the best way to preload the files in environment.rb for the plugins that Desert manages?

Thank you,

--
Andrei Erdoss

--
Andrei Erdoss

[Adam Milligan]

Any version of Rails 2.2 or later should eager load classes by default, so this shouldn't be an issue. 

Does that answer your question?

Hunted and pecked on my iPhone.

[Andrei Erdoss]

Thank you for your reply. I did some performance testing using a plugin that uses Desert and Passenger. The results were pretty bad.

This is done without preloading. Can you give me an example on how I would preload this plugin into environment.rb? Do I have to include all classes and modules?

Andrei

--
Andrei Erdoss

[Brian Takita]

On Thu, Mar 26, 2009 at 2:30 AM, Andrei Erdoss <erd...@gmail.com> wrote:
> Thank you for your reply. I did some performance testing using a plugin that
> uses Desert and Passenger. The results were pretty bad.
>
> This is done without preloading. Can you give me an example on how I would
> preload this plugin into environment.rb? Do I have to include all classes
> and modules?

That would probably be a good idea.

Desert causes more file searching and loading because it loads all
matches, rather than the first match, when you use require or the
constant autoloading. On Passenger, there would be performance issues
if each forked process needed to load all of the files.

> This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
> Copyright 2006 The Apache Software Foundation, http://www.apache.org/
>
> Benchmarking cauta.ro (be patient)
>
>
> Server Software: Apache/2.2.8
> Server Hostname: cauta.ro
> Server Port: 80
>
> Document Path: /home/index
> Document Length: 8465 bytes
>
> Concurrency Level: 10
> Time taken for tests: 526.253196 seconds
> Complete requests: 1000
> Failed requests: 0
> Write errors: 0
> Total transferred: 9002291 bytes
> HTML transferred: 8465000 bytes
> Requests per second: 1.90 [#/sec] (mean)
> Time per request: 5262.532 [ms] (mean)
> Time per request: 526.253 [ms] (mean, across all concurrent requests)
> Transfer rate: 16.70 [Kbytes/sec] received
>
> Connection Times (ms)
> min mean[+/-sd] median max
> Connect: 146 151 5.0 151 211
> Processing: 3156 5092 2973.3 4323 36119
> Waiting: 3004 4936 2973.4 4171 35967
> Total: 3308 5243 2973.7 4472 36273
>
> Percentage of the requests served within a certain time (ms)
> 50% 4472
> 66% 4931
> 75% 5415
> 80% 5721
> 90% 6703
> 95% 8345
> 98% 10321
> 99% 25562
> 100% 36273 (longest request)
>
>

[Brian Takita]

On Thu, Mar 26, 2009 at 10:39 AM, Brian Takita <br...@pivotallabs.com> wrote:
> On Thu, Mar 26, 2009 at 2:30 AM, Andrei Erdoss <erd...@gmail.com> wrote:
>> Thank you for your reply. I did some performance testing using a plugin that
>> uses Desert and Passenger. The results were pretty bad.
>>
>> This is done without preloading. Can you give me an example on how I would
>> preload this plugin into environment.rb? Do I have to include all classes
>> and modules?
> That would probably be a good idea.
>
> Desert causes more file searching and loading because it loads all
> matches, rather than the first match, when you use require or the
> constant autoloading. On Passenger, there would be performance issues
> if each forked process needed to load all of the files.

One simplistic solution is to do a directory glob of all of your
models, controllers, and lib files and require them.

[Andrei Erdoss]

On Thu, Mar 26, 2009 at 7:42 PM, Brian Takita <br...@pivotallabs.com> wrote:

On Thu, Mar 26, 2009 at 10:39 AM, Brian Takita <br...@pivotallabs.com> wrote:
> On Thu, Mar 26, 2009 at 2:30 AM, Andrei Erdoss <erd...@gmail.com> wrote:
>> Thank you for your reply. I did some performance testing using a plugin that
>> uses Desert and Passenger. The results were pretty bad.
>>
>> This is done without preloading. Can you give me an example on how I would
>> preload this plugin into environment.rb? Do I have to include all classes
>> and modules?
> That would probably be a good idea.
>
> Desert causes more file searching and loading because it loads all
> matches, rather than the first match, when you use require or the
> constant autoloading. On Passenger, there would be performance issues
> if each forked process needed to load all of the files.

One simplistic solution is to do a directory glob of all of your
models, controllers, and lib files and require them.

Something like this?

Dir.foreach( “#{RAILS_ROOT}/app/controllers” ) {|f| $logger.d “r #{f}”; silence_warnings{require_dependency f} if f =~ /\.rb$/}
Dir.foreach( “#{RAILS_ROOT}/app/models” ) {|f| $logger.d “r #{f}”; silence_warnings{require_dependency f} if f =~ /\.rb$/}

Can you give me an example of this. I'm not 100% on how to do that.

 

--
Andrei Erdoss

[Brian Takita]

You may want something like:

Dir["#{RAILS_ROOT}/app/controllers/**/*.rb"] do |f|

$logger.d “r #{f}”;
silence_warnings{require_dependency

f.gsub("#{RAILS_ROOT}/app/controllers/", "")}
end
Dir["#{RAILS_ROOT}/app/models/**/*.rb"] do |f|
$logger.d “r #{f}”;
silence_warnings{require_dependency f.gsub("#{RAILS_ROOT}/app/models/", "")}
end

This will cause the models/controllers in the plugins to be loaded first.

[Brian Takita]

This solution is not complete, because plugins may define classes that
are not reopened in the application.
This means plugins need to be accounted for.

[Andrei Erdoss]

Well, I would be interested to load all the files (models, controllers) from the plugins managed by desert. Currently I am using a Tog, a platform that is made up of different plugins for social networking.

What would be a good way to load the plugins? 

--
Andrei Erdoss