Mongo Ruby driver and Phusion Passenger

[nicolas_]

Hi,

I just wanted to remind that Rails developers need to tweak their
'environment.rb' if they use MongoDB with Passenger.

By default, Passenger spawns Ruby processes with fork(). And, as fork
() shares file descriptors, the Rails app has to reopen the connection
to MongoDB in the fresh new "process". See
http://www.modrails.org/documentation/Users%20guide.html#RailsSpawnMethod
for more details.

Here is what I did with MongoMapper: http://pastie.org/572282

Anyone on Passenger here ? How do you guys do ?

Nicolas

[Aníbal Rojas]

Nicolas,

Thanks for the information, I wan't aware of this issue. I am
Rails developer, and I have been watching Mongo as an alternative to
CouchDB, if we jump into Mongo I will let you know my experiences with
your fixes.

--
Aníbal Rojas
Ruby on Rails Web Developer
http://www.google.com/profiles/anibalrojas

On Aug 6, 5:24 am, nicolas_ <nicolas.fou...@gmail.com> wrote:
> Hi,
>
> I just wanted to remind that Rails developers need to tweak their
> 'environment.rb' if they use MongoDB with Passenger.
>
> By default, Passenger spawns Ruby processes with fork(). And, as fork
> () shares file descriptors, the Rails app has to reopen the connection

> to MongoDB in the fresh new "process". Seehttp://www.modrails.org/documentation/Users%20guide.html#RailsSpawnMe...

[Jim Mulholland]

Nicolas,

Thanks for the tip.

We have been using MongoDB with nginx Passenger for quite a while now
without any modifications to our environment.rb without any issues.

Is this code to help performance?

On Aug 5, 5:24 am, nicolas_ <nicolas.fou...@gmail.com> wrote:
> Hi,
>
> I just wanted to remind that Rails developers need to tweak their
> 'environment.rb' if they use MongoDB with Passenger.
>
> By default, Passenger spawns Ruby processes with fork(). And, as fork
> () shares file descriptors, the Rails app has to reopen the connection

> to MongoDB in the fresh new "process". Seehttp://www.modrails.org/documentation/Users%20guide.html#RailsSpawnMe...

[John Nunemaker]

I wonder if this code protects against the failed to allocate memory
issue. Jim, you guys are using mongo record, which shares connection
stuff through all models with a class variable. Maybe the fact that
mongomapper allows customizing connection/database per model has
something messed up that makes it so you need this stuff if on
passenger.

Nicolas, is the failed to allocate issue what you ran into and does
this fix it or was it something else?

[Nicolas Fouché]

Jim,

In a nutshell: our application does not work at all without updating
'environment.rb'. Here are more details:

I use Apache, and I set RailsSpawnMethod to 'smart-lv2'. If I just do
a "ab -n 100 -c 100 http://xxx.com", I will have a timeout (of course,
the page I benchmark does one query to MongoDB). In the Ruby driver,
there is one socket per database, and this socket is protected by a
semaphore. As I use one database for my whole app
(MongoMapper.database), calls are blocked by the semaphore. Here are
the Apache/Passenger logs:

[Tue Aug 04 17:25:54 2009] [notice] Apache/2.2.9 (Unix) DAV/2
Phusion_Passenger/2.2.4 configured -- resuming normal operations
[Tue Aug 04 17:28:03 2009] [error] [client 127.0.0.1] Premature end of
script headers: notes
[ pid=25149 file=ext/apache2/Hooks.cpp:638 time=2009-08-04
17:28:03.880 ]:
No data received from the backend application (process 25187) within
120000 msec. Either the backend application is frozen, or your TimeOut
value of 120 seconds is too low. Please check whether your application
is frozen, or increase the value of the TimeOut configuration
directive.
[Tue Aug 04 17:28:04 2009] [error] [client 127.0.0.1] Premature end of
script headers: notes
[ pid=25151 file=ext/apache2/Hooks.cpp:638 time=2009-08-04
17:28:04.36 ]:
No data received from the backend application (process 25189) within
120000 msec. Either the backend application is frozen, or your TimeOut
value of 120 seconds is too low. Please check whether your application
is frozen, or increase the value of the TimeOut configuration
directive.

In the MongoDB logs, I see one permanent connection, and another
connection appears let's say every 5 seconds and closes
instantaneously.

If I applied what I shown in my first post (reset the MongoDB
connection in each passenger fork()), I just see as many opened
connections as there are forked processes.

Nicolas

[Nicolas Fouché]

John,

I don't know MongoRecord. But if it sets an instance of
XGen::Mongo::Driver::DB at a class level, the application will
eventually fail. As I said earlier, the socket will be shared by every
Passenger forks, so every single requests is queued.

I did not see any 'failed to allocate memory' error, but perhaps I
didn't look at the right place. (my syslog configuration was not set
properly at this time).

But yeah, when I see the backtrace of
http://groups.google.com/group/mongodb-user/browse_thread/thread/ff7d244ffd5350ab/f4db57aa01982251?lnk=gst&q=memory#f4db57aa01982251
I think that it could be linked. It's too bad he did not give the full
trace. It *seems* that from Cursor.to_a to DB.recv, the semaphore is
not used to protect the socket. That's could explain why the socket is
overloaded. (again, I'm not sure, we don't have the full trace).

Nicolas

[Nicolas Fouché]

Jim,

I forgot one thing: Passenger did not recover from the 'ab' test. I
had to stop then start Apache to be able to use the application again.

Nicolas

On Aug 5, 7:10 pm, Nicolas Fouché <nicolas.fou...@gmail.com> wrote:
> Jim,
>
> In a nutshell: our application does not work at all without updating
> 'environment.rb'. Here are more details:
>
> I use Apache, and I set RailsSpawnMethod to 'smart-lv2'. If I just do

> a "ab -n 100 -c 100http://xxx.com", I will have a timeout (of course,

[Jim Mulholland]

Very bizarre. I wonder if it has something to do with MongoMapper
like John mentioned in this thread earlier.

We have been using MongoRecord in Production for several months now
with hundreds of thousands of page views a month on Passenger without
any issues. It seems like we would have seen this issue by now.

[Nicolas Fouché]

Jim,

if you didn't set the spawn method to "conservative", you're right,
that's very bizarre.
Just to share, here the Passenger configuration for Memcached:
http://ericentin.com/2009/06/rails-caching-phusion-passenger-smart-spawning/

Nicolas

[John Nunemaker]

I'm pretty sure the error only happens with mongomapper. MongoRecord
stores database and connections tuff with @@ class variables so it
just uses one. I don't know much about this but that is my guess.

[John Nunemaker]

Nicolas,
Is there anything I can move into MM itself to help with this problem?
I don't want everyone to have to do all that junk in rails env and
initializers if they don't have to.

[Nicolas Fouché]

John,

I don't think it's a "problem". It's part of how Passenger works, and
the documentation says that each developer has to do this manually.
Passenger only handles ActiveRecord to open new sockets in forks.

Don't you think that it's better to create new connections in each
fork instead of using the same connections ? That way, it acts as an
equivalent to a Mongrel or Thin cluster. The MongoDB server sees x
clients instead of one, like MySQL sees x clients. So you don't
overload one single client socket when you're slashdotted.

Spawn and New Relic RPM handle it in their own code:
http://github.com/tra/spawn/commit/3a7a730bd7b08a89b757ae6fca26dfd7d60d9109
http://github.com/newrelic/rpm/commit/466554dd103dfeaffdae0b962e552c2a62e2bb32

Nanite and multi_db warn the developer in the README:
http://github.com/ezmobius/nanite/tree/master
http://github.com/schoefmax/multi_db/tree/master

Perhaps MongoMapper could find all XGen::Mongo::Driver::DB instances
and call +DB.connect_to_master+. I did not look deeply so I don't know
if this would break the 'replica pair' thing in the driver.

Nicolas

On Aug 5, 8:21 pm, John Nunemaker <nunema...@gmail.com> wrote:
> Nicolas,
> Is there anything I can move into MM itself to help with this problem?  
> I don't want everyone to have to do all that junk in rails env and  
> initializers if they don't have to.
>
> On Aug 5, 2009, at 1:54 PM, Nicolas Fouché wrote:
>
>
>
>
>
> > Jim,
>
> > if you didn't set the spawn method to "conservative", you're right,
> > that's very bizarre.
> > Just to share, here the Passenger configuration for Memcached:

> >http://ericentin.com/2009/06/rails-caching-phusion-passenger-smart-sp...

[Jim Mulholland]

Nicolas,

We did our first Pro build with MongoMapper last night and were
getting the dreaded "failed to allocate memory" error using Nginx
Passenger.

We added your Pastie fix and all appears to be working now, so John
was right in that is MongoMapper specific.

Thank you so much for sharing!

- Jim

On Aug 6, 6:21 am, Nicolas Fouché <nicolas.fou...@gmail.com> wrote:
> John,
>
> I don't think it's a "problem". It's part of how Passenger works, and
> the documentation says that each developer has to do this manually.
> Passenger only handles ActiveRecord to open new sockets in forks.
>
> Don't you think that it's better to create new connections in each
> fork instead of using the same connections ? That way, it acts as an
> equivalent to a Mongrel or Thin cluster. The MongoDB server sees x
> clients instead of one, like MySQL sees x clients. So you don't
> overload one single client socket when you're slashdotted.
>

> Spawn and New Relic RPM handle it in their own code:http://github.com/tra/spawn/commit/3a7a730bd7b08a89b757ae6fca26dfd7d6...http://github.com/newrelic/rpm/commit/466554dd103dfeaffdae0b962e552c2...
>
> Nanite and multi_db warn the developer in the README:http://github.com/ezmobius/nanite/tree/masterhttp://github.com/schoefmax/multi_db/tree/master

[Michael Dirolf]

After looking into this issue a little more it appears to be an issue
with more than just MongoMapper. I have added some notes to the ruby
driver README as well as a fix for the issue for the activerecord-
mongo-adaptor (which was straightforward since passenger handles this
for activerecord).

It would probably be possible to deal with this within MongoMapper by
registering an event handler for passenger starting_worker_process
events.

[monkeyhelper]

On Sep 14, 8:30 pm, Michael Dirolf <m...@10gen.com> wrote:
> After looking into this issue a little more it appears to be an issue  
> with more than just MongoMapper. I have added some notes to the ruby  
> driver README as well as a fix for the issue for the activerecord-
> mongo-adaptor (which was straightforward since passenger handles this  
> for activerecord).
>
> It would probably be possible to deal with this within MongoMapper by  
> registering an event handler for passenger starting_worker_process  
> events.
>

I've been using mongomapper with Rails for a month or so now and
haven't seen any issues like this (failed to allocate memory), have I
just been lucky ?

[Michael Dirolf]

I'm not convinced that this is the same issue as the failed to
allocate memory issue, which some people have run into and we've had
very little luck narrowing down.

This is an issue with shared resources when Passenger forks a new
rails process. I'd expect you would see this when hitting a passenger
server with concurrent requests unless you are using conservative
spawning or have made the changes suggested at the start of this thread.