Thanks for the information, I wan't aware of this issue. I am
Rails developer, and I have been watching Mongo as an alternative to
CouchDB, if we jump into Mongo I will let you know my experiences with
your fixes.
> I just wanted to remind that Rails developers need to tweak their
> 'environment.rb' if they use MongoDB with Passenger.
> By default, Passenger spawns Ruby processes with fork(). And, as fork
> () shares file descriptors, the Rails app has to reopen the connection
> to MongoDB in the fresh new "process". Seehttp://www.modrails.org/documentation/Users%20guide.html#RailsSpawnMe...
> for more details.
> I just wanted to remind that Rails developers need to tweak their
> 'environment.rb' if they use MongoDB with Passenger.
> By default, Passenger spawns Ruby processes with fork(). And, as fork
> () shares file descriptors, the Rails app has to reopen the connection
> to MongoDB in the fresh new "process". Seehttp://www.modrails.org/documentation/Users%20guide.html#RailsSpawnMe...
> for more details.
I wonder if this code protects against the failed to allocate memory
issue. Jim, you guys are using mongo record, which shares connection
stuff through all models with a class variable. Maybe the fact that
mongomapper allows customizing connection/database per model has
something messed up that makes it so you need this stuff if on
passenger.
Nicolas, is the failed to allocate issue what you ran into and does
this fix it or was it something else?
On Aug 5, 2009, at 11:48 AM, Jim Mulholland wrote:
> We have been using MongoDB with nginx Passenger for quite a while now
> without any modifications to our environment.rb without any issues.
> Is this code to help performance?
> On Aug 5, 5:24 am, nicolas_ <nicolas.fou...@gmail.com> wrote:
>> Hi,
>> I just wanted to remind that Rails developers need to tweak their
>> 'environment.rb' if they use MongoDB with Passenger.
>> By default, Passenger spawns Ruby processes with fork(). And, as fork
>> () shares file descriptors, the Rails app has to reopen the
>> connection
>> to MongoDB in the fresh new "process". Seehttp://www.modrails.org/documentation/Users%20guide.html#RailsSpawnMe >> ...
>> for more details.
In a nutshell: our application does not work at all without updating
'environment.rb'. Here are more details:
I use Apache, and I set RailsSpawnMethod to 'smart-lv2'. If I just do
a "ab -n 100 -c 100 http://xxx.com", I will have a timeout (of course,
the page I benchmark does one query to MongoDB). In the Ruby driver,
there is one socket per database, and this socket is protected by a
semaphore. As I use one database for my whole app
(MongoMapper.database), calls are blocked by the semaphore. Here are
the Apache/Passenger logs:
[Tue Aug 04 17:25:54 2009] [notice] Apache/2.2.9 (Unix) DAV/2
Phusion_Passenger/2.2.4 configured -- resuming normal operations
[Tue Aug 04 17:28:03 2009] [error] [client 127.0.0.1] Premature end of
script headers: notes
[ pid=25149 file=ext/apache2/Hooks.cpp:638 time=2009-08-04
17:28:03.880 ]:
No data received from the backend application (process 25187) within
120000 msec. Either the backend application is frozen, or your TimeOut
value of 120 seconds is too low. Please check whether your application
is frozen, or increase the value of the TimeOut configuration
directive.
[Tue Aug 04 17:28:04 2009] [error] [client 127.0.0.1] Premature end of
script headers: notes
[ pid=25151 file=ext/apache2/Hooks.cpp:638 time=2009-08-04
17:28:04.36 ]:
No data received from the backend application (process 25189) within
120000 msec. Either the backend application is frozen, or your TimeOut
value of 120 seconds is too low. Please check whether your application
is frozen, or increase the value of the TimeOut configuration
directive.
In the MongoDB logs, I see one permanent connection, and another
connection appears let's say every 5 seconds and closes
instantaneously.
If I applied what I shown in my first post (reset the MongoDB
connection in each passenger fork()), I just see as many opened
connections as there are forked processes.
Nicolas
On Aug 5, 5:48 pm, Jim Mulholland <j...@squeejee.com> wrote:
> We have been using MongoDB with nginx Passenger for quite a while now
> without any modifications to our environment.rb without any issues.
> Is this code to help performance?
> On Aug 5, 5:24 am, nicolas_ <nicolas.fou...@gmail.com> wrote:
> > Hi,
> > I just wanted to remind that Rails developers need to tweak their
> > 'environment.rb' if they use MongoDB with Passenger.
> > By default, Passenger spawns Ruby processes with fork(). And, as fork
> > () shares file descriptors, the Rails app has to reopen the connection
> > to MongoDB in the fresh new "process". Seehttp://www.modrails.org/documentation/Users%20guide.html#RailsSpawnMe...
> > for more details.
I don't know MongoRecord. But if it sets an instance of
XGen::Mongo::Driver::DB at a class level, the application will
eventually fail. As I said earlier, the socket will be shared by every
Passenger forks, so every single requests is queued.
I did not see any 'failed to allocate memory' error, but perhaps I
didn't look at the right place. (my syslog configuration was not set
properly at this time).
But yeah, when I see the backtrace of
http://groups.google.com/group/mongodb-user/browse_thread/thread/ff7d... I think that it could be linked. It's too bad he did not give the full
trace. It *seems* that from Cursor.to_a to DB.recv, the semaphore is
not used to protect the socket. That's could explain why the socket is
overloaded. (again, I'm not sure, we don't have the full trace).
Nicolas
On Aug 5, 5:53 pm, John Nunemaker <nunema...@gmail.com> wrote:
> I wonder if this code protects against the failed to allocate memory
> issue. Jim, you guys are using mongo record, which shares connection
> stuff through all models with a class variable. Maybe the fact that
> mongomapper allows customizing connection/database per model has
> something messed up that makes it so you need this stuff if on
> passenger.
> Nicolas, is the failed to allocate issue what you ran into and does
> this fix it or was it something else?
> On Aug 5, 2009, at 11:48 AM, Jim Mulholland wrote:
> > Nicolas,
> > Thanks for the tip.
> > We have been using MongoDB with nginx Passenger for quite a while now
> > without any modifications to our environment.rb without any issues.
> > Is this code to help performance?
> > On Aug 5, 5:24 am, nicolas_ <nicolas.fou...@gmail.com> wrote:
> >> Hi,
> >> I just wanted to remind that Rails developers need to tweak their
> >> 'environment.rb' if they use MongoDB with Passenger.
> >> By default, Passenger spawns Ruby processes with fork(). And, as fork
> >> () shares file descriptors, the Rails app has to reopen the
> >> connection
> >> to MongoDB in the fresh new "process". Seehttp://www.modrails.org/documentation/Users%20guide.html#RailsSpawnMe > >> ...
> >> for more details.
> In a nutshell: our application does not work at all without updating
> 'environment.rb'. Here are more details:
> I use Apache, and I set RailsSpawnMethod to 'smart-lv2'. If I just do
> a "ab -n 100 -c 100http://xxx.com", I will have a timeout (of course,
> the page I benchmark does one query to MongoDB). In the Ruby driver,
> there is one socket per database, and this socket is protected by a
> semaphore. As I use one database for my whole app
> (MongoMapper.database), calls are blocked by the semaphore. Here are
> the Apache/Passenger logs:
> [Tue Aug 04 17:25:54 2009] [notice] Apache/2.2.9 (Unix) DAV/2
> Phusion_Passenger/2.2.4 configured -- resuming normal operations
> [Tue Aug 04 17:28:03 2009] [error] [client 127.0.0.1] Premature end of
> script headers: notes
> [ pid=25149 file=ext/apache2/Hooks.cpp:638 time=2009-08-04
> 17:28:03.880 ]:
> No data received from the backend application (process 25187) within
> 120000 msec. Either the backend application is frozen, or your TimeOut
> value of 120 seconds is too low. Please check whether your application
> is frozen, or increase the value of the TimeOut configuration
> directive.
> [Tue Aug 04 17:28:04 2009] [error] [client 127.0.0.1] Premature end of
> script headers: notes
> [ pid=25151 file=ext/apache2/Hooks.cpp:638 time=2009-08-04
> 17:28:04.36 ]:
> No data received from the backend application (process 25189) within
> 120000 msec. Either the backend application is frozen, or your TimeOut
> value of 120 seconds is too low. Please check whether your application
> is frozen, or increase the value of the TimeOut configuration
> directive.
> In the MongoDB logs, I see one permanent connection, and another
> connection appears let's say every 5 seconds and closes
> instantaneously.
> If I applied what I shown in my first post (reset the MongoDB
> connection in each passenger fork()), I just see as many opened
> connections as there are forked processes.
> Nicolas
> On Aug 5, 5:48 pm, Jim Mulholland <j...@squeejee.com> wrote:
> > Nicolas,
> > Thanks for the tip.
> > We have been using MongoDB with nginx Passenger for quite a while now
> > without any modifications to our environment.rb without any issues.
> > Is this code to help performance?
> > On Aug 5, 5:24 am, nicolas_ <nicolas.fou...@gmail.com> wrote:
> > > Hi,
> > > I just wanted to remind that Rails developers need to tweak their
> > > 'environment.rb' if they use MongoDB with Passenger.
> > > By default, Passenger spawns Ruby processes with fork(). And, as fork
> > > () shares file descriptors, the Rails app has to reopen the connection
> > > to MongoDB in the fresh new "process". Seehttp://www.modrails.org/documentation/Users%20guide.html#RailsSpawnMe...
> > > for more details.
Very bizarre. I wonder if it has something to do with MongoMapper
like John mentioned in this thread earlier.
We have been using MongoRecord in Production for several months now
with hundreds of thousands of page views a month on Passenger without
any issues. It seems like we would have seen this issue by now.
On Aug 5, 12:32 pm, Nicolas Fouché <nicolas.fou...@gmail.com> wrote:
> I forgot one thing: Passenger did not recover from the 'ab' test. I
> had to stop then start Apache to be able to use the application again.
> Nicolas
> On Aug 5, 7:10 pm, Nicolas Fouché <nicolas.fou...@gmail.com> wrote:
> > Jim,
> > In a nutshell: our application does not work at all without updating
> > 'environment.rb'. Here are more details:
> > I use Apache, and I set RailsSpawnMethod to 'smart-lv2'. If I just do
> > a "ab -n 100 -c 100http://xxx.com", I will have a timeout (of course,
> > the page I benchmark does one query to MongoDB). In the Ruby driver,
> > there is one socket per database, and this socket is protected by a
> > semaphore. As I use one database for my whole app
> > (MongoMapper.database), calls are blocked by the semaphore. Here are
> > the Apache/Passenger logs:
> > [Tue Aug 04 17:25:54 2009] [notice] Apache/2.2.9 (Unix) DAV/2
> > Phusion_Passenger/2.2.4 configured -- resuming normal operations
> > [Tue Aug 04 17:28:03 2009] [error] [client 127.0.0.1] Premature end of
> > script headers: notes
> > [ pid=25149 file=ext/apache2/Hooks.cpp:638 time=2009-08-04
> > 17:28:03.880 ]:
> > No data received from the backend application (process 25187) within
> > 120000 msec. Either the backend application is frozen, or your TimeOut
> > value of 120 seconds is too low. Please check whether your application
> > is frozen, or increase the value of the TimeOut configuration
> > directive.
> > [Tue Aug 04 17:28:04 2009] [error] [client 127.0.0.1] Premature end of
> > script headers: notes
> > [ pid=25151 file=ext/apache2/Hooks.cpp:638 time=2009-08-04
> > 17:28:04.36 ]:
> > No data received from the backend application (process 25189) within
> > 120000 msec. Either the backend application is frozen, or your TimeOut
> > value of 120 seconds is too low. Please check whether your application
> > is frozen, or increase the value of the TimeOut configuration
> > directive.
> > In the MongoDB logs, I see one permanent connection, and another
> > connection appears let's say every 5 seconds and closes
> > instantaneously.
> > If I applied what I shown in my first post (reset the MongoDB
> > connection in each passenger fork()), I just see as many opened
> > connections as there are forked processes.
> > Nicolas
> > On Aug 5, 5:48 pm, Jim Mulholland <j...@squeejee.com> wrote:
> > > Nicolas,
> > > Thanks for the tip.
> > > We have been using MongoDB with nginx Passenger for quite a while now
> > > without any modifications to our environment.rb without any issues.
> > > Is this code to help performance?
> > > On Aug 5, 5:24 am, nicolas_ <nicolas.fou...@gmail.com> wrote:
> > > > Hi,
> > > > I just wanted to remind that Rails developers need to tweak their
> > > > 'environment.rb' if they use MongoDB with Passenger.
> > > > By default, Passenger spawns Ruby processes with fork(). And, as fork
> > > > () shares file descriptors, the Rails app has to reopen the connection
> > > > to MongoDB in the fresh new "process". Seehttp://www.modrails.org/documentation/Users%20guide.html#RailsSpawnMe...
> > > > for more details.
> Very bizarre. I wonder if it has something to do with MongoMapper
> like John mentioned in this thread earlier.
> We have been using MongoRecord in Production for several months now
> with hundreds of thousands of page views a month on Passenger without
> any issues. It seems like we would have seen this issue by now.
> On Aug 5, 12:32 pm, Nicolas Fouché <nicolas.fou...@gmail.com> wrote:
> > Jim,
> > I forgot one thing: Passenger did not recover from the 'ab' test. I
> > had to stop then start Apache to be able to use the application again.
> > Nicolas
> > On Aug 5, 7:10 pm, Nicolas Fouché <nicolas.fou...@gmail.com> wrote:
> > > Jim,
> > > In a nutshell: our application does not work at all without updating
> > > 'environment.rb'. Here are more details:
> > > I use Apache, and I set RailsSpawnMethod to 'smart-lv2'. If I just do
> > > a "ab -n 100 -c 100http://xxx.com", I will have a timeout (of course,
> > > the page I benchmark does one query to MongoDB). In the Ruby driver,
> > > there is one socket per database, and this socket is protected by a
> > > semaphore. As I use one database for my whole app
> > > (MongoMapper.database), calls are blocked by the semaphore. Here are
> > > the Apache/Passenger logs:
> > > [Tue Aug 04 17:25:54 2009] [notice] Apache/2.2.9 (Unix) DAV/2
> > > Phusion_Passenger/2.2.4 configured -- resuming normal operations
> > > [Tue Aug 04 17:28:03 2009] [error] [client 127.0.0.1] Premature end of
> > > script headers: notes
> > > [ pid=25149 file=ext/apache2/Hooks.cpp:638 time=2009-08-04
> > > 17:28:03.880 ]:
> > > No data received from the backend application (process 25187) within
> > > 120000 msec. Either the backend application is frozen, or your TimeOut
> > > value of 120 seconds is too low. Please check whether your application
> > > is frozen, or increase the value of the TimeOut configuration
> > > directive.
> > > [Tue Aug 04 17:28:04 2009] [error] [client 127.0.0.1] Premature end of
> > > script headers: notes
> > > [ pid=25151 file=ext/apache2/Hooks.cpp:638 time=2009-08-04
> > > 17:28:04.36 ]:
> > > No data received from the backend application (process 25189) within
> > > 120000 msec. Either the backend application is frozen, or your TimeOut
> > > value of 120 seconds is too low. Please check whether your application
> > > is frozen, or increase the value of the TimeOut configuration
> > > directive.
> > > In the MongoDB logs, I see one permanent connection, and another
> > > connection appears let's say every 5 seconds and closes
> > > instantaneously.
> > > If I applied what I shown in my first post (reset the MongoDB
> > > connection in each passenger fork()), I just see as many opened
> > > connections as there are forked processes.
> > > Nicolas
> > > On Aug 5, 5:48 pm, Jim Mulholland <j...@squeejee.com> wrote:
> > > > Nicolas,
> > > > Thanks for the tip.
> > > > We have been using MongoDB with nginx Passenger for quite a while now
> > > > without any modifications to our environment.rb without any issues.
> > > > Is this code to help performance?
> > > > On Aug 5, 5:24 am, nicolas_ <nicolas.fou...@gmail.com> wrote:
> > > > > Hi,
> > > > > I just wanted to remind that Rails developers need to tweak their
> > > > > 'environment.rb' if they use MongoDB with Passenger.
> > > > > By default, Passenger spawns Ruby processes with fork(). And, as fork
> > > > > () shares file descriptors, the Rails app has to reopen the connection
> > > > > to MongoDB in the fresh new "process". Seehttp://www.modrails.org/documentation/Users%20guide.html#RailsSpawnMe...
> > > > > for more details.
I'm pretty sure the error only happens with mongomapper. MongoRecord
stores database and connections tuff with @@ class variables so it
just uses one. I don't know much about this but that is my guess.
> Very bizarre. I wonder if it has something to do with MongoMapper
> like John mentioned in this thread earlier.
> We have been using MongoRecord in Production for several months now
> with hundreds of thousands of page views a month on Passenger without
> any issues. It seems like we would have seen this issue by now.
> On Aug 5, 12:32 pm, Nicolas Fouché <nicolas.fou...@gmail.com> wrote:
>> Jim,
>> I forgot one thing: Passenger did not recover from the 'ab' test. I
>> had to stop then start Apache to be able to use the application
>> again.
>> Nicolas
>> On Aug 5, 7:10 pm, Nicolas Fouché <nicolas.fou...@gmail.com> wrote:
>>> Jim,
>>> In a nutshell: our application does not work at all without updating
>>> 'environment.rb'. Here are more details:
>>> I use Apache, and I set RailsSpawnMethod to 'smart-lv2'. If I just
>>> do
>>> a "ab -n 100 -c 100http://xxx.com", I will have a timeout (of
>>> course,
>>> the page I benchmark does one query to MongoDB). In the Ruby driver,
>>> there is one socket per database, and this socket is protected by a
>>> semaphore. As I use one database for my whole app
>>> (MongoMapper.database), calls are blocked by the semaphore. Here are
>>> the Apache/Passenger logs:
>>> [Tue Aug 04 17:25:54 2009] [notice] Apache/2.2.9 (Unix) DAV/2
>>> Phusion_Passenger/2.2.4 configured -- resuming normal operations
>>> [Tue Aug 04 17:28:03 2009] [error] [client 127.0.0.1] Premature
>>> end of
>>> script headers: notes
>>> [ pid=25149 file=ext/apache2/Hooks.cpp:638 time=2009-08-04
>>> 17:28:03.880 ]:
>>> No data received from the backend application (process 25187)
>>> within
>>> 120000 msec. Either the backend application is frozen, or your
>>> TimeOut
>>> value of 120 seconds is too low. Please check whether your
>>> application
>>> is frozen, or increase the value of the TimeOut configuration
>>> directive.
>>> [Tue Aug 04 17:28:04 2009] [error] [client 127.0.0.1] Premature
>>> end of
>>> script headers: notes
>>> [ pid=25151 file=ext/apache2/Hooks.cpp:638 time=2009-08-04
>>> 17:28:04.36 ]:
>>> No data received from the backend application (process 25189)
>>> within
>>> 120000 msec. Either the backend application is frozen, or your
>>> TimeOut
>>> value of 120 seconds is too low. Please check whether your
>>> application
>>> is frozen, or increase the value of the TimeOut configuration
>>> directive.
>>> In the MongoDB logs, I see one permanent connection, and another
>>> connection appears let's say every 5 seconds and closes
>>> instantaneously.
>>> If I applied what I shown in my first post (reset the MongoDB
>>> connection in each passenger fork()), I just see as many opened
>>> connections as there are forked processes.
>>> Nicolas
>>> On Aug 5, 5:48 pm, Jim Mulholland <j...@squeejee.com> wrote:
>>>> Nicolas,
>>>> Thanks for the tip.
>>>> We have been using MongoDB with nginx Passenger for quite a while
>>>> now
>>>> without any modifications to our environment.rb without any issues.
>>>> Is this code to help performance?
>>>> On Aug 5, 5:24 am, nicolas_ <nicolas.fou...@gmail.com> wrote:
>>>>> Hi,
>>>>> I just wanted to remind that Rails developers need to tweak their
>>>>> 'environment.rb' if they use MongoDB with Passenger.
>>>>> By default, Passenger spawns Ruby processes with fork(). And, as
>>>>> fork
>>>>> () shares file descriptors, the Rails app has to reopen the
>>>>> connection
>>>>> to MongoDB in the fresh new "process". Seehttp://www.modrails.org/documentation/Users%20guide.html#RailsSpawnMe >>>>> ...
>>>>> for more details.
Nicolas,
Is there anything I can move into MM itself to help with this problem?
I don't want everyone to have to do all that junk in rails env and
initializers if they don't have to.
> On Aug 5, 7:38 pm, Jim Mulholland <j...@squeejee.com> wrote:
>> Very bizarre. I wonder if it has something to do with MongoMapper
>> like John mentioned in this thread earlier.
>> We have been using MongoRecord in Production for several months now
>> with hundreds of thousands of page views a month on Passenger without
>> any issues. It seems like we would have seen this issue by now.
>> On Aug 5, 12:32 pm, Nicolas Fouché <nicolas.fou...@gmail.com> wrote:
>>> Jim,
>>> I forgot one thing: Passenger did not recover from the 'ab' test. I
>>> had to stop then start Apache to be able to use the application
>>> again.
>>> Nicolas
>>> On Aug 5, 7:10 pm, Nicolas Fouché <nicolas.fou...@gmail.com> wrote:
>>>> Jim,
>>>> In a nutshell: our application does not work at all without
>>>> updating
>>>> 'environment.rb'. Here are more details:
>>>> I use Apache, and I set RailsSpawnMethod to 'smart-lv2'. If I
>>>> just do
>>>> a "ab -n 100 -c 100http://xxx.com", I will have a timeout (of
>>>> course,
>>>> the page I benchmark does one query to MongoDB). In the Ruby
>>>> driver,
>>>> there is one socket per database, and this socket is protected by a
>>>> semaphore. As I use one database for my whole app
>>>> (MongoMapper.database), calls are blocked by the semaphore. Here
>>>> are
>>>> the Apache/Passenger logs:
>>>> [Tue Aug 04 17:25:54 2009] [notice] Apache/2.2.9 (Unix) DAV/2
>>>> Phusion_Passenger/2.2.4 configured -- resuming normal operations
>>>> [Tue Aug 04 17:28:03 2009] [error] [client 127.0.0.1] Premature
>>>> end of
>>>> script headers: notes
>>>> [ pid=25149 file=ext/apache2/Hooks.cpp:638 time=2009-08-04
>>>> 17:28:03.880 ]:
>>>> No data received from the backend application (process 25187)
>>>> within
>>>> 120000 msec. Either the backend application is frozen, or your
>>>> TimeOut
>>>> value of 120 seconds is too low. Please check whether your
>>>> application
>>>> is frozen, or increase the value of the TimeOut configuration
>>>> directive.
>>>> [Tue Aug 04 17:28:04 2009] [error] [client 127.0.0.1] Premature
>>>> end of
>>>> script headers: notes
>>>> [ pid=25151 file=ext/apache2/Hooks.cpp:638 time=2009-08-04
>>>> 17:28:04.36 ]:
>>>> No data received from the backend application (process 25189)
>>>> within
>>>> 120000 msec. Either the backend application is frozen, or your
>>>> TimeOut
>>>> value of 120 seconds is too low. Please check whether your
>>>> application
>>>> is frozen, or increase the value of the TimeOut configuration
>>>> directive.
>>>> In the MongoDB logs, I see one permanent connection, and another
>>>> connection appears let's say every 5 seconds and closes
>>>> instantaneously.
>>>> If I applied what I shown in my first post (reset the MongoDB
>>>> connection in each passenger fork()), I just see as many opened
>>>> connections as there are forked processes.
>>>> Nicolas
>>>> On Aug 5, 5:48 pm, Jim Mulholland <j...@squeejee.com> wrote:
>>>>> Nicolas,
>>>>> Thanks for the tip.
>>>>> We have been using MongoDB with nginx Passenger for quite a
>>>>> while now
>>>>> without any modifications to our environment.rb without any
>>>>> issues.
>>>>> Is this code to help performance?
>>>>> On Aug 5, 5:24 am, nicolas_ <nicolas.fou...@gmail.com> wrote:
>>>>>> Hi,
>>>>>> I just wanted to remind that Rails developers need to tweak their
>>>>>> 'environment.rb' if they use MongoDB with Passenger.
>>>>>> By default, Passenger spawns Ruby processes with fork(). And,
>>>>>> as fork
>>>>>> () shares file descriptors, the Rails app has to reopen the
>>>>>> connection
>>>>>> to MongoDB in the fresh new "process". Seehttp://www.modrails.org/documentation/Users%20guide.html#RailsSpawnMe >>>>>> ...
>>>>>> for more details.
I don't think it's a "problem". It's part of how Passenger works, and
the documentation says that each developer has to do this manually.
Passenger only handles ActiveRecord to open new sockets in forks.
Don't you think that it's better to create new connections in each
fork instead of using the same connections ? That way, it acts as an
equivalent to a Mongrel or Thin cluster. The MongoDB server sees x
clients instead of one, like MySQL sees x clients. So you don't
overload one single client socket when you're slashdotted.
Perhaps MongoMapper could find all XGen::Mongo::Driver::DB instances
and call +DB.connect_to_master+. I did not look deeply so I don't know
if this would break the 'replica pair' thing in the driver.
Nicolas
On Aug 5, 8:21 pm, John Nunemaker <nunema...@gmail.com> wrote:
> Nicolas,
> Is there anything I can move into MM itself to help with this problem?
> I don't want everyone to have to do all that junk in rails env and
> initializers if they don't have to.
> On Aug 5, 2009, at 1:54 PM, Nicolas Fouché wrote:
> > On Aug 5, 7:38 pm, Jim Mulholland <j...@squeejee.com> wrote:
> >> Very bizarre. I wonder if it has something to do with MongoMapper
> >> like John mentioned in this thread earlier.
> >> We have been using MongoRecord in Production for several months now
> >> with hundreds of thousands of page views a month on Passenger without
> >> any issues. It seems like we would have seen this issue by now.
> >> On Aug 5, 12:32 pm, Nicolas Fouché <nicolas.fou...@gmail.com> wrote:
> >>> Jim,
> >>> I forgot one thing: Passenger did not recover from the 'ab' test. I
> >>> had to stop then start Apache to be able to use the application
> >>> again.
> >>> Nicolas
> >>> On Aug 5, 7:10 pm, Nicolas Fouché <nicolas.fou...@gmail.com> wrote:
> >>>> Jim,
> >>>> In a nutshell: our application does not work at all without
> >>>> updating
> >>>> 'environment.rb'. Here are more details:
> >>>> I use Apache, and I set RailsSpawnMethod to 'smart-lv2'. If I
> >>>> just do
> >>>> a "ab -n 100 -c 100http://xxx.com", I will have a timeout (of
> >>>> course,
> >>>> the page I benchmark does one query to MongoDB). In the Ruby
> >>>> driver,
> >>>> there is one socket per database, and this socket is protected by a
> >>>> semaphore. As I use one database for my whole app
> >>>> (MongoMapper.database), calls are blocked by the semaphore. Here
> >>>> are
> >>>> the Apache/Passenger logs:
> >>>> [Tue Aug 04 17:25:54 2009] [notice] Apache/2.2.9 (Unix) DAV/2
> >>>> Phusion_Passenger/2.2.4 configured -- resuming normal operations
> >>>> [Tue Aug 04 17:28:03 2009] [error] [client 127.0.0.1] Premature
> >>>> end of
> >>>> script headers: notes
> >>>> [ pid=25149 file=ext/apache2/Hooks.cpp:638 time=2009-08-04
> >>>> 17:28:03.880 ]:
> >>>> No data received from the backend application (process 25187)
> >>>> within
> >>>> 120000 msec. Either the backend application is frozen, or your
> >>>> TimeOut
> >>>> value of 120 seconds is too low. Please check whether your
> >>>> application
> >>>> is frozen, or increase the value of the TimeOut configuration
> >>>> directive.
> >>>> [Tue Aug 04 17:28:04 2009] [error] [client 127.0.0.1] Premature
> >>>> end of
> >>>> script headers: notes
> >>>> [ pid=25151 file=ext/apache2/Hooks.cpp:638 time=2009-08-04
> >>>> 17:28:04.36 ]:
> >>>> No data received from the backend application (process 25189)
> >>>> within
> >>>> 120000 msec. Either the backend application is frozen, or your
> >>>> TimeOut
> >>>> value of 120 seconds is too low. Please check whether your
> >>>> application
> >>>> is frozen, or increase the value of the TimeOut configuration
> >>>> directive.
> >>>> In the MongoDB logs, I see one permanent connection, and another
> >>>> connection appears let's say every 5 seconds and closes
> >>>> instantaneously.
> >>>> If I applied what I shown in my first post (reset the MongoDB
> >>>> connection in each passenger fork()), I just see as many opened
> >>>> connections as there are forked processes.
> >>>> Nicolas
> >>>> On Aug 5, 5:48 pm, Jim Mulholland <j...@squeejee.com> wrote:
> >>>>> Nicolas,
> >>>>> Thanks for the tip.
> >>>>> We have been using MongoDB with nginx Passenger for quite a
> >>>>> while now
> >>>>> without any modifications to our environment.rb without any
> >>>>> issues.
> >>>>> Is this code to help performance?
> >>>>> On Aug 5, 5:24 am, nicolas_ <nicolas.fou...@gmail.com> wrote:
> >>>>>> Hi,
> >>>>>> I just wanted to remind that Rails developers need to tweak their
> >>>>>> 'environment.rb' if they use MongoDB with Passenger.
> >>>>>> By default, Passenger spawns Ruby processes with fork(). And,
> >>>>>> as fork
> >>>>>> () shares file descriptors, the Rails app has to reopen the
> >>>>>> connection
> >>>>>> to MongoDB in the fresh new "process". Seehttp://www.modrails.org/documentation/Users%20guide.html#RailsSpawnMe > >>>>>> ...
> >>>>>> for more details.
> I don't think it's a "problem". It's part of how Passenger works, and
> the documentation says that each developer has to do this manually.
> Passenger only handles ActiveRecord to open new sockets in forks.
> Don't you think that it's better to create new connections in each
> fork instead of using the same connections ? That way, it acts as an
> equivalent to a Mongrel or Thin cluster. The MongoDB server sees x
> clients instead of one, like MySQL sees x clients. So you don't
> overload one single client socket when you're slashdotted.
> Perhaps MongoMapper could find all XGen::Mongo::Driver::DB instances
> and call +DB.connect_to_master+. I did not look deeply so I don't know
> if this would break the 'replica pair' thing in the driver.
> Nicolas
> On Aug 5, 8:21 pm, John Nunemaker <nunema...@gmail.com> wrote:
> > Nicolas,
> > Is there anything I can move into MM itself to help with this problem?
> > I don't want everyone to have to do all that junk in rails env and
> > initializers if they don't have to.
> > On Aug 5, 2009, at 1:54 PM, Nicolas Fouché wrote:
> > > On Aug 5, 7:38 pm, Jim Mulholland <j...@squeejee.com> wrote:
> > >> Very bizarre. I wonder if it has something to do with MongoMapper
> > >> like John mentioned in this thread earlier.
> > >> We have been using MongoRecord in Production for several months now
> > >> with hundreds of thousands of page views a month on Passenger without
> > >> any issues. It seems like we would have seen this issue by now.
> > >> On Aug 5, 12:32 pm, Nicolas Fouché <nicolas.fou...@gmail.com> wrote:
> > >>> Jim,
> > >>> I forgot one thing: Passenger did not recover from the 'ab' test. I
> > >>> had to stop then start Apache to be able to use the application
> > >>> again.
> > >>> Nicolas
> > >>> On Aug 5, 7:10 pm, Nicolas Fouché <nicolas.fou...@gmail.com> wrote:
> > >>>> Jim,
> > >>>> In a nutshell: our application does not work at all without
> > >>>> updating
> > >>>> 'environment.rb'. Here are more details:
> > >>>> I use Apache, and I set RailsSpawnMethod to 'smart-lv2'. If I
> > >>>> just do
> > >>>> a "ab -n 100 -c 100http://xxx.com", I will have a timeout (of
> > >>>> course,
> > >>>> the page I benchmark does one query to MongoDB). In the Ruby
> > >>>> driver,
> > >>>> there is one socket per database, and this socket is protected by a
> > >>>> semaphore. As I use one database for my whole app
> > >>>> (MongoMapper.database), calls are blocked by the semaphore. Here
> > >>>> are
> > >>>> the Apache/Passenger logs:
> > >>>> [Tue Aug 04 17:25:54 2009] [notice] Apache/2.2.9 (Unix) DAV/2
> > >>>> Phusion_Passenger/2.2.4 configured -- resuming normal operations
> > >>>> [Tue Aug 04 17:28:03 2009] [error] [client 127.0.0.1] Premature
> > >>>> end of
> > >>>> script headers: notes
> > >>>> [ pid=25149 file=ext/apache2/Hooks.cpp:638 time=2009-08-04
> > >>>> 17:28:03.880 ]:
> > >>>> No data received from the backend application (process 25187)
> > >>>> within
> > >>>> 120000 msec. Either the backend application is frozen, or your
> > >>>> TimeOut
> > >>>> value of 120 seconds is too low. Please check whether your
> > >>>> application
> > >>>> is frozen, or increase the value of the TimeOut configuration
> > >>>> directive.
> > >>>> [Tue Aug 04 17:28:04 2009] [error] [client 127.0.0.1] Premature
> > >>>> end of
> > >>>> script headers: notes
> > >>>> [ pid=25151 file=ext/apache2/Hooks.cpp:638 time=2009-08-04
> > >>>> 17:28:04.36 ]:
> > >>>> No data received from the backend application (process 25189)
> > >>>> within
> > >>>> 120000 msec. Either the backend application is frozen, or your
> > >>>> TimeOut
> > >>>> value of 120 seconds is too low. Please check whether your
> > >>>> application
> > >>>> is frozen, or increase the value of the TimeOut configuration
> > >>>> directive.
> > >>>> In the MongoDB logs, I see one permanent connection, and another
> > >>>> connection appears let's say every 5 seconds and closes
> > >>>> instantaneously.
> > >>>> If I applied what I shown in my first post (reset the MongoDB
> > >>>> connection in each passenger fork()), I just see as many opened
> > >>>> connections as there are forked processes.
> > >>>> Nicolas
> > >>>> On Aug 5, 5:48 pm, Jim Mulholland <j...@squeejee.com> wrote:
> > >>>>> Nicolas,
> > >>>>> Thanks for the tip.
> > >>>>> We have been using MongoDB with nginx Passenger for quite a
> > >>>>> while now
> > >>>>> without any modifications to our environment.rb without any
> > >>>>> issues.
> > >>>>> Is this code to help performance?
> > >>>>> On Aug 5, 5:24 am, nicolas_ <nicolas.fou...@gmail.com> wrote:
> > >>>>>> Hi,
> > >>>>>> I just wanted to remind that Rails developers need to tweak their
> > >>>>>> 'environment.rb' if they use MongoDB with Passenger.
> > >>>>>> By default, Passenger spawns Ruby processes with fork(). And,
> > >>>>>> as fork
> > >>>>>> () shares file descriptors, the Rails app has to reopen the
> > >>>>>> connection
> > >>>>>> to MongoDB in the fresh new "process". Seehttp://www.modrails.org/documentation/Users%20guide.html#RailsSpawnMe > > >>>>>> ...
> > >>>>>> for more details.
After looking into this issue a little more it appears to be an issue
with more than just MongoMapper. I have added some notes to the ruby
driver README as well as a fix for the issue for the activerecord- mongo-adaptor (which was straightforward since passenger handles this
for activerecord).
It would probably be possible to deal with this within MongoMapper by
registering an event handler for passenger starting_worker_process
events.
> We did our first Pro build with MongoMapper last night and were
> getting the dreaded "failed to allocate memory" error using Nginx
> Passenger.
> We added your Pastie fix and all appears to be working now, so John
> was right in that is MongoMapper specific.
> Thank you so much for sharing!
> - Jim
> On Aug 6, 6:21 am, Nicolas Fouché <nicolas.fou...@gmail.com> wrote:
>> John,
>> I don't think it's a "problem". It's part of how Passenger works, and
>> the documentation says that each developer has to do this manually.
>> Passenger only handles ActiveRecord to open new sockets in forks.
>> Don't you think that it's better to create new connections in each
>> fork instead of using the same connections ? That way, it acts as an
>> equivalent to a Mongrel or Thin cluster. The MongoDB server sees x
>> clients instead of one, like MySQL sees x clients. So you don't
>> overload one single client socket when you're slashdotted.
>> Spawn and New Relic RPM handle it in their own code:http:// >> github.com/tra/spawn/commit/ >> 3a7a730bd7b08a89b757ae6fca26dfd7d6...http://github.com/newrelic/rpm/ >> commit/466554dd103dfeaffdae0b962e552c2...
>> Nanite and multi_db warn the developer in the README:http:// >> github.com/ezmobius/nanite/tree/masterhttp://github.com/schoefmax/ >> multi_db/tree/master
>> Perhaps MongoMapper could find all XGen::Mongo::Driver::DB instances
>> and call +DB.connect_to_master+. I did not look deeply so I don't
>> know
>> if this would break the 'replica pair' thing in the driver.
>> Nicolas
>> On Aug 5, 8:21 pm, John Nunemaker <nunema...@gmail.com> wrote:
>>> Nicolas,
>>> Is there anything I can move into MM itself to help with this
>>> problem?
>>> I don't want everyone to have to do all that junk in rails env and
>>> initializers if they don't have to.
>>> On Aug 5, 2009, at 1:54 PM, Nicolas Fouché wrote:
>>>> On Aug 5, 7:38 pm, Jim Mulholland <j...@squeejee.com> wrote:
>>>>> Very bizarre. I wonder if it has something to do with MongoMapper
>>>>> like John mentioned in this thread earlier.
>>>>> We have been using MongoRecord in Production for several months
>>>>> now
>>>>> with hundreds of thousands of page views a month on Passenger
>>>>> without
>>>>> any issues. It seems like we would have seen this issue by now.
>>>>> On Aug 5, 12:32 pm, Nicolas Fouché <nicolas.fou...@gmail.com>
>>>>> wrote:
>>>>>> Jim,
>>>>>> I forgot one thing: Passenger did not recover from the 'ab'
>>>>>> test. I
>>>>>> had to stop then start Apache to be able to use the application
>>>>>> again.
>>>>>> Nicolas
>>>>>> On Aug 5, 7:10 pm, Nicolas Fouché <nicolas.fou...@gmail.com>
>>>>>> wrote:
>>>>>>> Jim,
>>>>>>> In a nutshell: our application does not work at all without
>>>>>>> updating
>>>>>>> 'environment.rb'. Here are more details:
>>>>>>> I use Apache, and I set RailsSpawnMethod to 'smart-lv2'. If I
>>>>>>> just do
>>>>>>> a "ab -n 100 -c 100http://xxx.com", I will have a timeout (of
>>>>>>> course,
>>>>>>> the page I benchmark does one query to MongoDB). In the Ruby
>>>>>>> driver,
>>>>>>> there is one socket per database, and this socket is protected
>>>>>>> by a
>>>>>>> semaphore. As I use one database for my whole app
>>>>>>> (MongoMapper.database), calls are blocked by the semaphore. Here
>>>>>>> are
>>>>>>> the Apache/Passenger logs:
>>>>>>> [Tue Aug 04 17:25:54 2009] [notice] Apache/2.2.9 (Unix) DAV/2
>>>>>>> Phusion_Passenger/2.2.4 configured -- resuming normal operations
>>>>>>> [Tue Aug 04 17:28:03 2009] [error] [client 127.0.0.1] Premature
>>>>>>> end of
>>>>>>> script headers: notes
>>>>>>> [ pid=25149 file=ext/apache2/Hooks.cpp:638 time=2009-08-04
>>>>>>> 17:28:03.880 ]:
>>>>>>> No data received from the backend application (process 25187)
>>>>>>> within
>>>>>>> 120000 msec. Either the backend application is frozen, or your
>>>>>>> TimeOut
>>>>>>> value of 120 seconds is too low. Please check whether your
>>>>>>> application
>>>>>>> is frozen, or increase the value of the TimeOut configuration
>>>>>>> directive.
>>>>>>> [Tue Aug 04 17:28:04 2009] [error] [client 127.0.0.1] Premature
>>>>>>> end of
>>>>>>> script headers: notes
>>>>>>> [ pid=25151 file=ext/apache2/Hooks.cpp:638 time=2009-08-04
>>>>>>> 17:28:04.36 ]:
>>>>>>> No data received from the backend application (process 25189)
>>>>>>> within
>>>>>>> 120000 msec. Either the backend application is frozen, or your
>>>>>>> TimeOut
>>>>>>> value of 120 seconds is too low. Please check whether your
>>>>>>> application
>>>>>>> is frozen, or increase the value of the TimeOut configuration
>>>>>>> directive.
>>>>>>> In the MongoDB logs, I see one permanent connection, and another
>>>>>>> connection appears let's say every 5 seconds and closes
>>>>>>> instantaneously.
>>>>>>> If I applied what I shown in my first post (reset the MongoDB
>>>>>>> connection in each passenger fork()), I just see as many opened
>>>>>>> connections as there are forked processes.
>>>>>>> Nicolas
>>>>>>> On Aug 5, 5:48 pm, Jim Mulholland <j...@squeejee.com> wrote:
>>>>>>>> Nicolas,
>>>>>>>> Thanks for the tip.
>>>>>>>> We have been using MongoDB with nginx Passenger for quite a
>>>>>>>> while now
>>>>>>>> without any modifications to our environment.rb without any
>>>>>>>> issues.
>>>>>>>> Is this code to help performance?
>>>>>>>> On Aug 5, 5:24 am, nicolas_ <nicolas.fou...@gmail.com> wrote:
>>>>>>>>> Hi,
>>>>>>>>> I just wanted to remind that Rails developers need to tweak
>>>>>>>>> their
>>>>>>>>> 'environment.rb' if they use MongoDB with Passenger.
>>>>>>>>> By default, Passenger spawns Ruby processes with fork(). And,
>>>>>>>>> as fork
>>>>>>>>> () shares file descriptors, the Rails app has to reopen the
>>>>>>>>> connection
>>>>>>>>> to MongoDB in the fresh new "process". Seehttp://www.modrails.org/documentation/Users%20guide.html#RailsSpawnMe >>>>>>>>> ...
>>>>>>>>> for more details.
On Sep 14, 8:30 pm, Michael Dirolf <m...@10gen.com> wrote:
> After looking into this issue a little more it appears to be an issue
> with more than just MongoMapper. I have added some notes to the ruby
> driver README as well as a fix for the issue for the activerecord-
> mongo-adaptor (which was straightforward since passenger handles this
> for activerecord).
> It would probably be possible to deal with this within MongoMapper by
> registering an event handler for passenger starting_worker_process
> events.
I've been using mongomapper with Rails for a month or so now and
haven't seen any issues like this (failed to allocate memory), have I
just been lucky ?
> On Sep 14, 8:30 pm, Michael Dirolf <m...@10gen.com> wrote: >> After looking into this issue a little more it appears to be an issue >> with more than just MongoMapper. I have added some notes to the ruby >> driver README as well as a fix for the issue for the activerecord- >> mongo-adaptor (which was straightforward since passenger handles this >> for activerecord).
>> It would probably be possible to deal with this within MongoMapper by >> registering an event handler for passenger starting_worker_process >> events.
> I've been using mongomapper with Rails for a month or so now and > haven't seen any issues like this (failed to allocate memory), have I > just been lucky ?
I'm not convinced that this is the same issue as the failed to allocate memory issue, which some people have run into and we've had very little luck narrowing down.
This is an issue with shared resources when Passenger forks a new rails process. I'd expect you would see this when hitting a passenger server with concurrent requests unless you are using conservative spawning or have made the changes suggested at the start of this thread.
|
