Database connection pool

[JacobM]

If my Rails application has the database pool size set to 5 (the
default) in my database.yml file, and I'm running using Phusion
Passenger, does that mean that there may be up to 5 database
connections for each process that Passenger spawns, or is it 5 total
across all processes?

[Ngoc Dao]

Rails processes do not know one another. I'm pretty sure the answer is
"5 database connections for each process".

[Rafi Jacoby]

Are you running your app in multi-threaded mode? 

AFAIK, that would be the only reason to worry about the pools.

[Fjan]

This stack overflow thread seems to suggest all Passenger threads share one pool:

http://stackoverflow.com/questions/2200586/passenger-hosted-rails-app-painfully-slow-but-the-server-is-a-beast

I'd love to get a definitive confirmation though.

[Hongli Lai]

On Mon, Jul 9, 2012 at 11:56 PM, Fjan <jmf...@gmail.com> wrote:
> This stack overflow thread seems to suggest all Passenger threads share one
> pool:
> http://stackoverflow.com/questions/2200586/passenger-hosted-rails-app-painfully-slow-but-the-server-is-a-beast
>
> I'd love to get a definitive confirmation though.

Not true.

For one, Phusion Passenger runs all applications in external processes
so it's impossible for different applications (or even different
processes of the same application) to share a pool.

There is also no such thing as a "Passenger thread". I assume you must
be referring to the application processes. They're processes, not
threads, and so do not share anything with other processes. There are
I/O threads inside Phusion Passenger but as far as applications are
concerned, this is an invisible implementation detail.

Not sure what the problem was in that StackOverflow post. If he
contacted us for consultancy we could have figured out what was going
on.

--
Phusion | Ruby & Rails deployment, scaling and tuning solutions

Web: http://www.phusion.nl/
E-mail: in...@phusion.nl
Chamber of commerce no: 08173483 (The Netherlands)

[Kai Krakow]

Hongli,

When I posted that solution to SO, my understanding of Passenger's architecture was this:

1. Passenger preloads the Rails stack of an app, this becomes the master
2. Master will be preinitialized, which means DB connections will open (thus opening FDs)
3. Master may prelaunch an app instance now by forking, or fork on demand by web request
4. On fork, open FDs be cloned and are shared now with the master instance
5. Due to forking, the pool is shared amongst all instances of the SAME Passenger app

Forking a process will clone its open FDs, thus it will clone open database connection. This is all done in the preloaded app instance which will become the fork master, that's where Passenger takes its speed from. Thus, the pool is shared amongst all forks. This is how things work in Unix.

Given that fact, it will perfectly explain why the solution I posted on SO does indeed work. In fact, argueing that forked processes do not share resources (like open FDs) is plain wrong. They DO share resources, but in another way: Forked process clone FDs (one process can close it without affecting another), threads actually share the SAME resources (one thread can close a FD, it will be closed for other threads, too). In that way, the pool is shared. It's probably that simple. Passenger wouldn't show its good performance if it worked in another way.

Sorry, I do not mean to know the code better than you, actually I don't. But the statement about forks not sharing resources is more or less wrong.

[Hongli Lai]

On Wed, Apr 9, 2014 at 9:58 PM, Kai Krakow <hurik...@gmail.com> wrote:
> Hongli,
>
> When I posted that solution to SO, my understanding of Passenger's
> architecture was this:
>
> Passenger preloads the Rails stack of an app, this becomes the master
> Master will be preinitialized, which means DB connections will open (thus
> opening FDs)
> Master may prelaunch an app instance now by forking, or fork on demand by
> web request
> On fork, open FDs be cloned and are shared now with the master instance
> Due to forking, the pool is shared amongst all instances of the SAME
> Passenger app

1-4 are about right. 5 is not right.

For one, the pool is not shared. Normally, such forking will cause
file descriptors to be shared. But file descriptors are not the pool.
They're just the connection. There is no way to share the pool between
processes (even processes belonging to the same app), because the pool
is an in-memory data structure, not a kernel resource.

Second, Phusion Passenger resets the Active Record connection after
forking, specifically to address this issue in the common case. Only
additional connections that the app made outside of Active Record,
will not be reset by Phusion Passenger.

So you are correct in that some resources are shared. Although that
doesn't mean the shared resources work properly.

--
Phusion | Web Application deployment, scaling, and monitoring solutions

[Kai Krakow]

Hi!

That leaves the question open why it fixed the performance issue.
However, that was Passenger 2 that time back. Passenger 4 is probably
very different. Still, I configure bigger DB pools without ever having
thought about it again. The default is 5, I usually set 10, 20, or 30,
depending on how much load is expected.

The difference to other spawning methods like mongrel clusters is,
that those didn't need bigger pools. And that time, the explanation I
found was about the different spawning methods used by both
implementations. And the difference was that mongrel spawned Rails
instances by spawning independent and fresh app instances while
Passenger spawned them by forking out of a preinitialized master app
instance (which is faster) - and that well explained the need for
bigger pools to me.

> --
> You received this message because you are subscribed to a topic in the Google Groups "Phusion Passenger Discussions" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/phusion-passenger/hfdeaysTqPo/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to phusion-passen...@googlegroups.com.
> To post to this group, send email to phusion-...@googlegroups.com.
> Visit this group at http://groups.google.com/group/phusion-passenger.
> To view this discussion on the web visit https://groups.google.com/d/msgid/phusion-passenger/CAM3ce8FD1c2r3RXX-5hAfDQv254BsF7OM6VOztmD49afcaZv9A%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.