Feature request: preforked workers
Wouter de Bie
Since my last post, I've been playing around with passenger and rolling
restarts. I've tried killing passenger workers and let the spawn server
take care of spawning new ones, but it seems that it doesn't really fit
our use-case. Since our application serves a lot of very small requests
(< 30 ms), a restart of the application adds to much time to the first
request.
We're currently running thins behind an nginx, but we're having issues
with not being able to use a global queue. A scenario we are currently
implementing is to detach our server from the load balancer, then
restart passenger with the restart.txt, hammer it with requests, so that
the amount of workers equals the MaxPoolSize and then attach it to the
load-balancer again. I don't think this is a very elegant solution, so
I'm opting for a different mechanism. What would be nice is to be able
to specify if the amount of workers equal to the maxpoolsize should be
created at startup or not. Next to that, it would be nice to be able to
do a rolling restart and stop a worker and start it again, without the
need for an incoming request to spawn it. For us it's pretty nessecary
to have workers ready that loaded new code, before they start serving.
Does anyone think this is possible?
Greetings,
Wouter
John Dell
http://github.com/defunkt/unicorn
John
--
You received this message because you are subscribed to the Google Groups "Phusion Passenger Discussions" group.
To post to this group, send email to phusion-...@googlegroups.com.
To unsubscribe from this group, send email to phusion-passen...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/phusion-passenger?hl=en.
Hongli Lai
--
Phusion | The Computer Science Company
Web: http://www.phusion.nl/
E-mail: in...@phusion.nl
Chamber of commerce no: 08173483 (The Netherlands)
JayPainter
(prespawn) for mod_passenger.
The patch is attached to the bug:
http://code.google.com/p/phusion-passenger/issues/detail?id=407
Right now I've hard-coded the patch to pre-spawn 2*MaxProcesses for
each domain.
This patch also kills the mod_passenger application restart feature.
Regards,
Jay
On Dec 14, 10:55 am, Hongli Lai <hon...@phusion.nl> wrote:
> E-mail: i...@phusion.nl
Jeremy Wohl
> I just finished with an alpha-quality patch that implements prefork
> (prespawn) for mod_passenger.
Nice work, Jay. +1 for hiding proc startup latency with some kind of
min-spare-procs feature.
Jay Painter
be started concurrently.
Do you know if the SpawnManager is thread-safe?
Hongli Lai
> I would also like to launch multiple threads so that the processes can
> be started concurrently.
> Do you know if the SpawnManager is thread-safe?
Yes it is.
--
Phusion | The Computer Science Company
Web: http://www.phusion.nl/
E-mail: in...@phusion.nl
Hongli Lai
> On Thu, Dec 31, 2009 at 7:46 PM, Jay Painter <jay.p...@gmail.com> wrote:
>> I would also like to launch multiple threads so that the processes can
>> be started concurrently.
>> Do you know if the SpawnManager is thread-safe?
>
> Yes it is.
However it does not support concurrency at this time. Even if you call
it from multiple threads it will only spawn one at a time. This is
something we want to address in the future.
JayPainter
create
more than one SpawnManager at a time in the StandardApplicationPool
class.
I think I can start adding start-up concurrency by creating at least
one SpawnManager
per Domain in the code, possibly just add a new setting like:
DomainSpawnConcurrancy 3
And that will launch 3 SpawnManagers per domain.
Does that sound like it would work?
On Dec 31 2009, 2:27 pm, Hongli Lai <hon...@phusion.nl> wrote:
> 2009/12/31 hongli <hon...@phusion.nl>:
>
> > On Thu, Dec 31, 2009 at 7:46 PM, Jay Painter <jay.pain...@gmail.com> wrote:
> >> I would also like to launch multiple threads so that the processes can
> >> be started concurrently.
> >> Do you know if the SpawnManager is thread-safe?
>
> > Yes it is.
>
> However it does not support concurrency at this time. Even if you call
> it from multiple threads it will only spawn one at a time. This is
> something we want to address in the future.
>
> --
> Phusion | The Computer Science Company
>
> Web:http://www.phusion.nl/
> E-mail: i...@phusion.nl
Hongli Lai
> After looking over the SpawnManager code, it appears that I can
> create
> more than one SpawnManager at a time in the StandardApplicationPool
> class.
> I think I can start adding start-up concurrency by creating at least
> one SpawnManager
> per Domain in the code, possibly just add a new setting like:
>
> DomainSpawnConcurrancy 3
>
> And that will launch 3 SpawnManagers per domain.
>
> Does that sound like it would work?
Yes, it would work. However this will start 3 different spawn servers.
A Ruby process spawned by one spawn server cannot share memory with a
Ruby process spawned by another spawn server.
--
Phusion | The Computer Science Company
Web: http://www.phusion.nl/
E-mail: in...@phusion.nl
JayPainter
> > Does that sound like it would work?
>
> Yes, it would work. However this will start 3 different spawn servers.
> A Ruby process spawned by one spawn server cannot share memory with a
> Ruby process spawned by another spawn server.
So it won't really work for Rails.
So, I need to modify the spawn server to support concurrency.
How about this: I modify the spawn server so that only the initial
"fork" from the spawn server is
under the SpawnManager lock. Then the spawn server process sends a
signal back to the SpawnManager indicating
it has completed fork()ing. The SpawnManager then can release the
main lock, but keeps blocking while the newly spawned
process starts up. That way calls to the SpawnManager will only be
serialized during the initial fork(). If the spawn process needs
to talk to the SpawnManager, it can do it from the fork()ed process
instead of the spawn process (where it would be serialized).
Does this sound like a better approach?
Hongli Lai
> So it won't really work for Rails.
> So, I need to modify the spawn server to support concurrency.
>
> How about this: I modify the spawn server so that only the initial
> "fork" from the spawn server is
> under the SpawnManager lock. Then the spawn server process sends a
> signal back to the SpawnManager indicating
> it has completed fork()ing. The SpawnManager then can release the
> main lock, but keeps blocking while the newly spawned
> process starts up. That way calls to the SpawnManager will only be
> serialized during the initial fork(). If the spawn process needs
> to talk to the SpawnManager, it can do it from the fork()ed process
> instead of the spawn process (where it would be serialized).
>
> Does this sound like a better approach?
Yes. I had something similar in mind for future refactoring. The way
it works now (oversimplified description) is that spawn_manager.rb
calls Railz::ApplicationSpawner#spawn, which returns an AppProcess
object, and sends the information in the AppProcess back to the C++
SpawnManager. This can take a while: #spawn waits for the app process
to create an owner pipe, which happens after Rails is loaded.
Actually, now that I think about it, it should be possible to solve
this problem in a simpler manner. At this time all communication with
the spawn server happens over a single IO channel, which is also used
to check whether the parent process has exited. If you can modify the
spawn server to listen on a Unix socket instead, with a separate owner
pipe for checking parent process liveliness, then you can make the
spawn server concurrent by handling each client in a separate thread.
Each client thread would use a different communication channel so the
messages won't interleave with each other.
JayPainter
I had an idea this morning that might actually make launching multiple
spawn server processes work. The SpawnManager could be modified to to
create a static (class-wide) spawn manager process that is not used
for any actual spawning. After the "master" spawn process is created,
new instances of SpawnManager get spawning processes that are forked
from the master spawn process instead of being forked by the
SpawnManager. That would allow the use of multiple concurrent
SpawnManager instances using spawn-manager processes that were all
forked from the same process. Then shared memory for Rails/Ruby
Enterprise Edition would still work.
Does that sound easier?
-Jay
Hongli Lai
[C++ SpawnManager]
|
|
[spawn server process]
|
+--------------------------------+
| |
[Railz::ApplicationSpawner process] |
| |
| [Another Railz::ApplicationSpawner
| process, for a different app]
|
|
+---+-----------------+
| |
[App process 1] [App process 2]
The ApplicationSpawner is forked from the spawn server process, and there's at most 1 ApplicationSpawner per app. It's the ApplicationSpawner that contains the Rails code and the application code, not the spawn server process. In your setup you'd have something like this:
[C++ SpawnManager]
|
|
[main spawn server process]
|
+---------------------------------+
| |
[sub spawn server 1] [sub spawn server 2]
| |
| |
| |
[Railz::ApplicationSpawner process] |
| |
| [Another Railz::ApplicationSpawner
| process, for the other sub spawn server]
| |
| |
+---+-----------------+ |
| | |
[App process 1] [App process 2] [App process 3]
Although sub spawn servers 1 and 2 are forked from the main spawn server, neither of them contain the Rails or application code. It's the ApplicationSpawners that make memory sharing possible. In this example app processes 1 and 2 share memory, but 3 does not.