Phusion Passenger + Workling + RabbitMQ

[Panos]

Hi everyone, I am trying to deploy a RoR app that does some
asynchronous task. I use workling for that and the message queue is
RabbitMQ. This combination worked flawlessly with Starling but we
decided to change the MQ for Rabbit. I read in an previous post that I
should include the following code in my environment.rb

require 'mq'
if defined?(PhusionPassenger)
PhusionPassenger.on_event(:starting_worker_process) do |forked|
if forked
if EM.reactor_running?
EM.stop_event_loop
EM.release_machine
EM.instance_variable_set( '@reactor_running', false )
end
Thread.current[:mq] = nil
AMQP.instance_variable_set('@conn', nil)
end
th = Thread.current
Thread.new{
AMQP.connect(:host => 'localhost'){
th.wakeup
}
}
Thread.stop
end
end

But that now Apache fails completely with message: The server
encountered an internal error or misconfiguration and was unable to
complete your request. I guess it should be something that has to do
with the workling plugin

[jimeh]

I'm working with Panos on this issue, and I've narrowed the problem
code down to the Thread part. If I comment it out, Apache doesn't die
with an Internal Server Error. But instead Workling complains that it
couldn't start the amq client.

[Aman Gupta]

Are there any errors in the logs? There must be a ruby backtrace
somewhere that will shed light on the issue.

Aman

[jimeh]

Here's what Apache's Error log contains when it dies with an internal
error:

*** Exception ThreadError in application (stopping only thread
note: use sleep to stop forever) (process 2255):
from ./config/environment.rb:68:in `stop'
from ./config/environment.rb:68
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/events.rb:34:in `call'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/events.rb:34:in `call_event'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/events.rb:33:in `each'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/events.rb:33:in `call_event'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/railz/application_spawner.rb:375:in
`start_request_handler'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/railz/application_spawner.rb:334:in
`handle_spawn_application'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/utils.rb:182:in `safe_fork'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/railz/application_spawner.rb:332:in
`handle_spawn_application'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/abstract_server.rb:351:in `__send__'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/abstract_server.rb:351:in `main_loop'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/abstract_server.rb:195:in `start_synchronously'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/abstract_server.rb:162:in `start'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/railz/application_spawner.rb:213:in `start'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/spawn_manager.rb:261:in `spawn_rails_application'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/abstract_server_collection.rb:126:in `lookup_or_add'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/spawn_manager.rb:255:in `spawn_rails_application'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/abstract_server_collection.rb:80:in `synchronize'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/abstract_server_collection.rb:79:in `synchronize'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/spawn_manager.rb:254:in `spawn_rails_application'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/spawn_manager.rb:153:in `spawn_application'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/spawn_manager.rb:286:in `handle_spawn_application'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/abstract_server.rb:351:in `__send__'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/abstract_server.rb:351:in `main_loop'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/lib/
phusion_passenger/abstract_server.rb:195:in `start_synchronously'
from /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.4/bin/passenger-
spawn-server:61
[Mon Jul 06 17:50:09 2009] [error] [client ::1] Premature end of
script headers:
[ pid=2206 file=ext/apache2/Hooks.cpp:645 time=2009-07-06
17:50:09.343 ]:
The backend application (process 2255) did not send a valid HTTP
response; instead, it sent nothing at all. It is possible that it has
crashed; please check whether there are crashing bugs in this
application.

[jimeh]

Ok, so it's spitting out slightly different error messages at random,
the 2nd of the three I've seen is below. The third simple lists the
trace through files without any info at the bottom...

*** Exception ThreadError in application (stopping only thread

note: use sleep to stop forever) (process 2249):

[ pid=2208 file=ext/apache2/Hooks.cpp:688 time=2009-07-06
17:50:05.975 ]:
Unexpected error in mod_passenger: Cannot set read timeout for
socket: Invalid argument (22)
Backtrace:
in 'int Hooks::handleRequest(request_rec*)' (Hooks.cpp:529)

On Jul 6, 5:33 pm, Aman Gupta <themastermi...@gmail.com> wrote:

[Aman Gupta]

Any luck tracking this down? Looks like it doesn't like the
Thread.stop and th.wakeup. You could remove those and simply do:

require 'mq'
if defined?(PhusionPassenger)
PhusionPassenger.on_event(:starting_worker_process) do |forked|
if forked
if EM.reactor_running?
EM.stop_event_loop
EM.release_machine
EM.instance_variable_set( '@reactor_running', false )
end
Thread.current[:mq] = nil
AMQP.instance_variable_set('@conn', nil)
end
th = Thread.current
Thread.new{
AMQP.connect(:host => 'localhost')
}

end
end

[Daniel DeLeo]

This issue popping up again inspired me to create a small monkey patch
to AMQP so you can put one line in environment.rb and have AMQP work
with thin, mongrel, and passenger. Anyone who'd like to use, break,
improve or criticize it can take a look here:

http://github.com/danielsdeleo/qusion/tree/master

Cheers,
Daniel DeLeo

[Panayiotis Papadopoulos]

Awesome!

So far so good, we also had to play a bit so that workling plays along. So far it seems to work though we may have to modify workling a bit so it uses the pool. Thank you!

--
Panagiotis Papadopoulos
Web Architect
http://6pna.com - http://twitter.com/panosjee

[Daniel DeLeo]

Cool, glad you like it!
As far as using workling, I'm not personally very familiar, but
looking at AmqpClient#connect in the source it seems to get a single
AMQP channel and reuse it. This should work fine for Passenger.

Channel pools are created on first use so if you don't want to use the
channel pool, just avoid calling Qusion.channel and you're good.

Be sure to let me know if you run into any problems.

Dan DeLeo

On Mon, Jul 13, 2009 at 9:06 PM, Panayiotis Papadopoulos
<pano...@gmail.com> wrote:

Awesome!So far so good, we also had to play a bit so that workling

[simianarmy]

Could you give me a pointer on how to use Workling properly with
Qusion/Passenger + rake or script/console?

It seems that my rabbitMQ queue gets loaded with messages when I call
a workling async_ method,
but workling never "sees" them, though it is running.

The workling job is created from a rake task that load the Rails
environment.

I'm sure this has something to do with the em thread...I just don't
know how to fix it.

On Jul 14, 7:51 am, Daniel DeLeo <d...@kallistec.com> wrote:
> Cool, glad you like it!

> As far as usingworkling, I'm not personally very familiar, but

> looking at AmqpClient#connect in the source it seems to get a single
> AMQP channel and reuse it. This should work fine for Passenger.
>
> Channel pools are created on first use so if you don't want to use the
> channel pool, just avoid calling Qusion.channel and you're good.
>
> Be sure to let me know if you run into any problems.
>
> Dan DeLeo
>
> On Mon, Jul 13, 2009 at 9:06 PM, Panayiotis Papadopoulos
>

> <panos...@gmail.com> wrote:
>
> Awesome!So far so good, we also had to play a bit so thatworkling

> plays along. So far it seems to work though we may have to modifyworklinga bit so it uses the pool. Thank you!

> > > > > with an Internal Server Error. But insteadWorklingcomplains that it

> > > > > couldn't start the amq client.
>
> > > > > On Jul 6, 3:06 pm, Panos <panos...@gmail.com> wrote:
> > > > >> Hi everyone, I am trying to deploy a RoR app that does some

> > > > >> asynchronous task. I useworklingfor that and the message queue is

[Panayiotis Papadopoulos]

In the environment.rb inside the config write

  config.after_initialize do

    require "#{Rails.root}/vendor/gems/qusion/lib/qusion"

    AMQP.start_web_dispatcher(:host => "localhost")

    Workling::Remote.invoker           = Workling::Remote::Invokers::EventmachineSubscriber

    Workling::Remote.dispatcher        = Workling::Remote::Runners::ClientRunner.new

    Workling::Remote.dispatcher.client = Workling::Clients::AmqpClient.new

  end

That s how it works for me

[Daniel DeLeo]

Did this help?

If you're still having issues, I'll try to reproduce and diagnose the issue if you can provide me some sample code to work with.

Dan DeLeo

[simianarmy]

Thanks for that tip, I used the config.after_initialize block, though
all that code was already in my environment.rb, though outside the
init block (at the end).

Rails loads fine (console, mongrel), but workling still fails after
loading Rails. I can see normal rails startup messages in my log file
after running script/workling_client start, but after loading it fails
with the following messages in workling.output:

** Rails loaded.
** Starting Workling::Remote::Invokers::EventmachineSubscriber...
** Use CTRL-C to stop.
** Exiting

Not really helpful debug message unfortunately. I'm not sure if those
log messages are from daemons gem, but will look.

Additional note about workers - the existing workers worked fine using
traditional Starling/memcached.

On Aug 3, 3:22 pm, Daniel DeLeo <d...@kallistec.com> wrote:
> Did this help?
>
> If you're still having issues, I'll try to reproduce and diagnose the issue
> if you can provide me some sample code to work with.
>
> Dan DeLeo
>
> On Mon, Aug 3, 2009 at 2:37 PM, Panayiotis Papadopoulos

> <panos...@gmail.com>wrote:

>
> > In the environment.rb inside the config write
>
> >>   config.after_initialize do
>
> >     require "#{Rails.root}/vendor/gems/qusion/lib/qusion"
>
> >     AMQP.start_web_dispatcher(:host => "localhost")
>
> >     Workling::Remote.invoker           =
> >> Workling::Remote::Invokers::EventmachineSubscriber
>
> >     Workling::Remote.dispatcher        =
> >> Workling::Remote::Runners::ClientRunner.new
>
> >     Workling::Remote.dispatcher.client = Workling::Clients::AmqpClient.new
>
> >   end
>
> > That s how it works for me
>

> ...
>
> read more »

[Daniel DeLeo]

I tracked it down. Basically what's happening is that Qusion tests for Passenger, then Thin, and if it doesn't find those it defaults to running EventMachine in a thread, which works with mongrel, but doesn't block Workling, so it just exits.

It should be fixed in github head, can you give it a try?

Also, if there's any webrick diehards, please drop me a line. This fix breaks webrick support (assuming it worked in the first place :-)

Thanks,

Dan DeLeo

[simianarmy]

After sending the last email, I watched workling_client output one
more time after finding its log messages in the listen.rb file - I saw
a new error message (perhaps due to the way I setup amqp start in
config.after_initialize?

Here is the new error message (after ** Exiting line):
(Additional note - I got the same error using Mongrel & Passenger)

workling.output:
=> Loading Rails...
...rails stuff

** Rails loaded.
** Starting Workling::Remote::Invokers::EventmachineSubscriber...
** Use CTRL-C to stop.
** Exiting

/path/to/app/vendor/plugins/workling/lib/workling/clients/
amqp_client.rb:17:in `connect': couldn't start amq client. if you're
running this in a server environment, then make sure the server is
evented (ie use thin or evented mongrel, not normal mongrel.)
(Workling::WorklingError)
from /vendor/plugins/workling/lib/workling/remote/invokers/
base.rb:66:in `connect'
from /vendor/plugins/workling/script/../lib/workling/remote/
invokers/eventmachine_subscriber.rb:21:in `listen'
from /vendor/gems/eventmachine-0.12.8/lib/eventmachine.rb:
228:in `call'
from /vendor/gems/eventmachine-0.12.8/lib/eventmachine.rb:
228:in `run'
from /vendor/plugins/workling/script/../lib/workling/remote/
invokers/eventmachine_subscriber.rb:20:in `listen'
from /vendor/plugins/workling/script/listen.rb:24
from /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/
daemons/application.rb:176:in `load'
from /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/
daemons/application.rb:176:in `start_load'
from /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/
daemons/application.rb:257:in `start'
from /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/
daemons/controller.rb:69:in `run'
from /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/
daemons.rb:139:in `run'
from /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/
daemons/cmdline.rb:105:in `call'
from /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/
daemons/cmdline.rb:105:in `catch_exceptions'
from /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.10/lib/
daemons.rb:138:in `run'
from script/workling_client:19

On Aug 3, 3:22 pm, Daniel DeLeo <d...@kallistec.com> wrote:

> Did this help?
>
> If you're still having issues, I'll try to reproduce and diagnose the issue
> if you can provide me some sample code to work with.
>
> Dan DeLeo
>
> On Mon, Aug 3, 2009 at 2:37 PM, Panayiotis Papadopoulos

> <panos...@gmail.com>wrote:

>
> > In the environment.rb inside the config write
>
> >>   config.after_initialize do
>
> >     require "#{Rails.root}/vendor/gems/qusion/lib/qusion"
>
> >     AMQP.start_web_dispatcher(:host => "localhost")
>
> >     Workling::Remote.invoker           =
> >> Workling::Remote::Invokers::EventmachineSubscriber
>
> >     Workling::Remote.dispatcher        =
> >> Workling::Remote::Runners::ClientRunner.new
>
> >     Workling::Remote.dispatcher.client = Workling::Clients::AmqpClient.new
>
> >   end
>
> > That s how it works for me
>

> ...
>
> read more »

[Daniel DeLeo]

Do you still have this problem with the latest HEAD?

Dan

[simianarmy]

That fixed it for me, I put some debug statements in the subscribe
method and it ran once for each worker/queue.

Thanks again! I will let you know if the new workling workers pick up
the old jobs now.

On Aug 3, 6:35 pm, Daniel DeLeo <d...@kallistec.com> wrote:
> Do you still have this problem with the latest HEAD?
> Dan
>

> ...
>
> read more »

[simianarmy]

Ok one more question and I promise I'll leave you guys alone!

Now that I am certain workling_client is starting up ok and
subscribing to rabbitmq queues,
I am testing the interaction by using a rake task that publishes jobs
using the Workling api.

I am watching the queues using rabbitmqctl list_queues and the rake
task is publishing jobs to the test queue...
but the worker subscribe block does not pick up the additions, even
after the rake task sends & amqp is closed.

I tried with a running workling_client, and starting workling
manually, but neither pick up any of the existing jobs or the new
ones.

Both the publisher (rake task) & subscriber (workling) startup
properly with Qusion in environment.rb
=> Qusion did not find a supported app server (This is normal for
workling workers)

So since I know both are using workling's amqp_client MQ.new
connection, why does the subscriber not see those jobs? amqp config
is the same...

Stumped, any suggestions would be awesome.

On Aug 3, 6:35 pm, Daniel DeLeo <d...@kallistec.com> wrote:
> Do you still have this problem with the latest HEAD?
> Dan
>

> ...
>
> read more »

[Panayiotis Papadopoulos]

Maybe you could have some feedback on the Workling group :-/

[Daniel DeLeo]

I can't seem to reproduce the problem. I can say that in script/console, I had to take matters into my own hands in order to get a usable connection to RabbitMQ:

>> t = Thread.new do 

?>     EM.run { AMQP.start }

>>   end

then I was able to run Workling async methods.  By the way, when you get that message "=> Qusion did not find a supported app server (This is normal for workling workers)" that means that Qusion didn't do any AMQP setup at all. I should probably make it load the AMQP settings (if you're using that feature), but for now the behavior is to do nothing. It doesn't seem like this is the problem though, because AMQP will raise an error if it can't connect to RabbitMQ (this gets caught by Workling, which re-raises with that "couldn't start amq client. etc..." message) and you're not seeing this error.

Could you try without Qusion? It shouldn't affect your rake tasks or workling workers as it is right now.

Thanks,

Dan DeLeo

[Aman Gupta]

One thing to keep in mind: when you start EM in a thread in irb/console, it doesn't actually run in the background. irb uses Readline, which blocks the ruby interpreter completely when its at the prompt, so nothing else can happen.

For example:

>> $em = Thread.new{ EM.run{ EM.add_periodic_timer(0.1){ puts 'hi' } } }

>>

nothing that nothing will happen until you enter some input and actually return control to the ruby vm:

>> 1

hi=> 1

>> 1

=> 1

>> 1

hi=> 1

  Aman

[simianarmy]

It looks like what is happening is that since Qusion does nothing with
amqp (when loading my rake/console), workling connects using basic
em.run / MQ.new for async publishes.
Since I was having problems with connecting to amqp before, I had
wrapped my rake 'publisher' task in a custom amqp start block with
amqp.yml settings (non-default).
The Workling client, on the other hand, connected to rabbitmq using
default settings, so while I saw the queue size grow on the custom
rabbitmq vhost, the client was not seeing anything on the default
guest vhost.

I updated my Qusion amqp.rb file to load the amqp settings file, as
you suggested in this thread:

amqp.rb:

def self.start_web_dispatcher
...
else
@settings = Qusion::AmqpConfig.new(*args).config_opts
puts "=> Qusion did not find a supported app server (This is
normal for workling workers)"
end

Perhaps this belongs in the workling part of environment.rb...but
shouldn't hurt even if not used.

On Aug 3, 8:49 pm, Daniel DeLeo <d...@kallistec.com> wrote:
> I can't seem to reproduce the problem. I can say that in script/console, I
> had to take matters into my own hands in order to get a usable connection to
> RabbitMQ:>> t = Thread.new do
>
> ?>     EM.run { AMQP.start }
>
> >>   end
>
> then I was able to run Workling
> async methods.  By the way, when you get that message "=> Qusion did not
> find a supported app server (This is normal for workling workers)" that
> means that Qusion didn't do any AMQP setup at all. I should probably make it
> load the AMQP settings (if you're using that feature), but for now the
> behavior is to do nothing. It doesn't seem like this is the problem though,
> because AMQP will raise an error if it can't connect to RabbitMQ (this gets
> caught by Workling, which re-raises with that "couldn't start amq client.
> etc..." message) and you're not seeing this error.
>
> Could you try without Qusion? It shouldn't affect your rake tasks or
> workling workers as it is right now.
>
> Thanks,
> Dan DeLeo
>

> ...
>
> read more »