Restarting extensions?

[John Axel Eriksson]

At first I thought I could base my Qu-scheduler extension on the resque scheduler extension, unfortunately I now don't know how to restart the extension on a redeploy/update. Quite often the jobs we run use code that was updated on a new deploy and therefore the workers must restart and reload. Perhaps an extension isn't the right way to do this?

For now I've just put my workers in an initializer (eg. running inside the rails-context) that starts them in their own threads. This way I could make sure they restart together with the application.

I've also had serious problems with the Quartz-scheduler (a trinidad extension) - on restart it crashes and doesn't seem to be able to reload the classes so I guess the same idea would apply to "cron-like" scheduling - simply put something in an initializer so it runs within Rails and gets restarted together with Rails.

These observations make me wonder however - what are extensions really good for? Am I missing something?

Best,

John

[kares]

 Hi John, right it's not that they aren't good for anything but one needs to understand his needs (and implement extension code correctly) ...

Unfortunately most worker extensions are a bit "poorly" written - they work but as you mention might not reload / restart correctly etc.
I personally am a bit in doubt of doing Threads directly within the Rails app, that's why I did https://github.com/kares/jruby-rack-worker which externalizes the Runtime and Thread code ...
It fits better e.g. when you have non thread-safe runtimes - workers will "work" in their own jruby runtimes and run in native Java daemon threads (that restart with the application).

I've just recently released a trinidad extension for this https://github.com/trinidad/trinidad_worker_extension
You can provide a piece of custom worker code/script that is going to start, thus might suit your needs (should be without the issues you mention if you write the worker code "correctly" - see how DJ, Resque are setup with an at_exit hook).

It's a first version (the extension), but JRuby-Rack-Worker already had quite some users for a while and seems stable.
But anyway, you might need to be a bit patient with it - feel free to report issues if you run into any ...

Cheers, K.

[John Axel Eriksson]

Ok. I still don't quite understand what's going on. The scenario I'm having trouble with is:

start app (which starts workers) using bundle exec trinidad --threadsafe --config config/trinidad.yml

make some changes

touch tmp/restart.txt

problem: workers either don't restart or crash, same with quartz-scheduler (which seems to mostly crash)

I tried using jruby-rack-worker and trinidad_worker_extension instead, unfortunately that doesn't seem to work either, on first start the workers boot up,

on touch tmp/restart.txt:

[org.kares.jruby.rack.DefaultWorkerManager] no worker script to execute - configure one using 'jruby.worker.script' or 'jruby.worker.script.path' parameter (or see previous errors if already configured)  !

So basically it now claims there is no path to the startup script for the workers... I didn't change the config or anything.

Also a weird thing about trinidad_worker_extension was that bundler complained about jruby-rack-worker not "in any sources". I had to "gem install jruby-rack-worker" to get it to run, didn't matter if I put

jruby-rack-worker in the gem file or no.

[kares]

 Hey again John, I'm sorry - talking about "incorrectly" written extensions I've come up with one as well :)

I shall release a new version of trinidad_worker_extension (0.2.0) today with a fix.
In the mean time setting up jruby-rack-worker.jar the  "Java" way should still work as expected ... if you're not into waiting for it.
As for the Quartz scheduler, try reporting the issue, I'm not the original author and since my time is getting pretty limited hopefully someone else could fix it.

Sorry again for the confusion - it should all work as you (originally) expected ... K.

On Saturday, October 27, 2012 12:51:33 PM UTC+2, John Axel Eriksson wrote:

[kares]

 Also the `gem install jruby-rack-worker` seems like a bundler issue - I'll try reporting if they can fix, it if not I shall change the jruby-rack-worker to not target jruby only ...

K.

[John Axel Eriksson]

Thanks! I'm not overly concerned with Quartz at the moment - it was just something I was trying out. The workers however are an important part of our infrastructure. Looking forward to new release :-). I might report the Quartz-issue but it seems as if it might be an abandoned project.

[Adrian Madrid]

Did you give any thought to add Sidekiq [1] support to jruby-rack-worker? I like it better and it would be great to have a trinidad extension instead of a separate process.

[1] https://github.com/mperham/sidekiq

[kares]

 Hi Adrian, this is my second attempt to respond (for whatever reason some of my google groups posts are getting deleted and I can't recover them - and I was not offensive :)).

I've written you an email previously regarding your Sidekiq Trinidad extension (last week) - if there's any help you need from me to make sure it works right ...
After all the real essence of OSS is maintaining (not creating) and I've find quite some obstacles in trying to maintain some of the worker extensions out there.

I definitely considered Sidekiq for JRuby-Rack-Worker there just was no demand yet (I have really zero Sidekiq experience thus to be honest I'm not sure of it's advantages over Resque when running thread-ed) and I'm quite stuck with some other OSS (non-payed - I really need to get a job :)) work around JRuby-Rack and Trinidad.

But I certainly do welcome any efforts in coming up with a Sidekiq::JRubyWorker.

With Regards, K.

[Adrian Madrid]

Sorry about missing your email. I did get something going but I could not get sidekiq to behave well (stop/start/restart when and I how I wanted). In the end I gave up on Trinidad altogether for Puma. I like better the simplicity of their on_restart block and how it is a separate process that shares nothing of the code between processes on restart.  It maybe my lack of Java zen but Puma seems to be working so much easier for me.

[kares]

 Thanks Adrian, it's not that much of Java Zen :) - actually more of Sidekiq internals knowledge required ...

To get it right of course, the extensions are operating on an abstracted level from all deployed applications thus it's not straight forward.

Anyways, the reasons for Puma are convincing me to get into it since it's actually what I'd like to avoid - to have a separate process :)

K.

[John Axel Eriksson]

I implemented support for the QU-scheduler (https://github.com/bkeepers/qu) which was pretty straight forward using the trinidad_worker_extension. We've looked at SideKiq and might possibly switch to that in the future. I've also looked at Puma but I don't think it's a good fit for JRuby. I.m.o using something like Trinidad, Torquebox or Warfile is the way to go in a java environment.

[John Axel Eriksson]

Also, the important thing to keep in mind when implementing support for a worker is to override any signal handling (eg Signal.trap('INT') or Signal.trap('TERM')) - all that should be done inside an at_exit handler on JRuby, otherwise it will prevent Trinidad from restarting correctly.

[Adrian Madrid]

Sharing code between the extension and trinidad is the part I'm very confused about. I could not get that going right. Sometimes it would work and sometimes it wouldn't. For workers it seems you would like it to be sharing all the code since even details like configuration or even the connection to Redis (my use case) should be shared. Maybe we could use more details on how the trinidad extension system works? Is it even possible (to share configuration and model code/data)?