Running scheduled jobs inside Goliath

[Antonin Kral]

Hello,

we have a (multiple) Goliath server-based apps, which provides different APIs. Some of then need to run a scheduled jobs. The natural thing would be to use external scheduler / job queue and execute them. E.g. something like using rails runner from cron. But I cannot find the nice way for doing so. My main motivation is, that apps are pretty complex, there are ActiveRecord models and some of them are expecting that they have AMQP exchange available etc.

I can certainly do something like:

def worker_wrapper
  puts "Work started"
  # do some work here
  EM.add_timer(10) { worker_wrapper }
end

EM.add_timer(10) { worker_wrapper }

But it doesn't look correct (not only, that I should probably run the worker in different thread not to block reactor, rigt?). Another approach would be to extract some logic from Goliath::Runner - setup_server and then do something similar to the Goliath::Console.run!().

What would be your best practice?

Thank you,

    Antonin

[Ilya Grigorik]

Hi Antonin. First off, take a look at these threads:

https://groups.google.com/forum/#!topic/goliath-io/zdv6mHXUCTw
https://groups.google.com/forum/#!topic/goliath-io/PmzkOCN6GK8

Long story short, while you *can* perform the work within the Goliath
process, I would recommend against it. Async tasks should be pushed into
their workflow.. otherwise you'll end up blocking processing on your API
server (unless you're very, very careful). Refactoring some common methods
for setup of exchanges, AR connections, could go a long way..

Having said that, if you must, you can use EM.defer, which will schedule
the block on a different thread (just be careful about all the usual issues
with threading). Further, on MRI, pushing onto a different thread won't get
you that far and you may still block the reactor.

ig

--
You received this message because you are subscribed to the Google Groups "Goliath.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to goliath-io+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Antonin Kral]

Hi Ilya,

thank you for comming back to me,

Dne pondělí, 8. července 2013 0:42:59 UTC+2 Ilya Grigorik napsal(a):

https://groups.google.com/forum/#!topic/goliath-io/zdv6mHXUCTw
https://groups.google.com/forum/#!topic/goliath-io/PmzkOCN6GK8

I've seen them. I am actually using this pattern right now. For example, one of the reasons why there is AMQP dependency is to offload things like sending email to the external process / worker.Typical example would be:

• user changes setting through UI
• UI is actually independent web application talking to my Goliath service through some API
• when the API endpoint is hit
• data are updated in PostgreSQL through ActiveRecord model
• there is additional logic in the model, e.g. it emmits a message to AMQP queue to send user an email with confirmation of change
• logic can be sometimes complex and involves multiple other models

I need to write a worker which will invoke certain methods regurarly. For example it will be looking for expired user's subscriptions and send the users an email. I already have majority of logic in existing AR model, I can send emails through AMQP messages. I can make a service API endpoint and then call it with curl from cron and execute job within the main process or use the EM.add_timer. But that it a bit smelly. Therefore I would like to use other process to run these workers but reuse the AR and other infrastructure I already have.

So really something like rails runner to be able to execute script within the context of the environment. That was the reason I was talking about reusing Goliath::Console.run!()

Thank you,

  Antonin

[Ilya Grigorik]

On Sun, Jul 7, 2013 at 10:09 PM, Antonin Kral <antoni...@gmail.com> wrote:

I need to write a worker which will invoke certain methods regurarly. For example it will be looking for expired user's subscriptions and send the users an email. I already have majority of logic in existing AR model, I can send emails through AMQP messages. I can make a service API endpoint and then call it with curl from cron and execute job within the main process or use the EM.add_timer. But that it a bit smelly. Therefore I would like to use other process to run these workers but reuse the AR and other infrastructure I already have.

I wouldn't necessarily shy away from this architecture.. It seems to do what you need, and it sounds sane. Nothing wrong with running a cron job with a curl ping. In effect, what you've just described is the same architecture as "scheduled tasks" exposes by GAE: https://developers.google.com/appengine/docs/python/config/cron

So really something like rails runner to be able to execute script within the context of the environment. That was the reason I was talking about reusing Goliath::Console.run!()

Right, I see. Well, we don't have anything like that in Goliath, but you can take a look at the initialization logic (it's pretty simple) and piece it together. 

ig

[Antonin Kral]

Dne pondělí, 8. července 2013 7:27:12 UTC+2 Ilya Grigorik napsal(a):

So really something like rails runner to be able to execute script within the context of the environment. That was the reason I was talking about reusing Goliath::Console.run!()

Right, I see. Well, we don't have anything like that in Goliath, but you can take a look at the initialization logic (it's pretty simple) and piece it together.

I have to do a little bit more research on this one. My initial implementation failed miserably as I've messed up Fibers for postgres driver, so the transactions we in dead fibers. My current implementation is quite crude - content of response is different when you execute server as worker, and cron runs the jobs through requesting job specific route (which is loaded only if the server is in worker mode).

Cheers,

   Antonin