[2.0] Scheduling asynchronous tasks in multi-server environment

[Oleg Golitsin]

Hello.

What is the best practice to use Play 2.0 akka scheduling asynchronous
tasks in cluster environment?
I mean when there are several servers over load balancer.
How to avoid starting the same task simultaneously on several servers?

[Guillaume Bort]

Probably using Akka remote.

> --
> You received this message because you are subscribed to the Google Groups "play-framework" group.
> To post to this group, send email to play-fr...@googlegroups.com.
> To unsubscribe from this group, send email to play-framewor...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/play-framework?hl=en.
>

--
Guillaume Bort

[Oleg Golitsin]

Thanks

So every server instance should use Remote Actors and some kind of
routing (Router Actor) should be used to route incoming messages?

On Feb 15, 1:45 pm, Guillaume Bort <guillaume.b...@gmail.com> wrote:
> Probably using Akka remote.
>

[Guillaume Bort]

I'm not sure to understand your use case. Can you explain what you need exactly?

[Oleg Golitsin]

Sorry for misunderstanding, I've just started learn Akka so I probable
miss some points.

The use case seams to be simple.

There is Play 2.0 app with scheduled jobs. As I understand scheduled
jobs in Play 2.0 is scheduled akka asynchronous tasks.
This app is deployed to several servers behind load balancer.
I need the same job to be started only on one of the servers.
But in case of some server failure (or downtime) the job needs to be
started on the other one.

I just can not find any examples...

[akhil]

Hi

I too would like to know possible solutions


On Feb 15, 6:29 pm, Oleg Golitsin <oleg.golit...@gmail.com> wrote:
> Sorry for misunderstanding, I've just started learnAkkaso I probable

> miss some points.
>
> The use case seams to be simple.
>

> There is Play2.0app with scheduled jobs. As I understand scheduled
> jobs in Play2.0is scheduledakkaasynchronous tasks.

> This app is deployed to several servers behind load balancer.
> I need the same job to be started only on one of the servers.
> But in case of some server failure (or downtime) the job needs to be
> started on the other one.
>
> I just can not find any examples...
>
> On Feb 15, 2:23 pm, Guillaume Bort <guillaume.b...@gmail.com> wrote:
>
>
>
>
>
>
>
> > I'm not sure to understand your use case. Can you explain what you need exactly?
>
> > On Wed, Feb 15, 2012 at 1:02 PM, Oleg Golitsin <oleg.golit...@gmail.com> wrote:
> > > Thanks
>
> > > So every server instance should use Remote Actors and some kind of
> > > routing (Router Actor) should be used to route incoming messages?
>
> > > On Feb 15, 1:45 pm, Guillaume Bort <guillaume.b...@gmail.com> wrote:
> > >> Probably usingAkkaremote.
>
> > >> On Wed, Feb 15, 2012 at 12:27 PM, Oleg Golitsin <oleg.golit...@gmail.com> wrote:
> > >> > Hello.
>

> > >> > What is the best practice to use Play2.0akkascheduling asynchronous

[peter hausel]

+1 for using akka remoting:

http://doc.akka.io/docs/akka/2.0/scala/remoting.html

http://doc.akka.io/docs/akka/2.0/java/remoting.html

[Oleg Golitsin]

It is not possible to use akka remoting with Heroku deployment. Am I right? 

There is no way to find out IP address of remote workers?

[James Ward]

You can communicate between nodes with a messaging system like RabbitMQ.  Here is a great article that shows how to do this:

https://devcenter.heroku.com/articles/scaling-out-with-scala-and-akka

-James

[Oleg Golitsin]

Yes, thanks, I know it. But RabbitMQ is in private beta now.

[James Ward]

You can apply to the Heroku beta program at:
http://beta.heroku.com/signup

There is also:
https://addons.heroku.com/iron_mq

-James

On 05/31/12 11:03, Oleg Golitsin wrote:
> Yes, thanks, I know it. But RabbitMQ is in private beta now.
>
> On Thursday, May 31, 2012 4:47:11 PM UTC+3, James Ward wrote:
>
> You can communicate between nodes with a messaging system like
> RabbitMQ. Here is a great article that shows how to do this:

> https://devcenter.heroku.com/articles/scaling-out-with-scala-and-akka <https://devcenter.heroku.com/articles/scaling-out-with-scala-and-akka>

>
> -James
>
>
> On Wednesday, May 30, 2012 9:42:40 AM UTC-6, Oleg Golitsin wrote:
>
> It is not possible to use akka remoting with Heroku deployment.
> Am I right?
> There is no way to find out IP address of remote workers?
>
> On Thursday, March 22, 2012 3:40:12 PM UTC+2, peter hausel wrote:
>
> +1 for using akka remoting:
> http://doc.akka.io/docs/akka/2.0/scala/remoting.html

> <http://doc.akka.io/docs/akka/2.0/scala/remoting.html>
> http://doc.akka.io/docs/akka/2.0/java/remoting.html

> <http://doc.akka.io/docs/akka/2.0/java/remoting.html>
>
> On Wednesday, February 15, 2012 6:27:26 AM UTC-5, Oleg
> Golitsin wrote:
>
> Hello.
>
> What is the best practice to use Play 2.0 akka
> scheduling asynchronous
> tasks in cluster environment?
> I mean when there are several servers over load balancer.
> How to avoid starting the same task simultaneously on
> several servers?
>

> --
> You received this message because you are subscribed to the Google
> Groups "play-framework" group.

> To view this discussion on the web visit
> https://groups.google.com/d/msg/play-framework/-/RpWLqcfcoEYJ.

[Steven Luu]

On another note, MongoDB has a feature called find and update. Basically if you have a queue, you can look for an item in the queue that has not been grabbed, and then update the flag in one atomic operation. Mongo puts a lock on that queue when you're doing a "find-and-update" operation so you can know for sure that only 1 of your workers can tap into the queue at any instance. 

In case of failure, I would have 1 (or 2) worker patrol around the queue looking for items that has been grabbed quite a while ago but haven't been marked as finished (or haven't been removed form the queue) and reset the flags on them.

[James Ward]

The RabbitMQ add-on is now GA:

https://addons.heroku.com/cloudamqp

-James

> To post to this group, send email to play-framework@googlegroups.com.

> To unsubscribe from this group, send email to

> play-framework+unsubscribe@googlegroups.com.

[James Roper]

Hi Oleg,

As far as I can see, no one has really answered your question, apart from the general "use akka remoting".  I've read the Akka remoting docs, and I don't really understand how it can be used to achieve the "run this on one and only one node" that you want to do, without implementing your own algorithms for the play nodes to elect who is currently responsible for running the job, and detecting when that node dies and so electing a new node.  If I'm wrong, and Akka does have first class support for this, please someone chime in at this point because it's not jumping out at me from reading the docs.

It also sounds like you're using Heroku.  I think in the Heroku way of doing things, what you do is you fire up worker dyno nodes to do async stuff.  In your case you would have one worker dyno doing the async stuff (it might be the same play application as runs your web dynos, but with a different application.conf file that instructed it to be in a "do async stuff" mode).  Then Heroku is responsible for ensuring that there is always one of those worker dyno nodes running, so you don't need to worry about failover, because if the worker goes down, Heroku just starts a new one.

Another approach would be to step back and think about what business function you are trying to achieve, before jumping ahead and requiring that your job only runs on one node.  Many business functions can be achieved by running the job on multiple nodes, but ensuring that the job is idempotent.  For example, if your business requirement is to send an email to your users once every two weeks, you might have a timestamp on your user records in your database, and then use an atomic find and update call on the database to get a list of users to send an email to (for example, if using MongoDB, you'd use findAndModify, querying for objects where the timestamp was older than 2 weeks, and setting it to the current timestamp).  Then you can run your job on every node.

Cheers,

James

> >> > To post to this group, send email to play-framework@googlegroups.com.
> >> > To unsubscribe from this group, send email to play-framework+unsubscribe@googlegroups.com.

> >> > For more options, visit this group athttp://groups.google.com/group/play-framework?hl=en.
>
> >> --
> >> Guillaume Bort
>
> > --
> > You received this message because you are subscribed to the Google Groups "play-framework" group.

> > To post to this group, send email to play-framework@googlegroups.com.
> > To unsubscribe from this group, send email to play-framework+unsubscribe@googlegroups.com.

[Oleg Golitsin]

Hi James,

Thanks for you explanation. The ways you described are the same I started to think about.

I also did not understand how to use akka remoting for this task.

Another solution could be to send scheduled messages to RabbitMQ that will route it to one of the worker dynos to trigger long-running processing.