Resque equivalent in Play/Scala

[Camille Wanty]

Hi,

I have a very simple need for a Job queue in my application. 

I don't need my queue to be distributed, it won't need to handle millions concurrent requests. 

It just need to be persistent and shared among my play nodes.

Rails/Ruby has Resque, Python has Celery, is there an equivalent in the Scala world ?

I know I could code it myself with Akka and persistent actor backed by something like Redis, but I'm not really in the business of building such a tool. 

And I'm guessing that such a need is quite common, we don't all develop Twitter or Facebook size systems.

I would appreciate some help on this,

Thanks

[Christian Schmitt]

You are looking for http://doc.akka.io/docs/akka/2.4.2/scala/persistence.html#persistence-scala ?

[Will Sargent]

Do you mean something like https://github.com/adamw/reactmq ?

Will.

--
You received this message because you are subscribed to the Google Groups "play-framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to play-framewor...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/play-framework/da9cba9c-162d-4cef-a5c6-7b030d240275%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Manuel Bernhardt]

For an "idiomatic" Scala / Play / Akka solution the preferred way would be to use akka-persistence or if you want to use a queue that can interoperate with other technology stacks you could use RabbitMQ and the akka adapter at https://github.com/thenewmotion/akka-rabbitmq

To view this discussion on the web visit https://groups.google.com/d/msgid/play-framework/etPan.56ed80a2.6c0b19a7.d39f%40Williams-MacBook-Pro-2.local.

[Camille Wanty]

Hi thanks everyone for the answers already given,

It seems to me that what I would need is a durable mailbox, can it be done with akka-persistence ? I had the impression that it was made first for event sourcing as in "append only" repository. I think I might miss something here but I don't want to store every existing messages sent to my actor. Do you mean that I could just persist whatever state I want, like a mutable.Queue, each time a "job" message is consumed ? Sorry if I sound a bit confused, but I am :)

Thanks,

Camille

[Camille Wanty]

I eventually followed your suggestions and used Akka and persistence Actors backed by Redis using akka-persistence-redis plugin.

My queue persistence actor maintain a state (mutable queue) and dispatch work to worker actor within a configurable pool of worker actor.

The work actor then transfers the work to an executor actor to do the actual calculation.

For the moment the whole thing is quite custom, but I guess it can be generalised. 

The solution I came up with feels quite engaged and I'm not confident it is production ready. 

But anyway, it works for my use case for now.

Do any one that needs a job queue manager in their play/scala or akka apps build it all by hand, each time ?

[Will Sargent]

Hi Camille,

> Do any one that needs a job queue manager in their play/scala or akka apps build it all by hand, each time ?

If you're able to share your solution as a github project and write it up as a blog post, 

I'm sure that a number of people would find it useful.

I can also add it to the blog posts page:

https://github.com/playframework/playframework/blob/master/documentation/manual/gettingStarted/Tutorials.md

Also, if you're asking if other people do this, James Ward has finished an example project showing Kafka working with Websocket and Akka Streams that might be useful:

* https://www.jamesward.com/2016/05/25/combining-reactive-streams-heroku-kafka-and-play-framework/

* https://github.com/jamesward/hello-play-kafka

Will.

--

You received this message because you are subscribed to the Google Groups "play-framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to play-framewor...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/play-framework/af58cacb-3bc6-4347-8d0b-d7b10a832464%40googlegroups.com.

[Thibault Meyer]

You can try RabbitMQ with async consumer (and dont us Akka worker).  Lack of built in persistence in Akka is, in fact, is a really bad point for Play.

With RabbitMQ you lose ability to schedule (ie: run this task in 30 minutes) but you get the security that a task is never lost, even if worker crash, the task is not deleted from the queue  until the work ACK it.