Asynchronous & Results Status for Progress Bar?

[Kurtis Mullins]

Hey,

I'm attempting to run a large set of tasks which depend upon each other. There are two things in the documents that I'm trying to follow.

First, I'd like to run these tasks Asynchrounously as described on http://docs.celeryproject.org/en/latest/userguide/tasks.html#avoid-launching-synchronous-subtasks . However, I'd also like to be able to report the current progress back to the main task so that I can display a Progress Bar to the user.

I know I can restructure my code to follow more of a functional paradigm as opposed to a procedural one to run it asynchronously but then I don't know how to grab the status information properly. Here's the general outline of what I'm following to pull that Progress data: http://ask.github.com/celery/userguide/tasks.html#custom-states

Does anybody have any suggestions on how to do this? Or should I just go synchronous and risk the deadlock problem?

Thanks! 

[Jonathan French]

It's probably easier to send your progress messages some other way, in your database or e.g. Redis if you're using it. If your application is web-based, you could look into using Orbited/Morbid to send realtime progress messages to the browser via STOMP, avoiding the need for your 'main task' to keep running at all.

- ojno

--
You received this message because you are subscribed to the Google Groups "celery-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/celery-users/-/bgeK_8_KPdIJ.
To post to this group, send email to celery...@googlegroups.com.
To unsubscribe from this group, send email to celery-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/celery-users?hl=en.

[Kurtis Mullins]

Thanks for the quick reply! I think I may go the Redis route since I'm already using it anyways.

Also, thanks for introducing me to Orbited/Morbid and STOMP. I've never heard of them. From a quick DuckDuckGo Search and some reading, STOMP reminds me a lot of Nathan Marz's Storm architecture/paradigm which was powerful and very pleasant to use (after the initial learning curve). I'll definitely play around with them in the future.

- ojno

To unsubscribe from this group, send email to celery-users+unsubscribe@googlegroups.com.

[Ask Solem]

On 4 Mar 2012, at 09:53, Kurtis Mullins wrote:

> Thanks for the quick reply! I think I may go the Redis route since I'm already using it anyways.
>
> Also, thanks for introducing me to Orbited/Morbid and STOMP. I've never heard of them. From a quick DuckDuckGo Search and some reading, STOMP reminds me a lot of Nathan Marz's Storm architecture/paradigm which was powerful and very pleasant to use (after the initial learning curve). I'll definitely play around with them in the future.
>
> On Saturday, March 3, 2012 8:46:27 AM UTC-5, ojno wrote:
> It's probably easier to send your progress messages some other way, in your database or e.g. Redis if you're using it. If your application is web-based, you could look into using Orbited/Morbid to send realtime progress messages to the browser via STOMP, avoiding the need for your 'main task' to keep running at all.
>

If you can update the browser in real-time that is the best solution,
but you can also have each task return the result of its subtask(s) to traverse
the graph of dependencies and that by only having a reference to the first
result, see my answer in this thread:
https://groups.google.com/group/celery-users/browse_thread/thread/ccb2414bb044d3a8/cf27c05565f1d18a?lnk=gst&q=callback+state+propagation#cf27c05565f1d18a

--
Ask Solem
twitter.com/asksol | +44 (0)7713357179