[rabbitmq-discuss] Getting stats from celery

[Siddharth Saha]

Hi,

I want to write a script which will keep track of what tasks have been submitted and processed and whether the status of the job was Success / Failure and how much time did each task take to finish. I want to dump these info in a database so that I can pull out reports now and then about the performance.

Please give me some cues on how I can proceed on this.

I am using RabbitMQ with Celery in my Python project. Looking forward for some help.

--
-Siddharth Saha

[Steven Taylor]

a few thoughts:

 

declare "monitor" queue:

* register task (must be done via RPC)  --> dump to database

* report on status at each stage: started, pending, completed.  --> dump to database

 

Why RPC?  Because rabbit guarantees delivery, but not the order in which messages are delievered in.  RPC at the start ensures that later monitoring has a context.  It just seems a little tidier.

 

Let me know if this helps.

 

btw: I haven't had a chance to look at Celery... so don't know on that.

 

-Steven

 

 

 

_______________________________________________
rabbitmq-discuss mailing list
rabbitmq...@lists.rabbitmq.com
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[Steven Taylor]

btw: ---->  was supposed to mean "send over queue" to a consumer... just making that clear.

[Ask Solem]

Hey,

Have you not seen the django-celery monitor?
It does exactly what you describe:

http://docs.celeryproject.org/en/latest/userguide/monitoring.html#django-admin-monitor

[Siddharth Saha]

Unfortunately, I am not doing a Django project and don't want any dependencies on Django. Any API library using which I can write my own small customized script ? 

On Thu, Jul 21, 2011 at 2:57 PM, Ask Solem <a...@rabbitmq.com> wrote:

On 21 Jul 2011, at 06:04, Siddharth Saha wrote:

> Hi,
>
> I want to write a script which will keep track of what tasks have been submitted and processed and whether the status of the job was Success / Failure and how much time did each task take to finish. I want to dump these info in a database so that I can pull out reports now and then about the performance.
>
> Please give me some cues on how I can proceed on this.
>
> I am using RabbitMQ with Celery in my Python project. Looking forward for some help.
>

Hey,

Have you not seen the django-celery monitor?
It does exactly what you describe:

http://docs.celeryproject.org/en/latest/userguide/monitoring.html#django-admin-monitor

--
-Siddharth Saha

[Ask Solem]

On 21 Jul 2011, at 10:29, Siddharth Saha wrote:

> Unfortunately, I am not doing a Django project and don't want any dependencies on Django. Any API library using which I can write my own small customized script ?
>

The code to do this is mostly part of Celery itself, django-celery just writes
it to a Django model:

https://github.com/ask/django-celery/blob/master/djcelery/snapshot.py