RhodeCode does not send tasks to Celery and RabbitMQ

[Vairav]

Hi,
I have configured Rhodecode, Celery and RabbitMQ.

I started RabbitMQ first, followed by Celery :
paster celeryd production.ini

Then I started RhodeCode:
paster serve production.ini

Everything seems to work fine, but when I access the console of
RabbitMQ at:
http://locahost:55672/

I can see the celeryuser and celeryvhost, but I dont see any Queued
messages, i.e. the total messages is always 0 (zero) even after I did
several pull, pushes and after re-indexing Whoosh with the repsotory
also.

My production.ini Celery setting is as follows:
use_celery = true
broker.host = localhost
broker.vhost = celeryvhost
broker.port = 5672
broker.user = celeryuser
broker.password = celerypass

celery.imports = rhodecode.lib.celerylib.tasks

celery.result.backend = amqp
celery.result.dburi = amqp://
celery.result.serialier = json

#celery.send.task.error.emails = true
#celery.amqp.task.result.expires = 18000

celeryd.concurrency = 2
#celeryd.log.file = celeryd.log
celeryd.log.level = debug
celeryd.max.tasks.per.child = 1
#tasks will never be sent to the queue, but executed locally instead.
celery.always.eager = true


I ran it with the above configuration, but I can see Celery connecting
to RabbitMQ, but nothing from the Rhodecode. Am I missing something
here?

Also, I did change the setting:
celery.always.eager = false

But this too dint work, meaning, I dont see any messages on the
RabbitMQ page.


Please advice and any guidance would be really great :)


Thanks and regards,
Vairav

[Vairav]

Just update on the issue that I am having:

I issued a Whoosh re-indexing task for scratch re-building of the
index from the Web front-end.
Then later I checked back into console and found that RhodeCode had
started to perform the indexing operation.
After the indexing was completed, it then later showed the log message
stating that there was a task for Celery through some libraries in
RhodeCode.
I apologize, but I do not have the log file with me, I will update
them once when I get back to my office on Monday.

Just wanted to post this update so that it might help others who are
having the same doubts.


I've still not been able to verify if RabbitMQ is actually used by
Celery and secondly, if Celery is being used by RhodeCode.

If anyone could throw some light on how RhodeCode is using Celery it
would be really great and really helpful.


Thanks again,
Vairav

[Matt Zuba]

#tasks will never be sent to the queue, but executed locally instead. 
celery.always.eager = true 

Have you tried setting that to false?

The best way to see if rhodecode is using celery (which implies that rabbit is also being used) is to either do a 'tail -f' on the celeryd log file (if you set one), or monitor the output from the command line after running the 'paster celeryd' command.  You will actively see results scrolling by during page loads or indexing if it is working properly.

[Vairav]

Thank you so much for the information Matt :)

I set the 'celery.always.eager = false' and now I'm able to see that
Celery is receiving the tasks from RhodeCode and executing them.
And I apologize for my ignorance on how the systems were working. But
now I kinda understand on how the systems are working together.


Also, I have another quick general question. Is it that Celery just
uses the "Worker Processes" from RabbitMQ for performing its
activities, and in relation to that, RabbitMQ does not show up any
messages to it. So does it mean that RabbitMQ is only for providing
the worker processes for Celery and does not receive or send any
messages?


Thank you once again for the help :)

I hope this thread helps people like me who are looking to understand
the systems flow in terms of connecting Rhodecode, Celery and
RabbitMQ.


Regards,
Vairav

[mbrownnyc]

Matt/Vairav/Marcin (if you're watching):

Why is this set to true by default?

If the jobs aren't submitted to the queue efficiently, does this
indicate that there is a problem with the queue config?


From what Vairav reports, it appears that celery wasn't servicing
efficiently with celery.always.eager set to 'true.' Why is this?


Thanks,

Matt

[Matt Zuba]

My basic understanding of the interaction between Rhodecode, RabbitMQ and Celery is kinda like this:

When you set your ini file to use celery, Rhodecode creates a socket connection to RabbitMQ based on the 'broker' settings you give it.  When there are tasks to be completed, Rhodecode sends these tasks to RabbitMQ to be queued.

After Celery is started (with however many threads/processes you give it), it then connects to RabbitMQ to see if there are any items waiting for it in the queue for it process.  If so, it retrieves it and processes it.

To directly answer your question, the worker threads are spawned from Celery, not Rabbit.  Rabbit is only a service to receive and hold messages until the Celery workers retrieve them.

[Matt Zuba]

I checked the deployment ini template on Bitbucket and this option is set to 'false' in at least version 1.2.3 and on.

My guess is that this option reverts the overall behavior of the script to act as if you weren't using Celery at all.  Marcin might be able to provide more insight into why this option is included in the config file, I can only speculate.

[Marcin Kuzminski]

CELERY_ALWAYS_EAGER is false by default : https://secure.rhodecode.org/rhodecode/files/default/rhodecode/config/deployment.ini_tmpl#L82

Always eager make the tasks run in sync mode, which IMHO basically kills the benefits of having CELERY at all.

This option is there for tests, if you're having problem with a message broker(rabbitmq for example) it's sometimes good to run tasks with eager mode for testing and debugging.


Cheers

[mbrownnyc]

Thanks guys!

On Jan 10, 12:06 am, Marcin Kuzminski <marcin...@gmail.com> wrote:
> CELERY_ALWAYS_EAGER is false by default :https://secure.rhodecode.org/rhodecode/files/default/rhodecode/config...

[Vairav]

Thank you so much for the information Matt and Marcin. I googled up
for the production.ini and now I noticed that it was a old one that I
was using and so the default in that was "true". I've corrected my
mistake :)
I'm now able to understand on how the systems work together.

But I have another query, wherein, I have configured RhodeCode,
RabbitMQ and Celery like you have mentioned. When I checked RabbitMQ,
I'm not able to see any messages being queued to it which would be
consumed by Celery.
But what I noticed is that when Celery registers itself with RabbitMQ,
the total workers consumed in RabbitMQ goes up.
Also, I noticed that RhodeCode is sending the work to Celery and
Celery starts processing it immediately and in the mean time I checked
RabbitMQ webconsole to see if it has received any messages, but dint
see any there, so I'm not sure on how the messages go to the queue in
RabbitMQ. Some light on this would be really great in understanding
much more.


Thank you again,
Vairav