Re: [celery-users] Celery + mysql as results and broker

[Ask Solem]

On 9 Oct 2012, at 16:50, Recif Deregle <leger.f...@gmail.com> wrote:

> Hi all,
>
> I work in a company who is giving a try to celery. So far its seems exactly what we need
> but i have pb configuring it to use a mysql backend for both broker & results.
>
> i know this is not the privileged way but we already have to integrate with an existent solution
>
> does an expert can give me a hand to have this working ?
>
> can't find any ref to the error i am having or any tutos for using it with mysql.
> CELERY_RESULT_BACKEND = "database"
> CELERY_RESULT_DBURI = "sqla+mysql://dbuser:dbpass@dbhost/celery"


The RESULT_DBURI setting is always a SQLAlchemy connection url so shouldn't
have the 'sqla+' prefix:

CELERY_RESULT_DBURI = "mysql://dbuser:dbpass@dbhost/celery"


AFAIR the sqlalchemy broker transport has a serious flaw in that
it never removes any messages. I fixed the Django transport to clean up
messages from time to time, but the sqla transport has not been updated yet.

If you are dedicated to using sqla this shouldn't be too hard to implement,
you can look at the Django transport to see how I did it there:
https://github.com/celery/kombu/blob/master/kombu/transport/django/managers.py#L50-78

Or you can just add a periodic task to delete rows from the Message table with
the visible flag set.

The implementation of both is rather rudimentary since I don't have time
to maintain them, but it shouldn't take much work to make much better.
--
Ask Solem
twitter.com/asksol | +44 (0)7713357179

[Recif Deregle]

Hi Ask,

Thanks for the answer. Indeed that was my last try to make it works. But the same behavior is hapenning when using :

CELERY_RESULT_DBURI   = "mysql://dbuser:dbpass@dbhost/celery"

Do you have any idea of what i'm missing ?

The clear of the messages queue should not be a problem, and if it works well we might go for RabbitMQ.

Thanks for your help

Cheers

[Recif Deregle]

For the record, it seems i fixed my own problem in 2 steps :

1. back to CELERY_RESULT_DBURI   = "mysql://dbuser:dbpass@dbhost/celery" settings
2. set PYTHONPATH in /etc/default/celery

Cheers

[Dhana Raghu]

Hi Recif Deregle ,

                                 There is no file named  celery in the etc/default folder. I have celery.py in a different folder. Can you please tell me what code are you writing inside celery.py and more importantly in which method.

I mean to ask how are you setting the PYTHONPATH. Can't we set it else where?

Regards

Dhana

[Dhana Raghu]

Hi,
        There is a better solution. You can ignore the CELERY_RESULT_DBURI property as mentioned in the  http://docs.celeryproject.org/en/latest/configuration.html#celery-result-dburi.

In the celeryconfig.py set the following :-
CELERY_RESULT_BACKEND = 'db+mysql://dbuser:dbpass@dbhost/celery'

In the task01.py:-
Declare celery as follows:-
celery = Celery('task01')

That's all. You are all set to go.

Regards
Dhana

On Tuesday, October 9, 2012 9:20:38 PM UTC+5:30, Recif Deregle wrote:

Hi all,

I work in a company who is giving a try to celery. So far its seems exactly what we need

but i have pb configuring it to use a mysql backend for both broker & results.

i know this is not the privileged way but we already have to integrate with an existent solution

does an expert can give me a hand to have this working ?

can't find any ref to the error i am having or any tutos for using it with mysql.

any help much appreciated.

cheers

=== task01.py ===

from celery import Celery

# define a queue

celery = Celery('task01', backend='database', broker='sqla+mysql://dbuser:dbpass@dbhost/celery')

# load the configuration

celery.config_from_object('celeryconfig')

# a task definition

@celery.task

def add(x, y):

    return x + y

=== celeryconfig.py ==

## Broker settings.

BROKER_URL = "sqla+mysql://dbuser:dbpass@dbhost/celery"

## Using the database to store task state and results.

CELERY_RESULT_BACKEND = "database"

CELERY_RESULT_DBURI   = "sqla+mysql://dbuser:dbpass@dbhost/celery"

# echo enables verbose logging from SQLAlchemy.

# CELERY_RESULT_ENGINE_OPTIONS = {"echo": True}

# timezone

CELERY_TIMEZONE = 'Europe/Paris'

=== running the queue ===

# /opt/celery/bin/celery -A task01 --loglevel=info worker

<snip>

[2012-10-09 17:47:32,155: WARNING/PoolWorker-6] return backend(app=self, url=url)

[2012-10-09 17:47:32,155: WARNING/PoolWorker-6] File "/opt/celery/lib/python2.6/site-packages/celery-3.0.11-py2.6.egg/celery/backends/database/__init__.py", line 70, in __init__

[2012-10-09 17:47:32,155: WARNING/PoolWorker-6] 'Missing connection string! Do you have '

[2012-10-09 17:47:32,155: WARNING/PoolWorker-6] ImproperlyConfigured: Missing connection string! Do you have CELERY_RESULT_DBURI set to a real value?