Django-utils @async decorator and gunicorn

[Anton Pirker]

Hello!

I use djutils [1] @async decorator for calling a function asynchronous.
This works well when i start my server with the standard ./manage.py
runserver command. But when i run my django app under gunicorn and i
call the function with the @async decorator nothing happens at all...

Do i have to configure djutils or gunicorn in a special way so i can run
function asynchronous?

Any hints are greatly appreciated!

Thank's
Anton

[1] http://charlesleifer.com/docs/djutils/

--
DI(FH) Anton Pirker

---------------------------------------------
IGNAZ Softwaremanufaktur
mobile . web . usability . project management

kranzgasse 2/15
a-1150 wien
tel: +43 699 1234 0 456
skype: antonpirker

http://ignaz.at

[Javier Guerra Giraldez]

On Mon, Jun 27, 2011 at 4:50 AM, Anton Pirker <an...@ignaz.at> wrote:
> But when i run my django app under gunicorn and i call the function with the
> @async decorator nothing happens at all...

are you running the queue consumer daemon?

--
Javier

[Anton Pirker]

Hello!

I use djutils [1] @async decorator for calling a function asynchronous. This works well when i start my server with the standard ./manage.py runserver command. But when i run my django app under gunicorn and i call the function with the @async decorator nothing happens at all...

[Anton Pirker]

Hi!

Ah! There is a queue consumer daemon! ;) Thank's for this hint!

I have now the queue consumer configured in my supervisord.conf like this:

[program:preview_queue_consumer]
environment=PYTHONPATH="/home/preview/.virtualenvs/preview/source/myproject/:/home/preview/.virtualenvs/preview/lib/python2.6/site-packages/djutils/:$PYTHONPATH"
directory=/home/preview/source/myproject/
command=python manage.py queue_consumer --settings=myproject.settings -l
logs/queue_consumer.log --verbosity=2 -t 2
user=preview
autostart=true
autorestart=true

when i start the daemon the logs says this:

delay: 0.1
backoff: 1.15
threads: 2
2011-06-28 09:18:06,329:djutils.queue.logger:INFO:Loaded classes:
djutils.commands.queuecmd_delayed_resize
djutils.queue.queue.QueueCommand
djutils.queue.queue.PeriodicQueueCommand
2011-06-28 09:18:06,329:djutils.queue.logger:INFO:Starting periodic
command execution thread
2011-06-28 09:18:06,329:djutils.queue.logger:INFO:created thread
"1098918224"
2011-06-28 09:18:06,329:djutils.queue.logger:INFO:created thread
"1107310928"

so i assume the daemon is started.

but when i run my @async function nothing happens. in the log output
above i can see that QueueCommand and PeriodicQueueCommand are loaded.
can it be, that the consumer can only consume this sort of functions and
not functions with @async?

i tried to add the @queue_command instead of the @async to my function,
but then i get the error:
"can't pickle StringO objects"
and i do not have a clue what this means. (maybe it's my lack of english
skills, but what should 'pickle' in this context mean?)

anyone an idea or hint where i should look next, to get my @async
functions running in gunicorn?

thank's in advance,
Anton

[Javier Guerra Giraldez]

On Tue, Jun 28, 2011 at 2:53 AM, Anton Pirker <an...@ignaz.at> wrote:
> Ah! There is a queue consumer daemon! ;)  Thank's for this hint!

just checked and the @async decorator doesn't use the queue/consumer
facility also included. sorry for the wrong hint.

what @async does is adding to a python-standard Queue object, and also
spawining one or more threads to consume the queue (so no need for a
daemon).

another thing I don't know about is gunicorn; but the 'g' in the name
comes from 'green threads'. do you (or anybody else) know if that
means that it patches the python thread implementation? if so, that
could be interfering with the @async thread.

OTOH, now that you have the daemon running, maybe it would be easier
to use the djutils task queue. try changing the @async with
@queue_command (found in djutils.queue.decorators)

--
Javier

[Anton Pirker]

Hi!

On 06/28/2011 02:45 PM, Javier Guerra Giraldez wrote:
> another thing I don't know about is gunicorn; but the 'g' in the name
> comes from 'green threads'. do you (or anybody else) know if that
> means that it patches the python thread implementation? if so, that
> could be interfering with the @async thread.

"g" stands for green unicorn. and we use gevent with gunicorn. so it
could be that there is a problem with the threading implementation..

> OTOH, now that you have the daemon running, maybe it would be easier
> to use the djutils task queue. try changing the @async with
> @queue_command (found in djutils.queue.decorators)
>

i have tried this, but it resulted in an strange error.

i think i will skip the asynchronous stuff from djutils. tomorrow i will
install celery [1] and try my luck with that. :)

thanks for your help,
Anton

[1] http://celeryproject.org/