Re: how to start Celery worker in web2py

[Massimo Di Pierro]

Why do you want to use celery?

On Wednesday, 6 March 2013 19:47:12 UTC-6, Eric S wrote:

I would like to use Celery in my web2py application, but I'm having trouble with how to start a Celery worker (I know there is a web2py scheduler but I would like to use Celery).

To start a custom scheduler in web2py I would use:

python web2py.py -S appName -M -R worker.py

Celery workers, however, are launched from the command line such as with the following command, which won't easily substitute into 'worker.py' above:

celery -A tasks worker --loglevel=info

Has anyone had success integrating web2py and Celery? How do you (robustly) get around this problem?

Thanks,

Eric

[Bruno Rocha]

I also would like to see Celery, Solr, Elastic Search and other fantastic tools working with web2py!

I think this is an important issue and I am sure it is completely easy and possible to make it.

I personally do not like to use the built-in scheduler, so I am using python-rq (Redis Queue) for some production sites and it works very well and offer almost all celery functionalities.

Maybe someone can follow my python-rq[1] example and create a wen2py-celery tutorial

[1]http://rochacbruno.com.br/web2py-and-redis-queue/ 

I dont think web2py needs to always reinvent the wheel so I would like to see more integrations.

wish list:

Whoosh (WIP)
Solr (maybe a haystack clone for web2py)
ElasticSearch
Celery
Thumbor
Neo4J

On Wed, Mar 6, 2013 at 10:47 PM, Eric S <ericea...@gmail.com> wrote:

I would like to use Celery in my web2py application, but I'm having trouble with how to start a Celery worker (I know there is a web2py scheduler but I would like to use Celery).

To start a custom scheduler in web2py I would use:

python web2py.py -S appName -M -R worker.py

Celery workers, however, are launched from the command line such as with the following command, which won't easily substitute into 'worker.py' above:

celery -A tasks worker --loglevel=info

Has anyone had success integrating web2py and Celery? How do you (robustly) get around this problem?

Thanks,

Eric

--
 
---
You received this message because you are subscribed to the Google Groups "web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to web2py+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Niphlod]

+1 .
Scheduler is a great tool because its feature packed and exploits what is "at hand" in a normal deployment environment (and it's the best shot at replacing cron & likes). The minute I had a redis-backed scheduler at hand (its there, sitting on my disk) I was kinda sad, because what makes the scheduler great is the ease of interaction with the database.... All users needed to switch to a complete different toolset to queue tasks, at that point... better to rely on something battle-tested (always a problem in web2py environment, too many newbies and little or few testers).

For a realtime offloading tool Celery it's probably the best solution out there (kinda harder to config, but hey....), pyres it's my favourite and rq and huey are littler in comparison but much easier to config and solve the 80% of the problems. I used them only as a mere task dispatchers/processors, i.e. outside the web world.

PS: Bruno, I tried to follow your code when you originally posted it, but it had 2 "problems" :
- 1: every request ends up building a connection towards Redis (I think a singleton is needed)
- 2: I remember some problems about scheduling functions defined in models (i.e. adapting to the "crazy" execution model of web2py)
As soon as I'll get home I'll try again, but is there a chance that in the meantime those problems have been already fixed ?

[Massimo Di Pierro]

What I want to know is what do you think celery buys you that the built-in scheduler does not provide?

Celery is faster at transferring messages from the application t the workers and vice versa but normally when you want to run background tasks you have different bottle-necks: computation cycles of background tasks and database access from the tasks. In this respect I do not see celery being any better than the built-in scheduler. Actually the built-in scheduler makes your life easier by exposing the web2py environment to workers which is something celery would not be able to do.

[Niphlod]

more than celery we'd need a generalized web2py "create the context" recipe .... running tasks defined in modules is easy..... running tasks outside web2py that needs the usual environment is a PITA ....

On Tuesday, March 12, 2013 6:17:47 PM UTC+1, Eric S wrote:

I'm interested in a robust, widely-adopted scheduler. The current web2py Scheduler is clearly changing very rapidly, which is great, for now I want a scheduler that is mature.

Can anyone answer my original question -- how have you gotten Celery workers to run with web2py?

[Eric S]

Is it not possible launch a Celery worker that has access to the web2py environment? This is possible for custom schedulers with commands such as the following -- why would it not be available to Celery workers?

python web2py.py -S appName -M -R worker.py

On Tuesday, March 12, 2013 10:17:47 AM UTC-7, Eric S wrote:

I'm interested in a robust, widely-adopted scheduler. The current web2py Scheduler is clearly changing very rapidly, which is great, for now I want a scheduler that is mature.

Can anyone answer my original question -- how have you gotten Celery workers to run with web2py?

On Thursday, March 7, 2013 9:47:25 AM UTC-8, Massimo Di Pierro wrote:

[Niphlod]

cause I know exactly how to create a web2py environment in "my own module" but for the life of me I can't figure out how to apply the same method to a celery worker (if you're not a fan of "magic" in web2py, celery workers instances do a LOOOOT of magic behind the scenes)

[Niphlod]

ok, I think I got it ..... most of it is cumbersome and fruit of multiple reiteration (e.g. lots of trials and errors). There's no way in hell to put up celery docs pertaining the particular usecase in web2py.
Pleeeeeease, watch it carefully, may burn your house to the ground.
@Bruno: maybe the same thing can be applied to rq (I still have problems with running that one with functions not defined in models outside web2py environment)
Chosen broker and result backend --> redis (absolutely no time to install rabbitmq, sorry)

1st issue: models are executed at every request, so a celery instance defined in models continues to instantiate new connections to Redis. Although redis is fast as hell, it just mean wasting resources (a single redis instance makes use by default of a connection pool, but if the object is recycled and istantiated at every request the pool doesn't survice). So, celery instance must be defined in a module and imported
1st step, hence --> create modules/w2p_celery.py with

from celery import Celery
mycelery = Celery('tasks', broker='redis://localhost:6379/0', backend='redis://localhost:6379/0')

2nd issue unresolved: the default task decorator computes a name based on the current execution instance (for newcomers to web2py, __restricted__)
Celery has a task decorator that takes a name parameter, so it's quite easy to patch (even so, I'd really like to be able to customize that. Suggestions ?)
However, if you put this in models/thecelerymodel.py

from w2p_celery import mycelery
celery = mycelery
@celery.task(name='tasks.gen_url')
def gen_url(x):
    return A(x, _href=URL('rule_the_world'))

@celery.task(name='tasks.add_user')
def add_user():
    try:
       db.auth_user.insert(first_name='John')
       db.commit()
    except:
       db.rollback()

you'll have two tasks registered as 'tasks.add' and 'tasks.add_user'

3rd step: f*******k the world. having something runnable by a celery worker. This part was the absolute nightmare of the last two evenings. I really didn't get the problems in it, but seems that task "code" is somewhat-where-when shipped around (or, my nightmares were in fact the reality....)
Given that, let's assume that copying the tasks defined before elsewhere won't get you killed in action (i.e. yes, I'd like too to have celery workers figuring out where to pick something from, but I just couldn't get there)
3rd step --> create a tasks.py in the same folder as web2py.py

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import gluon.widget        #forgot: why this is apparently the only way to fix custom imports ?
from gluon.shell import env
from gluon import current
from celery import Celery

def make_celery(app):
    celery = Celery('tasks', broker='redis://localhost:6379/0', backend='redis://localhost:6379/0')
    TaskBase = celery.Task
    class ContextTask(TaskBase):
        abstract = True
        def __call__(self, *args, **kwargs):
            _env = env(a=app, import_models=True)
            globals().update(_env)
            return TaskBase.__call__(self, *args, **kwargs)
    celery.Task = ContextTask
    return celery

celery = make_celery('yourappname') #be sure that you write this correctly

@celery.task(name='tasks.gen_url')
def gen_url(x):
    return A(x, _href=URL(x, 'rule_the_world'))

@celery.task(name='tasks.add_user')
def add_user():   #yes, for the love of your apps, wrap all db operations!
    try:
        db.auth_user.insert(first_name='miao')
        db.commit()
    except:
        db.rollback()

now, this **seems** to work without issues, e.g. in a controller you can do without harm

def myfunction():
    async = add_user.delay()
    return dict(async=async)

and start an instance of celery, cding into web2py.py folder and doing

>>> celery worker -A tasks

Again, this is not the recommended way nor the "documented somewhere" one. It is just the first **iteration** that didn't lend in exceptions all over the place.
If someone wants to pick this up, go to ask (solem) and get it corrected/revised, I'll be glad to have spent 6 hours for the web2py community :P

@PS on the robust-widely-adopted argument: celery vs web2py's scheduler is like comparing web2py to web.py. They have VERY DIFFERENT goals in mind and celery is by far the most complete task queue solution out there (even comparing other programming languages task queues). With that in mind (i.e. even Niphlod in the need of a gigantic solution for out-of-the-band processes chooses celery), celery broke a few times backward compatibility. Web2py's scheduler got new features every iteration and just broke code when it had no API whatsoever to retrieve task results (has just changed the db schema once, to accomodate all engines reserved keywords) 

[Massimo Di Pierro]

This belongs to a blog post!

[Niphlod]

if it's right, and I'm not sure it is since I couldn't get the to the "inner circle of knowledge" required to run such a thing and tell all the others "people, THIS is the way" ^_^

[黄祥]

trying to learn web2py with celery, but an error occured

- install celery

source activate test2

pip install celery

- start redis server from source install (stable version 4.0.9)

./src/redis-server

- start web2py (stable version 2.16.1 on python 2.7)

source activate test2

python ~/python/web2py/web2py.py --nogui --no-banner -a a -i 0.0.0.0 -p 8000

- create new web2py app named : celery

- create modules

modules/w2p_celery.py

from celery import Celery

mycelery = Celery('tasks', broker='redis://localhost:6379/0', backend='redis://localhost:6379/0')

- create models

models/thecelerymodel.py

from w2p_celery import mycelery

celery = mycelery

@celery.task(name='tasks.gen_url')

def gen_url(x):

    return A(x, _href=URL('rule_the_world'))

@celery.task(name='tasks.add_user')

def add_user():

    try:

       db.auth_user.insert(first_name='John')

       db.commit()

    except:

       db.rollback()

- create tasks.py in the same folder as web2py.py

#!/usr/bin/env python

# -*- coding: utf-8 -*-

import gluon.widget        #forgot: why this is apparently the only way to fix custom imports ?

from gluon.shell import env

from gluon import current

from celery import Celery

def make_celery(app):

    celery = Celery('tasks', broker='redis://localhost:6379/0', backend='redis://localhost:6379/0')

    TaskBase = celery.Task

    class ContextTask(TaskBase):

        abstract = True

        def __call__(self, *args, **kwargs):

            _env = env(a=app, import_models=True)

            globals().update(_env)

            return TaskBase.__call__(self, *args, **kwargs)

    celery.Task = ContextTask

    return celery

celery = make_celery('celery') #be sure that you write this correctly

@celery.task(name='tasks.gen_url')

def gen_url(x):

    return A(x, _href=URL(x, 'rule_the_world'))

@celery.task(name='tasks.add_user')

def add_user():   #yes, for the love of your apps, wrap all db operations!

    try:

        db.auth_user.insert(first_name='miao')

        db.commit()

    except:

        db.rollback()

- start celery

cd ~/python/web2py/

source activate test2

python

>>> import celery

>>> celery worker -A tasks

  File "<stdin>", line 1

    celery worker -A tasks

                ^

SyntaxError: invalid syntax

any idea?

thanks and best regards,

stifan

[Anthony]

>>> celery worker -A tasks

  File "<stdin>", line 1

    celery worker -A tasks

                ^

SyntaxError: invalid syntax

The above is not Python code -- it belongs at the command line.

Also, I notice you often add replies to threads that are several years old. Unless you are directly replying to that thread, you should instead just start a new post with an appropriate title -- that way we don't develop very long threads that branch off into only tangentially related questions (if it is helpful for context, your new post can include links to old threads).

Anthony

[James O' Driscoll]

I have implemented the above, the gen_url function is working but the add_user function is not (receiving the error db is not defined from tasks.py.)

Regards,

James

[James O' Driscoll]

I used DAL(uri, folder, import_models=True) in the task.py function and this does the job.

If possible can someone explain if:

1.    This lightweight connect to celery is preferred over https://code.google.com/archive/p/web2py-celery/.

2.    Is running the worker using task manager as a windows service the best practice?

Regards,

James

[Alexandra Lee]

hello. 

Can you unsubscribe me please? I am not currenly a web2py user. But maybe in future. 

Please remove my email from you list for now. 

Kind regards, 

Alexandra 

--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)

---
You received this message because you are subscribed to the Google Groups "web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to web2py+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.