New Pyramid Celery integration implementation

[Mikko Ohtamaa]

Hi all,

After struggling with this for few years I think I have finally nailed Pyramid and Celery integration. The design goals include

* No global variables or import time side effects. Everything is set up through Pyramid registry using venusian compatible @task decorator.

* Configured through INI settings

* Kick off delayed task on transaction commit

* Transaction retries in the case the task runs into database conflict resolution

* Support for eager execution

* Support for scheduled (beat) tasks

* Celery 4.0+ compatible

Here: http://websauna.org/docs/narrative/misc/task.html

If there is market demand I might factor this out to a separate package

--

Mikko Ohtamaa
http://opensourcehacker.com
http://twitter.com/moo9000

[Jonathan Vanasco]

On Sunday, August 21, 2016 at 6:41:35 AM UTC-4, Mikko Ohtamaa wrote:

* Configured through INI settings

* Kick off delayed task on transaction commit

 

If there is market demand I might factor this out to a separate package

I think it would be GREAT if there were a separate package/technique that standardized getting pyramid's ini file into celery.  

I think you may also want to support two types of firing for tasks:

A. Fires on commit

B. Fires as part of a two-phase commit (ie, a failure to fire votes NO)

I've been trying to get that second type to work in my spare time.  

 

[Ergo]

This is how I've been using celery in my pyramid applications with success, in a way that seems to work whever celery changes some internals:

https://github.com/ergo/testscaffold/blob/master/testscaffold/celery/__init__.py

then in app init i do:

    # configure celery in later phase

    def wrap_config_celery():

        configure_celery(config.registry)

    config.action(None, wrap_config_celery, order=PHASE3_CONFIG + 999)

This even allows for working with application plugins nicely.

I should probably make a cookbook recipe for it.

[Brian Findlay]

Mikko,

I't be awesome if you could factor this out. Amazing how many things Websauna gets right.

[jens.t...@gmail.com]

Thank you for this thread, good pointers!

I'd be curious to hear your opinions to this code review request: http://codereview.stackexchange.com/questions/148872/integrating-celery-task-queue-with-pyramid/

Thanks!