Creating sub-processes from Pyramid application

[Thierry Florac]

Hi,

I need to create custom sub-processes from my main Pyramid application; these processes are used to handle "commands" received from ZeroMQ messages.

If I use the components registry to register adapters and utilities which are also required in these processes, is there a way to start these processes so that they can "inherit" from the main application registry ?

I'm actually using Pyramid 1.10.5, and I generally do a "config.hook_zca()" on application startup before doing the "config.setup_registry()" and "config.scan()".

I tried to do the same operations in my sub-process "run" method, but I always get the "global" registry instead of my "webapp" registry which was configured in my main application...

Best regards,

Thierry

-- 

  https://www.ulthar.net -- http://pyams.readthedocs.io

[Jonathan Vanasco]

Have you considered using Celery for this?  I started offloading everything related to subprocesses and message queues to it a while back, and have been much happier.

[Thierry Florac]

Hi Jonathan,

I didn't have a look at Celery yet, maybe it could work...

My goal is to be able to start long-running tasks (medias files conversions for example) whose definition is stored in database and managed by end users from my main Pyramid application; for this, I create a dedicated sub-process which is "managed" using ZeroMQ messages (I use this because it's also used for other non-Python/Pyramid applications).

Tasks are then started using dedicated threads.

Everything is OK until now... My only requirement now is to be able to get access to main Pyramid's registry from my sub-process and it's threads, and I don't really understand why I get a pointer to

ZCA global registry...  :(

Best regards,

Thierry

-- 

  https://www.ulthar.net -- http://pyams.readthedocs.io

--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/9a3e8a04-1a29-4f18-885e-dc3ac1c759b3n%40googlegroups.com.

[Thierry Florac]

Hi (again)!

I think it's finally OK!

I forgot ZCA hook is based on Pyramid's "current registry", which is bound to a thread-local variable, so that's why I have to push them on threadlocal manager in my threads "run" methods (and not in the constructor as I have tried before)!!

Maybe there is another better way to handle that anyway, so any advice is always welcome!

Best regards,

Thierry

-- 

  https://www.ulthar.net -- http://pyams.readthedocs.io

[Jonathan Vanasco]

Thierry,

That is what I mostly use Celery for (it's also used for generating Reports, ACME SSL Certificates and a few other things). A Pyramid Request will defer a job(s) to Celery via the "Transport Backend" and receive a task/messaging ID.  Subsequent requests will poll the Celery "Result Backend"  for the status of that job, based on the ID.  This way we show the "Still processing!" message to users.

Celery is run as a worker with multiple processes. The Celery task manager grabs a task off the queue, then does the resizing, uploads to S3, and notifies the Result Backend when complete.  I use Redis for Result and Transport.  I think there was once a ZeroMQ integration; there is definitely RabbitMQ integration, it's one of the more popular options.

[Thierry Florac]

Hi Jonathan,

Thank you for this description of Celery!

I'll try to have a look at it if I can get a little time, it seems to be a good replacement (probably more reliable!) of my own developments...

Best regards,

Thierry

-- 

  https://www.ulthar.net -- http://pyams.readthedocs.io

To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/5ac83ff1-747f-4df9-994c-ffa48f339013n%40googlegroups.com.

[Adam Groszer]

Hi,

Yeah celery is the way to go.

There's https://github.com/sontek/pyramid_celery to check out.

Some things to keep in mind is, you'll need a shared DB, good-old ZODB with filestorage is not enough, because more processes will need to have access.

Do not store valuable data in the celery queue, consider it ephemeral, go through a real DB.

[Theron Luhn]

Seems like a good time to plug my new project:  https://pypi.org/project/pyramid-tasks/

Pyramid and Celery are so dramatically different it’s a pain to integrate them, and any code shared between them can’t access anything Pyramid-specific.  Got tired of trying to make a square peg fit in a round hole, so I brought my entire Pyramid application over to my Celery worker.  Having a request object available in a Celery task is incredibly convenient, and my code is much tidier not having to tiptoe around the differences in environments.

To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/0a0d6985-c5b5-48b6-88f9-cce5d837c583n%40googlegroups.com.

[Jonathan Vanasco]

pyramid_tasks looks very promising! thanks for sharing this.  i may use it in the future.  i'll have to see if it is compatible with our setup.

our celery app is mostly tasks that have been migrated from a twisted daemon.  we already ran web spiders and social-media-account importers through twisted, so all the resizing stuff was originally pyramid creating database entries (messages) for twisted to process.  although we still use twisted for those tasks (and more!) migrating to celery for these simpler functions has been really productive and timesaving.  the code is cleaner and easier to maintain, and troubleshooting/bugfixing is so much simpler.

[Thierry Florac]

I also have to be able to schedule tasks, so I actually include the APScheduler package in my projects to make this scheduling.

Can pyramid-tasks be used also in this context?

To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/aaea1ab5-6d86-475c-8499-9435cdba411bn%40googlegroups.com.

[Theron Luhn]

I’m not familiar with APScheduler.  celery-beat seems to be the de facto way to run scheduled tasks in Celery.  https://docs.celeryproject.org/en/stable/userguide/periodic-tasks.html

celery-beat should be compatible with pyramid-tasks, but I haven’t tested or documented it yet.

I personally use AWS CloudWatch Events -> SNS -> HTTP to my app -> validate_aws_sns_message -> trigger Celery task.

To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/CAPX_VWC5EhcCj%2BRmr_ABMObbgYtTUYskaSR8TL0Nexm_1fCZ-A%40mail.gmail.com.

[Steve Piercy]

On 1/6/21 9:51 AM, Theron Luhn wrote:
> Seems like a good time to plug my new project:  https://pypi.org/project/pyramid-tasks/ <https://pypi.org/project/pyramid-tasks/>

>
> Pyramid and Celery are so dramatically different it’s a pain to integrate them, and any code shared between them can’t access anything Pyramid-specific.  Got tired of trying to make a square peg fit in a round hole, so I brought my entire Pyramid application over to my Celery worker.  Having a request object available in a Celery task is incredibly convenient, and my code is much tidier not having to tiptoe around the differences in environments.

I think this would be a good add-on to add to Extending Pyramid.

https://trypyramid.com/extending-pyramid.html

If you submit a PR, we'll merge it and deploy.

Also if you Tweet to @PylonsProject, I'll retweet it.

--steve

[Adam Groszer]

You can schedule any task with various options with celery:

https://docs.celeryproject.org/en/stable/userguide/calling.html#eta-and-countdown

[Theron Luhn]

Hi Thierry,

I’ve added support for periodic tasks in Pyramid Tasks.  https://github.com/luhn/pyramid-tasks#periodic-tasks

To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/627ab125-7fb3-429d-ab43-7340caf75b5en%40googlegroups.com.

[Thierry Florac]

Fine!

I'll have to take some time to have a look at Celery and at your package, to see how I can adapt my own scheduler framework to it...

Best regards,

Thierry

-- 

  https://www.ulthar.net -- http://pyams.readthedocs.io

To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/06818065-18BB-4203-B598-04962D58372F%40luhn.com.