announce: beta release of pyramid_forksafe, feedback requested

[Jonathan Vanasco]

I recently had to update our production deployment to address a forking issue with mongodb, and realized that our "post fork" cleanup routine was built directly against the uWSGI API -- creating some technical debt.

The solution was a new library that I'd appreciate feedback on: `pyramid_forksafe` 

The library allows you to write a postfork routine against the generic event `pyramid_forksafe.events.ApplicationPostFork` -- which is invoked with the application's current `registry`.  Changing between containers can then be addressed within the deployment configuration, so you don't need to touch the application code.

It appears to work fine with uWSGI and gunicorn so far.  I'm not quite sure how to handle the tests or other servers, so PRs/comments are welcome.

https://github.com/jvanasco/pyramid_forksafe

[Mikko Ohtamaa]

Hi

> I recently had to update our production deployment to address a forking issue with mongodb, and realized that our "post fork" cleanup routine was built directly against the uWSGI API -- creating some technical debt.

The problem I was facing (there might be others) that you cannot fork process after SQLAlchemy pool is set up.

Instead of relying post fork hooks to deal with this, I made sure that the pool is created lazily.

- Nothing gets created import time (Websauna boasts zero globals http://websauna.org/docs/ )

- You access SQLAlchemy through request.dbsession reified method

- When you access request.dbsession for the first time it sets up the pool, basically a normal reified lazy initialization pattern. This happens always after post-fork as it happens when HTTP request hits the server process for the first time.

Cheers,

Mikko

On 15 November 2016 at 21:42, Jonathan Vanasco <jvan...@gmail.com> wrote:

The solution was a new library that I'd appreciate feedback on: `pyramid_forksafe` 

The library allows you to write a postfork routine against the generic event `pyramid_forksafe.events.ApplicationPostFork` -- which is invoked with the application's current `registry`.  Changing between containers can then be addressed within the deployment configuration, so you don't need to touch the application code.

It appears to work fine with uWSGI and gunicorn so far.  I'm not quite sure how to handle the tests or other servers, so PRs/comments are welcome.

https://github.com/jvanasco/pyramid_forksafe

--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discuss+unsubscribe@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/ef8d9b12-8ebf-4a42-95a5-7a608a851049%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Mikko Ohtamaa
http://opensourcehacker.com
http://twitter.com/moo9000

[Jonathan Vanasco]

On Tuesday, November 15, 2016 at 2:48:02 PM UTC-5, Mikko Ohtamaa wrote:

 

The problem I was facing (there might be others) that you cannot fork process after SQLAlchemy pool is set up.

You can fork after the SqlAlchemy pool is set up -- you just have to call `.dispose()` on the sqlalchemy engine.  This is supported and documented  [http://docs.sqlalchemy.org/en/latest/core/connections.html#engine-disposal]

In our deployment, we communicate with PostgreSQL and MongoDB to pull out certain data before the fork.  If we did not do that, we'd have a bottleneck of 12+ processes from each node hitting the db server at once, and would not be able to take advantage of copy-on-write functionality of the deployment containers.  

There are also other libraries that require a hook at fork to work properly, such as PyCrypto's random number generator.