[Web-SIG] A 'shutdown' function in WSGI

[Tarek Ziadé]

Hello

I need to be able to call a function when the web application shuts down (SIGTERM/SIGINT) -- the use case is to stop a background thread.

I am currently using signals because it seems to be the most clean way to do this. atexit is much trickier since you don't know when it's going to get called and you might try to call objects that were garbage collected unless you hack something to keep references alive.

But signals are also tricky beasts since you may compete with other code that are listening to them. For instance mod_wsgi don't like apps that have signal handlers.

Anyways, the bottom line is that the cleanest way to do this -- as per Chris McDonough idea, would be to introduce in the WSGI protocol a "shutdown" function the servers would be obligated to call before exiting.

I am not sure yet about its arguments, maybe a signum + frame or simply an exit code...

But how do you like the idea ?  That would solve for me the problem of having to deal differently here depending on if I am called with mod_wsgi or gunicorn or xxx


Cheers
Tarek

--
Tarek Ziadé | http://ziade.org

[Eric Larson]

CherryPy provides a bus that allows you to add events to the web server process. It is specified pretty clearly and CherryPy recently made it available as a standalone package, Magicbus (https://bitbucket.org/cherrypy/magicbus/overview). Specifically it allows you to send events on different signals the main server process might get. You can also use it for a general bus within the app server, but at its most basic level, the goal was to make the stop/start/restart events easy to hook into.

I've found it to be really helpful for managing processes and wrote a simple supervisor-ish app called Dad using it (http://bitbucket.org/elarson/dad).

HTH

Eric

--
Eric Larson

> _______________________________________________
> Web-SIG mailing list
> Web...@python.org (mailto:Web...@python.org)
> Web SIG: http://www.python.org/sigs/web-sig
> Unsubscribe: http://mail.python.org/mailman/options/web-sig/eric%40ionrock.org

_______________________________________________
Web-SIG mailing list
Web...@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: http://mail.python.org/mailman/options/web-sig/python-web-sig-garchive-9074%40googlegroups.com

[Tarek Ziadé]

On Mon, Feb 20, 2012 at 8:09 PM, Eric Larson <er...@ionrock.org> wrote:

CherryPy provides a bus that allows you to add events to the web server process. It is specified pretty clearly and CherryPy recently made it available as a standalone package, Magicbus (https://bitbucket.org/cherrypy/magicbus/overview). Specifically it allows you to send events on different signals the main server process might get. You can also use it for a general bus within the app server, but at its most basic level, the goal was to make the stop/start/restart events easy to hook into.

I've found it to be really helpful for managing processes and wrote a simple supervisor-ish app called Dad using it (http://bitbucket.org/elarson/dad).

Thanks for the pointer -- that looks pretty neat

I would be more interested though, in defining an extension to the WSGI standard

A rough example of what I am talking about:

If I take the wsgiref doc, here's an example of a minimal wsgi application (myapp.py):

from wsgiref.simple_server import make_server

def hello_world_app(environ, start_response):
    status = '200 OK' # HTTP Status
    headers = [('Content-type', 'text/plain')] 
    start_response(status, headers)
    return ["Hello World"]

def main():
 return make_server('', 8000, hello_world_app)

 

That module can be run by any web server out there that understands WSGI. For instance, with gunicorn I can do:

$ gunicorn myapp:main

What I am talking about is a second entry point for the shutdown - example:

from wsgiref.simple_server import make_server

def hello_world_app(environ, start_response):
    status = '200 OK' # HTTP Status
    headers = [('Content-type', 'text/plain')] 
    start_response(status, headers)
    return ["Hello World"]

def main():
   return make_server('', 8000, hello_world_app)


def shutdown():   # or maybe something else as an argument I don't know
   do_some_cleanup()
 

And point the shutdown callable to a web server:

$ gunicorn myapp:main myapp:shutdown

If this is defined in the WSGI standard it means any wsgi web server could call shutdown() , not only gunicorn

Cheers
Tarek

HTH

Eric

--
Eric Larson

On Monday, February 20, 2012 at 10:03 AM, Tarek Ziadé wrote:

> Hello
>
> I need to be able to call a function when the web application shuts down (SIGTERM/SIGINT) -- the use case is to stop a background thread.
>
> I am currently using signals because it seems to be the most clean way to do this. atexit is much trickier since you don't know when it's going to get called and you might try to call objects that were garbage collected unless you hack something to keep references alive.
>
> But signals are also tricky beasts since you may compete with other code that are listening to them. For instance mod_wsgi don't like apps that have signal handlers.
>
> Anyways, the bottom line is that the cleanest way to do this -- as per Chris McDonough idea, would be to introduce in the WSGI protocol a "shutdown" function the servers would be obligated to call before exiting.
>
> I am not sure yet about its arguments, maybe a signum + frame or simply an exit code...
>
> But how do you like the idea ? That would solve for me the problem of having to deal differently here depending on if I am called with mod_wsgi or gunicorn or xxx
>
>
> Cheers
> Tarek
>
> --
> Tarek Ziadé | http://ziade.org

> _______________________________________________
> Web-SIG mailing list
> Web...@python.org (mailto:Web...@python.org)
> Web SIG: http://www.python.org/sigs/web-sig
> Unsubscribe: http://mail.python.org/mailman/options/web-sig/eric%40ionrock.org

[Tarek Ziadé]

oops my examples were broken, should be:

def hello_world_app(environ, start_response):
    status = '200 OK' # HTTP Status
    headers = [('Content-type', 'text/plain')] 
    start_response(status, headers)
    return ["Hello World"]

def shutdown():   # or maybe something else as an argument I don't know
   do_some_cleanup()

and:

$ gunicorn myapp:hello_world_app myapp:shutdown



Cheers
Tarek

[PJ Eby]

The standard way to do this would be to define an "optional server extension" API supplied in the environ; for example, a 'x-wsgiorg.register_shutdown' function.  The wsgi.org wiki used to be the place to propose these sorts of things for standardization, but it appears to no longer be a wiki, so the mailing list is probably a good place to discuss such a proposal.

_______________________________________________
Web-SIG mailing list
Web...@python.org

Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe: http://mail.python.org/mailman/options/web-sig/pje%40telecommunity.com

[Chris McDonough]

On Mon, 2012-02-20 at 17:39 -0500, PJ Eby wrote:
> The standard way to do this would be to define an "optional server
> extension" API supplied in the environ; for example, a
> 'x-wsgiorg.register_shutdown' function.

Unlikely, AFACIT, as shutdown may happen when no request is active.
Even if this somehow happened to not be the case, asking the application
to put it in the environ is not useful, as the environ can't really be
relied on to retain values "up" the call stack.

- C

> Unsubscribe: http://mail.python.org/mailman/options/web-sig/chrism%40plope.com

_______________________________________________
Web-SIG mailing list
Web...@python.org
Web SIG: http://www.python.org/sigs/web-sig

Unsubscribe: http://mail.python.org/mailman/options/web-sig/python-web-sig-garchive-9074%40googlegroups.com

[Simon Sapin]

Le 21/02/2012 01:18, Chris McDonough a écrit :
> On Mon, 2012-02-20 at 17:39 -0500, PJ Eby wrote:
>> > The standard way to do this would be to define an "optional server
>> > extension" API supplied in the environ; for example, a
>> > 'x-wsgiorg.register_shutdown' function.
> Unlikely, AFACIT, as shutdown may happen when no request is active.
> Even if this somehow happened to not be the case, asking the application
> to put it in the environ is not useful, as the environ can't really be
> relied on to retain values "up" the call stack.

Hi,

I like environ['x-wsgiorg.register_shutdown']. It would work without
changes to WSGI itself.

I think that the idea is not to put your shutdown function in the
environment and hope it stays there "up" the stack, but to register it
by calling register_shutdown:

@environ.get('x-wsgiorg.register_shutdown', lambda f: f)
def do_cleanup():
pass

Also, a shutdown function would be used to clean up something that was
set up in a request. So if the server shuts down without having ever
served a request, there probably is nothing to clean up.

Regards,
--
Simon Sapin

[Graham Dumpleton]

On 21 February 2012 12:03, Simon Sapin <simon...@exyr.org> wrote:
> Le 21/02/2012 01:18, Chris McDonough a écrit :
>
>> On Mon, 2012-02-20 at 17:39 -0500, PJ Eby wrote:
>>>
>>> >  The standard way to do this would be to define an "optional server
>>> >  extension" API supplied in the environ; for example, a
>>> >  'x-wsgiorg.register_shutdown' function.
>>
>> Unlikely, AFACIT, as shutdown may happen when no request is active.
>> Even if this somehow happened to not be the case, asking the application
>> to put it in the environ is not useful, as the environ can't really be
>> relied on to retain values "up" the call stack.
>
>
> Hi,
>
> I like environ['x-wsgiorg.register_shutdown']. It would work without changes
> to WSGI itself.
>
> I think that the idea is not to put your shutdown function in the
> environment and hope it stays there "up" the stack, but to register it by
> calling register_shutdown:
>
> @environ.get('x-wsgiorg.register_shutdown', lambda f: f)
> def do_cleanup():
>    pass
>
> Also, a shutdown function would be used to clean up something that was set
> up in a request. So if the server shuts down without having ever served a
> request, there probably is nothing to clean up.

Using environ is not going to work it is supplied on a per request basis.

You would typically want an application scope cleanup handler to only
be registered once.

In this scheme you are relying on it being registered from within a
request scope.

To ensure that it is only registered once, the caller would need to
use a flag protected by a thread mutex to know whether should call a
second time, which is cumbersome.

If you don't do that you could end up registering a separate callback
for every single request that occurs and memory usage alone would blow
out just from recording them all.

Alternatively, you would have to require the underlying WSGI
server/adapter to weed out duplicates, but even if you do that, you
still waste the time of the per request scope registering it all the
time.

Even if you have a registration mechanism, especially with a WSGI
adapter riding on top of something else, how is the WSGI adapter going
to get notified to call them.

All you have therefore done is shift the problem of how it is
triggered somewhere else.

Overall the best chance of being able to do anything is relying on atexit.

You are though at the mercy of the WSGI hosting mechanism shutting
down the process and so the interpreter, in an orderly manner such
that atexit callbacks get called.

In Apache/mod_wsgi you get this guarantee, even in sub interpreters
where atexit callbacks wouldn't normally be called when they are
destroyed.

For uWSGI, atexit callbacks will not be called at the moment, by
Robert is making changes to it so you get a guarantee there as well.
It is possible he is only doing this though for case where main
interpreter is being used, as doing it for sub interpreters is a bit
fiddly.

Any pure Python WSGI servers shouldn't have issues so long as they
aren't force exiting the whole process and bypassing normal
interpreter destruction.

Graham

[PJ Eby]

2012/2/20 Chris McDonough <chr...@plope.com>

On Mon, 2012-02-20 at 17:39 -0500, PJ Eby wrote:
> The standard way to do this would be to define an "optional server
> extension" API supplied in the environ; for example, a
> 'x-wsgiorg.register_shutdown' function.

Unlikely, AFACIT, as shutdown may happen when no request is active.
Even if this somehow happened to not be the case, asking the application
to put it in the environ is not useful, as the environ can't really be
relied on to retain values "up" the call stack.

"Optional server extension APIs" are things that the server puts in the environ, not things the app puts there.  That's why it's 'register_shutdown', e.g. environ['x-wsgiorg.register_shutdown'](shutdown_function).  

[Chris McDonough]

On Mon, 2012-02-20 at 20:54 -0500, PJ Eby wrote:
> 2012/2/20 Chris McDonough <chr...@plope.com>
> On Mon, 2012-02-20 at 17:39 -0500, PJ Eby wrote:
> > The standard way to do this would be to define an "optional
> server
> > extension" API supplied in the environ; for example, a
> > 'x-wsgiorg.register_shutdown' function.
>
>
> Unlikely, AFACIT, as shutdown may happen when no request is
> active.
> Even if this somehow happened to not be the case, asking the
> application
> to put it in the environ is not useful, as the environ can't
> really be
> relied on to retain values "up" the call stack.
>
>
> "Optional server extension APIs" are things that the server puts in
> the environ, not things the app puts there. That's why it's
> 'register_shutdown', e.g.
> environ['x-wsgiorg.register_shutdown'](shutdown_function).

I get it now, but it's still not the right thing I don't think. Servers
shut down without issuing any requests at all.

- C

_______________________________________________
Web-SIG mailing list
Web...@python.org
Web SIG: http://www.python.org/sigs/web-sig

Unsubscribe: http://mail.python.org/mailman/options/web-sig/python-web-sig-garchive-9074%40googlegroups.com

[Tarek Ziadé]

2012/2/21 Chris McDonough <chr...@plope.com>

On Mon, 2012-02-20 at 20:54 -0500, PJ Eby wrote:
> 2012/2/20 Chris McDonough <chr...@plope.com>
>         On Mon, 2012-02-20 at 17:39 -0500, PJ Eby wrote:
>         > The standard way to do this would be to define an "optional
>         server
>         > extension" API supplied in the environ; for example, a
>         > 'x-wsgiorg.register_shutdown' function.
>
>
>         Unlikely, AFACIT, as shutdown may happen when no request is
>         active.
>         Even if this somehow happened to not be the case, asking the
>         application
>         to put it in the environ is not useful, as the environ can't
>         really be
>         relied on to retain values "up" the call stack.
>
>
> "Optional server extension APIs" are things that the server puts in
> the environ, not things the app puts there.  That's why it's
> 'register_shutdown', e.g.
> environ['x-wsgiorg.register_shutdown'](shutdown_function).

I get it now, but it's still not the right thing I don't think.  Servers
shut down without issuing any requests at all.

Yes, I also think shutting down the server is completely orthogonal to requests.

Maybe another option would be to call the application with the usual callable, but an "ending request" that's a signal for the application about being shut down.

When the app receives that very specific request, it would do the cleaning job. It sounds hackish but would work without changing the standard


 

- C

[Tarek Ziadé]

On Tue, Feb 21, 2012 at 2:39 AM, Graham Dumpleton <graham.d...@gmail.com> wrote:

...

Overall the best chance of being able to do anything is relying on atexit.

You are though at the mercy of the WSGI hosting mechanism shutting
down the process and so the interpreter, in an orderly manner such
that atexit callbacks get called.

In Apache/mod_wsgi you get this guarantee, even in sub interpreters
where atexit callbacks wouldn't normally be called when they are
destroyed.

For uWSGI, atexit callbacks will not be called at the moment, by
Robert is making changes to it so you get a guarantee there as well.
It is possible he is only doing this though for case where main
interpreter is being used, as doing it for sub interpreters is a bit
fiddly.

But I don't think you can guarantee that everything is still up in memory by the time atexit gets called,
so you can't really call cleanup code there.
 

Any pure Python WSGI servers shouldn't have issues so long as they
aren't force exiting the whole process and bypassing normal
interpreter destruction.

what do you mean by bypassing its destruction ?

Cheers
Tarek
 

 

Graham

_______________________________________________
Web-SIG mailing list
Web...@python.org
Web SIG: http://www.python.org/sigs/web-sig

Unsubscribe: http://mail.python.org/mailman/options/web-sig/ziade.tarek%40gmail.com

[Simon Sapin]

Le 21/02/2012 08:47, Tarek Ziadé a écrit :
> Yes, I also think shutting down the server is completely orthogonal to
> requests.

If the shutdown callback is next to the application and not registered
in a request, why not also have the symmetric "server start up" callback
that would not wait for a request? This would avoid workarounds like
Flask.before_first_request.

Both of these callbacks could be called once per process (aka. space
where requests share memory.)

Instead of having to provide two or three objects separately to a
server, how about making the callbacks attributes of the application
callable?

Regards,
--
Simon Sapin

[Tarek Ziadé]

On Tue, Feb 21, 2012 at 8:56 AM, Simon Sapin <simon...@exyr.org> wrote:

Le 21/02/2012 08:47, Tarek Ziadé a écrit :

Yes, I also think shutting down the server is completely orthogonal to
requests.

If the shutdown callback is next to the application and not registered in a request, why not also have the symmetric "server start up" callback that would not wait for a request? This would avoid workarounds like Flask.before_first_request.

Both of these callbacks could be called once per process (aka. space where requests share memory.)

Fair point,
 

Instead of having to provide two or three objects separately to a server, how about making the callbacks attributes of the application callable?

can you show us an example ?

Regards,
--
Simon Sapin

[Sylvain Hellegouarch]

Yes, I also think shutting down the server is completely orthogonal to requests.

CherryPy is using an approach where, indeed, both are orthogonal. Tools [1], which are similar to middlewares, are dealing with requests whereas the CherryPy engine [2] is the basis upon which the whole application and HTTP servers are running. The engine provides, amongst others, a clean way to stop and/or exit the whole stack independently from running requests. However, due to the way Robert Brewer designed it, it also means this can, if wanted, be performed from a request handler at any time. Note that the engine architecture, a bus, allows obviously for functions to subscribe at the time the engine shuts down in order to perform further operations that you would require.

Robert has also started the work of extracting the engine from CherryPy itself for other frameworks to rely on [3].

-- 
- Sylvain
http://www.defuze.org
http://twitter.com/lawouach

[1] http://docs.cherrypy.org/stable/concepts/tools.html

[2] http://docs.cherrypy.org/stable/concepts/engine.html

[3] https://bitbucket.org/cherrypy/magicbus

[Benoit Chesneau]

Hi Tarek,

In gunicorn we have the concept of hook which exactly does what you want
(on_exit hook). You can do this via the configuration file. And you can
do this for the other steps (reload, pre/post fork, pre/post request
...).

Anyway, I think there are 2 issues to solve with this shutdown thing
(and more) in a generic way:

1. how to make your application aware of the server capabilities (can it
or not handle shutdown)
2. how to pass such functions to the server.

There are also some others details like determining what is the context
needed by the application at this step.

Today the server is acting as a gateway so it just pass a request to an
application and return the response from the application. The
application by itself only lives in this context. Maybe, like some said,
if we want to give more info about that, we could imagine the application
object or function as a bus reacting on different contexts:

def application(environ, start_application):
pass

where the environ can be different depending on the context an action, we
could have:

http request {"wsgi.context": "http", ...}
shutdown {"wsgi.context"; "server", "action": "shutdown", ...}

smth like it. But how to know when the server can't pass the action
shutdown? Also what would be the environ at shutdown?

One other method, like the one described by Sylvain Hellegouarch, would
be adding some properties/attributes to the application object. So the
server would eventually check if they exists and eventually call them.
But again how to know when the server can't pass the action shutdown?
Though, I don't like to handle more than the application callable today,
I can see some problems when you have to handle server reload and such
things.

It seems important for me that the application knows about the server
capabilities if it want to handle such things like the `shutdown`. And
this step is solved by gunicorn & others by specifically handling a
configuration. But maybe we could just considers that a server always
handle the "reload", "shutdown", also pre/post "request" evenst and have
optionnals message passed to the application? (how do we handled process
context changes - if we re in a thread for ex - ?).

We could also imagine a web application package that provides hooks
handling depending on the server target... I quite like this idea.

Not sure what is the right way. What do you think?

- benoît

[Graham Dumpleton]

On 21 February 2012 18:53, Tarek Ziadé <ziade...@gmail.com> wrote:
>
>
> On Tue, Feb 21, 2012 at 2:39 AM, Graham Dumpleton
> <graham.d...@gmail.com> wrote:
>>
>> ...
>>
>> Overall the best chance of being able to do anything is relying on atexit.
>>
>> You are though at the mercy of the WSGI hosting mechanism shutting
>> down the process and so the interpreter, in an orderly manner such
>> that atexit callbacks get called.
>>
>> In Apache/mod_wsgi you get this guarantee, even in sub interpreters
>> where atexit callbacks wouldn't normally be called when they are
>> destroyed.
>>
>> For uWSGI, atexit callbacks will not be called at the moment, by
>> Robert is making changes to it so you get a guarantee there as well.
>> It is possible he is only doing this though for case where main
>> interpreter is being used, as doing it for sub interpreters is a bit
>> fiddly.
>>
>
> But I don't think you can guarantee that everything is still up in memory by
> the time atexit gets called,
> so you can't really call cleanup code there.

The only thing which is done prior to atexit callbacks being called is
waiting on threads which weren't marked as daemonised.

void
Py_Finalize(void)
{
PyInterpreterState *interp;
PyThreadState *tstate;

if (!initialized)
return;

wait_for_thread_shutdown();

/* The interpreter is still entirely intact at this point, and the
* exit funcs may be relying on that. In particular, if some thread
* or exit func is still waiting to do an import, the import machinery
* expects Py_IsInitialized() to return true. So don't say the
* interpreter is uninitialized until after the exit funcs have run.
* Note that Threading.py uses an exit func to do a join on all the
* threads created thru it, so this also protects pending imports in
* the threads created via Threading.
*/
call_sys_exitfunc();

...

>> Any pure Python WSGI servers shouldn't have issues so long as they
>> aren't force exiting the whole process and bypassing normal
>> interpreter destruction.
>
>
> what do you mean by bypassing its destruction ?

Non catchable signal from within process or from a distinct monitoring process.

One of this things I pointed out is being missed.

That is, a WSGI adapter may be running on top of another layer of
abstraction, such as FASTCGI for example, where the lower layer isn't
going to have any callback mechanism of its own to even notify the
WSGI layer to trigger registered cleanup callbacks.

This is why the only mechanism one can universally rely on is the
Python interpreters own atexit mechanism.

Graham
_______________________________________________
Web-SIG mailing list
Web...@python.org
Web SIG: http://www.python.org/sigs/web-sig

Unsubscribe: http://mail.python.org/mailman/options/web-sig/python-web-sig-garchive-9074%40googlegroups.com

[Simon Sapin]

Le 21/02/2012 09:23, Tarek Ziadé a écrit :
> Instead of having to provide two or three objects separately to a
> server, how about making the callbacks attributes of the application
> callable?
>
>
> can you show us an example ?

Proposal:

Function-based:

def startup():
return open_resource(something)

def shutdown(resource):
resource.close()

def application(environ, start_response):
# ...
return response_body

application.startup = startup
application.shutdown = shutdown

Class-based:

class App(object):
def startup(self):
return open_resource(something)

def shutdown(self, resource):
resource.close()

def __call__(self, environ, start_response):
# ...
return response_body

application = App()

The return value of startup() can be any python object and is opaque to
the server. It is passed as-is to shutdown()

startup() could take more parameters. Maybe the application (though can
we already have it as self for class-based or in a closure for
function-based)

[Graham Dumpleton]

You do realise you are just reinventing context managers?

With this 'application' do requests.

But then it was sort of suggested that was a bit too radical idea when
I have mentioned viewing it that way before. :-(

Graham

[Sylvain Hellegouarch]

One might wonder if having access to process management should be part of WSGI in the first place.

 

[Benoit Chesneau]

that's the thing. This is no more a gateway. Like I said in my
previous post, maybe having another spec describing a web app package
would do the trick?

- benoît

[Simon Sapin]

Le 21/02/2012 08:56, Simon Sapin a écrit :
> Both of these callbacks could be called once per process (aka. space
> where requests share memory.)

Sorry, that should be once per interpreter.

[Simon Sapin]

Le 21/02/2012 10:31, Graham Dumpleton a écrit :
> You do realise you are just reinventing context managers?
>
> With this 'application' do requests.

Indeed. I didn’t want to go too far from the initial "shutdown function"
proposal, but actual context managers would be better.

Le 21/02/2012 10:38, Benoit Chesneau a écrit :
>> One might wonder if having access to process management should be part of
>> > WSGI in the first place.
>> >
> that's the thing. This is no more a gateway. Like I said in my
> previous post, maybe having another spec describing a web app package
> would do the trick?

So a super-set of WSGI that is not just a gateway but also does process
management?

@contextlib.contextmanager
def super_application(server_infos):
with contextlib.closing(open_resource()) as resource:
wsgi_callable = make_app(server_infos, resource)
yield wsgi_callable

--
Simon Sapin

[Tarek Ziadé]

On Tue, Feb 21, 2012 at 10:24 AM, Graham Dumpleton <graham.d...@gmail.com> wrote:

...

> But I don't think you can guarantee that everything is still up in memory by
> the time atexit gets called,
> so you can't really call cleanup code there.

The only thing which is done prior to atexit callbacks being called is
waiting on threads which weren't marked as daemonised.

which can lead to completely lock the shutdown if a lib or the program has a thread with a loop that waits for a condition.

which it is not the case with signals, since you get a chance to properly stop everything beforehand.

>
> what do you mean by bypassing its destruction ?

> Non catchable signal from within process or from a distinct monitoring process.
> One of this things I pointed out is being missed.
> That is, a WSGI adapter may be running on top of another layer of
> abstraction, such as FASTCGI for example, where the lower layer isn't
> going to have any callback mechanism of its own to even notify the
> WSGI layer to trigger registered cleanup callbacks.
> This is why the only mechanism one can universally rely on is the
> Python interpreters own atexit mechanism.

I see.. but what I don't understand is the following: when the whole stack is shut down, the python process is being killed by *someone*.

And that someone, as far as I understand, is also able to send requests to the WSGI application.

So what makes it impossible to send a shutdown signal prior to killing the process ?

Sorry if I miss the obvious, I am probably over-simplifying things here :d

Cheers
Tarek

[Tarek Ziadé]

On Tue, Feb 21, 2012 at 11:07 AM, Simon Sapin <simon...@exyr.org> wrote:

...

 

So a super-set of WSGI that is not just a gateway but also does process management?


@contextlib.contextmanager
def super_application(server_infos):
   with contextlib.closing(open_resource()) as resource:
       wsgi_callable = make_app(server_infos, resource)
       yield wsgi_callable

I like this form a lot, but I think this is an implementation detail -- since we've not answered to the main question yet.

Here's my attempt to formulate how I understand the problem at this point

current assumptions/limitations:

-  the application can be shutdown without handling a request / which makes it orthogonal to the requests handling
-  the underlying code may use threads, making it unreliable to use atexit()
-  using signals may be problematic if some other code use it too -- for example the wsgi server itself

-> the cleanest way seems to ask the web server itself to ping the wsgi app.

problems:

- how can we declare a shutdown entry point in the application, the web server can use.
- how can this work with extra indirections (FASTCGI, etc)

leads:
- define startup/shutdown functions, declare them to the web server
- use the existing environ to send a 'shutdown request'
-

Cheers
Tarek

[Graham Dumpleton]

On 21 February 2012 21:41, Tarek Ziadé <ziade...@gmail.com> wrote:
>
>
> On Tue, Feb 21, 2012 at 10:24 AM, Graham Dumpleton
> <graham.d...@gmail.com> wrote:
>>
>> ...
>>
>> > But I don't think you can guarantee that everything is still up in
>> > memory by
>> > the time atexit gets called,
>> > so you can't really call cleanup code there.
>>
>> The only thing which is done prior to atexit callbacks being called is
>> waiting on threads which weren't marked as daemonised.
>
> which can lead to completely lock the shutdown if a lib or the program has a
> thread with a loop that waits for a condition.

In mod_wsgi at least there are fail safes such that background C
threads will force kill the process if such a lockup occurs on
shutdown.

> which it is not the case with signals, since you get a chance to properly
> stop everything beforehand.

Yes and no. For a signal handler to even be able to be triggered,
there must be Python code executing in the main thread that originally
created the main interpreter.

In an embedded system such as mod_wsgi, the main thread is never used
to handle requests and actually runs in C code blocked waiting for an
internal notification that process is being shutdown.

>> what do you mean by bypassing its destruction ?
>
>> Non catchable signal from within process or from a distinct monitoring
>> process.
>> One of this things I pointed out is being missed.
>> That is, a WSGI adapter may be running on top of another layer of
>> abstraction, such as FASTCGI for example, where the lower layer isn't
>> going to have any callback mechanism of its own to even notify the
>> WSGI layer to trigger registered cleanup callbacks.
>> This is why the only mechanism one can universally rely on is the
>> Python interpreters own atexit mechanism.
>
> I see.. but what I don't understand is the following: when the whole stack
> is shut down, the python process is being killed by *someone*.
>
> And that someone, as far as I understand, is also able to send requests to
> the WSGI application.
>
> So what makes it impossible to send a shutdown signal prior to killing the
> process ?

Is not impossible and in mod_wsgi at least a signal is used to
initiate shutdown, this coming either from itself in some cases, or
from Apache parent process in others. The signal handler then uses a
socketpair pipe to wake up the blocked main thread to begin shutdown
steps. Either way it is handled at C code level because can't rely on
Python level signal handlers to actually run.

To further complicate things, in a process with multiple sub
interpreters where would the Python signal handler even run. There is
no main thread running waiting to exit. It also can't just cause the
main Python interpreter to be exited. Simply exiting a main thread
even if it did exist wouldn't allow you to cleanup sub interpreters.

In short, embedded systems are going to be quite different to what you
are used to with pure WSGI servers. It is because it is doing all this
to ensure that reliable shutdown can occur that mod_wsgi ignores all
Python signal handler registrations by default.

That all said, technically mod_wsgi could on reception of its signal,
and if there was a registry of known WSGI applications, it could tell
them the process is being shutdown. Presence of sub interpreters makes
that a lot of fun, but would be doable.

Right now without such a registry of applications with enter/exit
methods as being discussed in this thread, the only way in mod_wsgi is
to rely on atexit.

[Graham Dumpleton]

On 21 February 2012 21:07, Simon Sapin <simon...@exyr.org> wrote:

> Le 21/02/2012 10:31, Graham Dumpleton a écrit :
>
>> You do realise you are just reinventing context managers?
>>
>> With this 'application' do requests.
>
> Indeed. I didn’t want to go too far from the initial "shutdown function"
> proposal, but actual context managers would be better.

FWIW, I have been playing with context managers in other ways to solve
per request resource cleanups issues as well. I will cover some of
what I have been doing with that in my State of WSGI 2 talk at PyCon
web summit.

I sort of wish this whole discussion could perhaps wait until the web
summit where after my talk I can perhaps discuss with interested
parties all the stuff I have been playing with around improving WSGI
rather than taking shots at little bits now when there is a lot more
to consider than just this.

Graham

[Antoine Pitrou]

Tarek Ziadé <ziade.tarek@...> writes:
>
>
> On Tue, Feb 21, 2012 at 10:24 AM, Graham Dumpleton
<graham.d...@gmail.com> wrote:
> ...
> > But I don't think you can guarantee that everything is still up in memory by
> > the time atexit gets called,
> > so you can't really call cleanup code there.
> The only thing which is done prior to atexit callbacks being called is
> waiting on threads which weren't marked as daemonised.
>
>
> which can lead to completely lock the shutdown if a lib or the program has a

> thread with a loop that waits for a condition.which it is not the case with

> signals, since you get a chance to properly stop everything beforehand.

That's a buggy lib or program. This has nothing to do with WSGI really. The
snippet Graham showed is run at any interpreter shutdown, even when you simply
run "python" in your shell.

Regards

Antoine.

[Tarek Ziadé]

On Tue, Feb 21, 2012 at 1:43 PM, Antoine Pitrou <soli...@pitrou.net> wrote:

Tarek Ziadé <ziade.tarek@...> writes:
>
>
> On Tue, Feb 21, 2012 at 10:24 AM, Graham Dumpleton
<graham.d...@gmail.com> wrote:
> ...
> > But I don't think you can guarantee that everything is still up in memory by
> > the time atexit gets called,
> > so you can't really call cleanup code there.
> The only thing which is done prior to atexit callbacks being called is
> waiting on threads which weren't marked as daemonised.
>
>
> which can lead to completely lock the shutdown if a lib or the program has a

> thread with a loop that waits for a condition.which it is not the case with

> signals, since you get a chance to properly stop everything beforehand.

That's a buggy lib or program. This has nothing to do with WSGI really.

No, that has to do with : please let me clean my program before you try to kill it because I can't use signals :)

 

The
snippet Graham showed is run at any interpreter shutdown, even when you simply
run "python" in your shell.

here's a very simple demo: http://tarek.pastebin.mozilla.org/1489505

Run it with plain python, and try to ctrl-C it. You won't reach atexit and will get locked.

(here: python 2.7 / mac os)

If you use signals instead of atexit, you'll have it working.

And this pattern (a thread in the background) is pretty common -- unless I am missing something here


Cheers
Tarek
 

Regards

Antoine.

_______________________________________________
Web-SIG mailing list
Web...@python.org
Web SIG: http://www.python.org/sigs/web-sig

Unsubscribe: http://mail.python.org/mailman/options/web-sig/ziade.tarek%40gmail.com

[Tarek Ziadé]

On Tue, Feb 21, 2012 at 2:46 PM, Tarek Ziadé <ziade...@gmail.com> wrote:

 

here's a very simple demo: http://tarek.pastebin.mozilla.org/1489505

There are two typos but the effect remains the same since you are locked before you reach those lines:

- atexit call worker.stop() instead  of worker.join()
- in worker.stop(), it calls worker.join() instead of worker.join(self)

[Antoine Pitrou]

Tarek Ziadé <ziade.tarek@...> writes:
>
> here's a very simple demo: http://tarek.pastebin.mozilla.org/1489505
>
> There are two typos but the effect remains the same since you are locked
> before you reach those lines:
> - atexit call worker.stop() instead 
> of worker.join()
> - in worker.stop(), it calls worker.join() instead of
> worker.join(self)

I find your example convincing. I think that's worth fixing in Python (e.g. by
offering an atexit() method on Thread objects). Perhaps you can open a bug?

Regards

Antoine.

_______________________________________________
Web-SIG mailing list
Web...@python.org
Web SIG: http://www.python.org/sigs/web-sig

Unsubscribe: http://mail.python.org/mailman/options/web-sig/python-web-sig-garchive-9074%40googlegroups.com

[Tarek Ziadé]

On Tue, Feb 21, 2012 at 3:02 PM, Antoine Pitrou <soli...@pitrou.net> wrote:

...

I find your example convincing. I think that's worth fixing in Python (e.g. by
offering an atexit() method on Thread objects). Perhaps you can open a bug?

Sure yeah --  Notice that I think it's still a good idea to provide the shutdown function in WSGI, because it gives the full control to the web server on the ordering of events.

Cheers
Tarek

 

Regards

Antoine.


_______________________________________________
Web-SIG mailing list
Web...@python.org
Web SIG: http://www.python.org/sigs/web-sig

Unsubscribe: http://mail.python.org/mailman/options/web-sig/ziade.tarek%40gmail.com

[Tarek Ziadé]

for the record: http://bugs.python.org/issue14073

[Simon Sapin]

Simon Sapin <simon...@exyr.org> a écrit :

>Both of these callbacks could be called once per process (aka. space
>where requests share memory.)

Sorry, that should be once per interpreter.

[Graham Dumpleton]

If you want to be able to control a thread like that from an atexit
callback, you need to create the thread as daemonised. Ie.
setDaemon(True) call on thread.

By default a thread will actually inherit the daemon flag from the
parent. For a command line Python where thread created from main
thread it will not be daemonised and thus why the thread will be
waited upon on shutdown prior to atexit being called.

If you ran the same code in mod_wsgi, my memory is that the thread
will actually inherit as being daemonised because request handler in
mod_wsgi, from which import is trigger, are notionally daemonised.

Thus the code should work in mod_wsgi. Even so, to be portable, if
wanting to manipulate thread from atexit, make it daemonised.

Example of background threads in mod_wsgi at:

http://code.google.com/p/modwsgi/wiki/ReloadingSourceCode#Monitoring_For_Code_Changes

shows use of setDaemon().

Graham

> http://mail.python.org/mailman/options/web-sig/graham.dumpleton%40gmail.com

>
_______________________________________________
Web-SIG mailing list
Web...@python.org
Web SIG: http://www.python.org/sigs/web-sig

Unsubscribe: http://mail.python.org/mailman/options/web-sig/python-web-sig-garchive-9074%40googlegroups.com

[Chris McDonough]

On Wed, 2012-02-22 at 09:06 +1100, Graham Dumpleton wrote:
> If you want to be able to control a thread like that from an atexit
> callback, you need to create the thread as daemonised. Ie.
> setDaemon(True) call on thread.
>
> By default a thread will actually inherit the daemon flag from the
> parent. For a command line Python where thread created from main
> thread it will not be daemonised and thus why the thread will be
> waited upon on shutdown prior to atexit being called.
>
> If you ran the same code in mod_wsgi, my memory is that the thread
> will actually inherit as being daemonised because request handler in
> mod_wsgi, from which import is trigger, are notionally daemonised.
>
> Thus the code should work in mod_wsgi. Even so, to be portable, if
> wanting to manipulate thread from atexit, make it daemonised.
>
> Example of background threads in mod_wsgi at:
>
> http://code.google.com/p/modwsgi/wiki/ReloadingSourceCode#Monitoring_For_Code_Changes
>
> shows use of setDaemon().
>
> Graham

I've read all the messages in this thread and the traffic on the bug
entry at http://bugs.python.org/issue14073 but I'm still not sure what
to tell people who want to invoke code "at shutdown".

Do we tell them to use atexit? If so, are we saying that atexit is
sufficient for all user-defined shutdown code that needs to run save for
code that needs to stop threads?

Is it sufficient to define "shutdown" as "when the process associated
with the application exits"? It still seems to not necessarily be
directly correlated.

- C