Exception in .wsgi file

[YKdvd]

We have a Flask-based mod_wsgi (4.6.4) app, apache 2.4.18 on Ubuntu 16.04LTS, using WSGIDaemonProcess, where WSGIScriptAlias points to the little ourapp.wsgi file which contains things like this:

sys.path.insert(0, "/srv/ourapp/modules")   # a local path with some needed modules (but not some_good_module)
import some_good_module  # this is from a path already on sys.path via "python-path", and is on an autofs mount
# if the above fails, an ImportError exception will be thrown, and the next line doesn't execute.
application = make_outflaskapp()

We've been have some trouble where the import line is failing on rare occasions.  We think we have some sort of flakiness where the "some_good_module" resides on a mounted drive via autofs, which occasionally dismounts, and it isn't a mod_wsgi thing.  But I'm curious what happens when ourapp.wsgi throws an exception like this before constructing the WSGI app and creating the "application" variable, and mod_wsgi takes over after the exception is thrown.  It looks like it keeps that process it has created, with the Python interpreter that has been spun up in it, and will continue to try and execute the "ourapp.wsgi" code in it at later times?  So the "sys.path.insert()" lines and anything above the exception point will be executed each time?  

We've caught subsequent executions of the code in that same process ID, with those path insertions stacked up, and eventually even when the filesystem mount causing the ImportError seems to be available again, the import still fails.  I didn't think Python would cache a failed import name in any way, and would be looking to disk for the module file each time, and should succeed if the mount was there, but I'm still working on that, so the state of the mod_wsgi process is of interest.

There isn't any option that can be set to cause mod_wsgi discard the process completely if it fails to get a WSGI app constructed in this case, instead of reusing it?

[Graham Dumpleton]

First up, you are better off also adding "/srv/ourapp/modules" using "python-path" option of WSGIDaemonProcess if you can. That will avoid double insertion problem. Otherwise if have to then guard it with a check to see whether is in "sys.path" if this becomes a big problem.

Next, for the case where a wsgi.py file isn't reentrant, that is, it cannot be imported more than once, and if it fails during import will keep failing if imported again, you should set the "startup-timeout" option on WSGIDaemonProcess". This is a timeout value which when triggered, due to the wsgi.py file not being able to be imported after multiple times, will cause the daemon process to be restarted. Presuming your auto mount then works okay, the application should keep going.

This 'startup-timeout' is necessary with some Python web frameworks (such as Django), as initialisation of the web framework itself is not reentrant (used to be for Django but they changed it). This means if you have a transient failure during initialisation, such as a database not being available, the startup timeout provides a way for the daemon process to be started over, and if transient condition which caused the failure is gone, then things should be okay from that point.

Graham

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/e2b193c2-9904-416b-8f91-63dab664a1ado%40googlegroups.com.

[YKdvd]

So if I read that right, mod_wsgi will try reuse the failed process and its Python interpreter for the wsgi code, but will discard it and start a new process after "startup-timeout" seconds (if specified)?  That should help mitigate our problem.  I'm still trying to figure out why the process should be "poisoned" somehow on the import once the mount is back (if that's what's happening), but getting fresh processes after 15 or 30 seconds should be helpful.

Is there a use-case for having "startup-timeout=-1" always create a new process after failure, for development or debugging, if one is willing to accept the extra overhead of process/python startup?  The negative number could be a failure count, instead of seconds, if one wanted it to be failure-count based rather than time-based?

Thanks for the pointer to startup-timeout.

Graham

To unsubscribe from this group and stop receiving emails from it, send an email to mod...@googlegroups.com.

[Graham Dumpleton]

If development only, you could set maximum-requests=1, but this will also restart if was okay. Performance would be as bad as CGI scripts.

The alternative is to do:

    import os

    import signal

    try:

        ... do initialisation

    except Exception:

        os.kill(os.getpid(), signal.SIGTERM)

This was the workaround before startup-timeout was added.

As to why the processes is poisoned, it is simply that some code may not like being reloaded as it performs an action which should only be done once in the process. Would have to see the exact error and traceback you were getting to speculate more.

Graham

To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/51a28fd0-c16a-4598-94ca-c9634cf79ff0o%40googlegroups.com.