Background Threads

[EricHolmberg]

I would like to have a background thread handle tasks such as sending
notification emails and doing background integrity checks on the
database. What is the best way to handle this? Possible solutions
include:

1. Start a thread in load_environment
2. Run a separate daemon process


Thread in load_environment
---------------------------------------
This works great for the stand-alone Pylons server daemon, but I will
be using mod_wsgi in the production system which results in multiple
instantiations of the Pylons app, so multiple threads will be
created. Should I just use this method and configure mod_wsgi to only
start one instance of Pylons?

Run a separate daemon process
-----------------------------------------------
This takes advantage of process isolation, but it will require
checking to ensure that the daemon process is still running. Is there
a way to have Apache do this background check? Any other solutions?

Thanks,

Eric

[Devin Torres]

I say run a separate daemon process and use supervisord to ensure it
can recover from a crash.

In fact, I recommend using supervisord for everything you ever deploy.

-Devin

[EricHolmberg]

> I say run a separate daemon process and use supervisord to ensure it
> can recover from a crash.
>
> In fact, I recommend using supervisord for everything you ever deploy.
>

Excellent - thanks for the reminder! I remember seeing a mention in
Linux Journal a while back, but never followed up on it. It looks
perfect for a Linux deployment (which is my immediate need).

Do you know of anything comparable for Windows? I have a few Windows-
only services that are buggy that could benefit from a similar
supervisor.

[Garland, Ken R]

I use a custom daemon for some of my deployments.

For windows I would look into creating a service to monitor things (services.msc)

[Graham Dumpleton]

If using mod_wsgi you can do this by configuring a daemon process
group with a single process, specifying that a specific script file
should be run when that process is started, and then simply never
directing any requests to that process.

Thus, the daemon process definition might look like:

WSGIDaemonProcess mybackgroundtasks processes=1 threads=1

We set threads=1 simply so that minimum number of handler threads are
created in pool. The threads will never actually be used as we will
never direct any requests to it.

To have specific code executed when this daemon process is started,
define:

WSGIImportScript /some/path/script.wsgi process-
group=mybackgroundtasks application-group=%{GLOBAL}

In other words, when background tasks daemon process is started, in
the main Python interpreter context load the Python code in /some/path/
script.wsgi. Such code will be executed as if code file was a module
and imported.

To execute a background task, that script file would load threading
module, create a thread and run it with your code. A background thread
is still used as script must return.

And that is it.

Because this is a managed mod_wsgi daemon process, there is no need
for using separate supervisor as Apache/mod_wsgi will manage process
for you. As a result, the process will be started, stopped and
restarted as Apache is triggered to do the same. If the process
crashes for some reason, Apache will restart it automatically.

Note that the above is only necessary if running your WSGI application
if mod_wsgi embedded mode or a multiprocess daemon mode configuration.
That is, where multiple processes. If you were running your WSGI
application in single multithreaded daemon process, you could just the
above in same process as you were using for web application.

Even if WSGI application running in embedded mode or multiprocess
daemon mode configuration, you could still send subset of URLs to this
single background task process. In other words, the background task
process could itself be a mini web application, thus allowing you to
control through the web the background tasks it is performing.

Thus you might have:

WSGIScriptAlias /tasks /some/path/taskapp.wsgi
WSGIScriptAlias / /some/path/mainapp.wsgi

<Location /tasks>
WSGIProcessGroup mybackgroundtasks
</Location>

In summary, you can do what you want with just mod_wsgi, you do not
need to separately exec a back end process managed under supervisor or
similar system.

Graham

[Jonathan Vanasco]

i've done two things to handle this in the past:

1- CRON
I used to just write a simple script to handle it all, then have cron
run at 1,5,10,15,30,60 minute intervals or whatever.. and use some PID
files to make sure I don't have an overlap in execution

2- Twisted
Now I often just run a twisted daemon on the server, that handles all
that kind of stuff. It has a scheduling functionality built in, plus
semaphores -- so I can keep all the process logic in python.

[EricHolmberg]

Excellent suggestions, guys! I'm impressed.

I originally just had a simple issue to running the background task to
complement the Pylons application, but looking over everybody's
suggestions, I can see that I can refactor a lot of other systems that
I deal with to make them easier to maintain and get rid of some of the
nasty issues. It looks like all of these suggestions will be able to
carve out some little niches in the various tasks that I have to do.

CRON jobs
---------
I'm currently using these (and the scheduler in Windows) and the
addition of monitoring code, overlap prevention code, polling response
vs. performance tradeoffs, etc all start to create a tangled web of
scheduling times and code. It's fine when everything is running well,
but if I need to shut down something for a while or a dependency task
fails, it's easy to make a mistake and cause unnecessary grief.

Supervisord
-----------
This looks great for replacing all of the inter-dependent Python
scripts / native applications that I have running under CRON. For the
web-related services, I see some of the issues of the CRON jobs where
I have dependencies between the WSGI web application and the scheduled
scripts, so while it would work fine for the web apps, I think that
I'll use the mod_wsgi approach for the background web scripts.
However, I like the idea of adding the monitoring scripts under
supervisord that verify that the web apps are behaving properly since
I want those isolated.

mod_wsgi
--------
First of all, Graham, you went above and beyond on the details. Thank
you very much! This is a great solution for running the web-related
tasks and it has the advantages that 1) I don't have to modify much of
my existing code and 2) that my web app and the background threads
will all be configured in one place (the Apache configuration file).

Twisted Daemon
--------------
This is another great solution and I'm going to dig through the
__Twisted Network Programming Essentials__ book to see if I can use
this for some future refactoring of the existing Python-based CRON
scripts, etc.

These suggestions are excellent, I'll add a to-do item to try and post
some of these ideas to the Pylons HQ wiki.

-Eric

[Graham Dumpleton]

On Jun 22, 3:41 am, EricHolmberg <eric.omnicuri...@gmail.com> wrote:
> mod_wsgi
> --------
> First of all, Graham, you went above and beyond on the details.  Thank
> you very much!  This is a great solution for running the web-related
> tasks and it has the advantages that 1) I don't have to modify much of
> my existing code and 2) that my web app and the background threads
> will all be configured in one place (the Apache configuration file).

Except that you mention Windows and mod_wsgi daemon mode isn't
available on Windows.

Graham

[EricHolmberg]

> Except that you mention Windows and mod_wsgi daemon mode isn't
> available on Windows.
>
> Graham

A shame that mode isn't available under Windows. For my immediate
problem, I am running Linux so it isn't an issue. Sorry for the
confusion -- I have several ongoing projects running on both Linux and
Windows machines.

-Eric