Code to spawn a new, independent process

[Corey]

Hey everyone,

I've seen references and questions in previous posts about how to spawn
a process to do some hefty processing. I had that need (processing of
1.6 million records at a time).

I created this code with help from other places on the net and it seems
to work pretty well. I'm putting it out there to get some further
testing and hope it is useful for someone else.

Usage is after the file.

Corey

daemonize.py:

#! /usr/bin/env python

import sys, os, time, tempfile
from signal import SIGTERM

def daemonize(stdout='/dev/null', stderr=None, stdin='/dev/null',
pidfile=None, startmsg = 'started with pid %s' ):
'''
This forks the current process into a daemon.
The stdin, stdout, and stderr arguments are file names that
will be opened and be used to replace the standard file
descriptors
in sys.stdin, sys.stdout, and sys.stderr.
These arguments are optional and default to /dev/null.
Note that stderr is opened unbuffered, so
if it shares a file with stdout then interleaved output
may not appear in the order that you expect.
'''
# Do first fork.
try:
pid = os.fork()
if pid > 0: sys.exit(0) # Exit first parent.
except OSError, e:
sys.stderr.write("fork #1 failed: (%d) %s\n" % (e.errno,
e.strerror))
sys.exit(1)

# Decouple from parent environment.
os.chdir("/")
os.umask(0)
os.setsid()

# Do second fork.
try:
pid = os.fork()
if pid > 0: sys.exit(0) # Exit second parent.
except OSError, e:
sys.stderr.write("fork #2 failed: (%d) %s\n" % (e.errno,
e.strerror))
sys.exit(1)

# Open file descriptors and print start message
if not stderr: stderr = stdout
si = file(stdin, 'r')
so = file(stdout, 'a+')
se = file(stderr, 'a+', 0)

pid = str(os.getpid())
if startmsg:
sys.stderr.write("\n%s\n" % startmsg % pid)
sys.stderr.flush()
if pidfile: file(pidfile,'w+').write("%s\n" % pid)

# Redirect standard file descriptors.
os.dup2(si.fileno(), sys.stdin.fileno())
os.dup2(so.fileno(), sys.stdout.fileno())
os.dup2(se.fileno(), sys.stderr.fileno())

def get_pid(pidfile):
"""
Tries to get the current pid, given a pid file.

Use this function to make sure a copy of this process isn't
already running.

@param String pidfile The path to the pid file
@return Integer or None
"""
try:
pf = file(pidfile,'r')
pid = int(pf.read().strip())
pf.close()
except StandardError:
pid = None
if pid:
mess = "Start aborted since pid file '%s' exists.\n"
sys.stderr.write(mess % pidfile)
return pid

if __name__ == "__main__":
pid_file = '/tmp/%s.pid' % sys.argv[2]

#########################
# We don't want two of the same processes happening at the same
time for any reason
#########################
if get_pid(pid_file):
f = open('/tmp/debuging.txt')
f.write('Exiting because a pid file already exists.')
f.close()
sys.exit(1)

#########################
# Split ourselves from the parent process
#########################
daemonize(stdout='/tmp/%s-out.txt' % sys.argv[2],
stderr='/tmp/%s-err.txt' % sys.argv[2], startmsg='', pidfile=pid_file)
print "daemonizing completed.\n"

#########################
# Generate the command to run
#########################

# We need to add the directory containing the project.
# This command may be crude, but it gets the directory of the
directory of the script.
thePath = os.path.dirname(os.path.dirname(sys.argv[0]))
print "Adding %s to sys.path.\n" % thePath
sys.path.append(thePath)

print "Importing %s.\n" % sys.argv[1]
m = __import__(sys.argv[1], globals(), locals(), '*')

command = r'm.%s(' % sys.argv[2]
for item in range(3,len(sys.argv)):
arg = sys.argv[item].strip()

# If we are passing tuples or lists, we don't want them quoted
if arg[:1] == '(' or arg[:1] == '[':
command += r' %s,' % arg
else:
command += r' "%s",' % arg

command = command[:-1] + r')'

########################
# Run the command
########################
print "running %s" % command
eval(command)

########################
# Delete the pid file
########################
os.remove(pid_file)
sys.exit(0)

##### End daemonize.py

In your view you put in code similar to this:

# To import the file without creating a long delay, we are going to
split off the script
# ...daemonize it. The daemonize script will then call
importfile.start_import(theTempFile)
# The daemonizer.py script is in the main project folder...
daemonizer = "%s%s" % (settings.ROOT_DIR,'daemonize.py')
module = "pennysaver.distribution.importfile"
function = "start_import"
os.spawnvp(os.P_NOWAIT, daemonizer, [daemonizer, module, function,
theTempPath])

return render_to_response('import_progress.html')

[oefe]

Corey wrote:
> Hey everyone,
>
> I've seen references and questions in previous posts about how to spawn
> a process to do some hefty processing. I had that need (processing of
> 1.6 million records at a time).
>
> I created this code with help from other places on the net and it seems
> to work pretty well. I'm putting it out there to get some further
> testing and hope it is useful for someone else.

Hi corey!

why don't you simply use python's subprocess module?

ciao
Martina

[Corey Oordt]

Martina,

Mostly because I'm new to python and in my research no one mentioned it!

You are right, this does look like a better fit.

Live and learn!

Corey