Forking a process in a python cgi script

[Peter Haight]

I've got a python script that I'm using as a CGI and I want it to fork
off another process which runs for a long time. Meanwhile I want the
parent process to print some html and exit.

All the above works except for one thing. I can't get the parent process
to exit. (When it exits it shows up as a zombie on my ps listing).
Because of this the browser sits there waiting for more data until the
long-term process finally finishes which is exactly what I don't want to
happen.

At one point in time I had the fork() mixed up and the child was the one
that was exiting, but now it is definitely the parent which is becomming
a zombie. I think it is because the cgi program which launches my script
is waiting for the file descriptors to close and I somehow haven't
managed to close all of them.

I've read a bunch of stuff about closing file descriptors and I have
tried various methods for trying to ensure that the forked process has
closed stdin, stdout, and stderr, but none of that seems to help.

Besides using fork, I've also tried using os.system() and putting shell
scripts in the background and using os.execv().

[Marko Balabanovic]

This is what I use, works fine with Apache/PyApache:

SUCCESS = 0

# Finish CGI output before this point

sys.stdout.flush()

# Fork a child process

f = os.fork()
if f != 0:
# Terminate the parent
sys.stdout.close()
sys.exit(SUCCESS)
# PARENT DEAD

# CHILD CONTINUES FROM HERE
sys.stdout.close()

Marko

.Marko Balabanovic.......Department of Computer Science....................
.Stanford University Email: ma...@cs.stanford.edu Office: Gates 132 .
.Gates Building 1A Phone: 415 725 8783 Fax: 415 725 1449 .
.Stanford CA 94305-9010 Url: http://robotics.stanford.edu/people/marko .
.USA.......................................................................

[Guido van Rossum]

> > I've got a python script that I'm using as a CGI and I want it to fork
> > off another process which runs for a long time. Meanwhile I want the
> > parent process to print some html and exit.
> >
> > All the above works except for one thing. I can't get the parent process
> > to exit. (When it exits it shows up as a zombie on my ps listing).
> > Because of this the browser sits there waiting for more data until the
> > long-term process finally finishes which is exactly what I don't want to
> > happen.
>

> Forgive me if I'm barking up the wrong tree here, but another
> consideration might be that the parent hangs around because it has
> a child (or, conversely, that you want the child to be allowed
> to continue after the parent dies). Perhaps you are already doing
> this, but I believe you want to reassign the process group of the
> child so it can run after the parent goes away. I notice that I am
> currently just calling os.setpgrp() with no arguments, which I believe
> sets the process group to that of the "controlling terminal".

Yes, this is the wrong tree -- the http server doesn't care about its
grandchildren, but it does wait until it sees an EOF on the pipe from
the child. And that will never happen as long as any of the
descendants still has the write end of the pipe open.

--Guido van Rossum (home page: http://www.python.org/~guido/)

[Sue Williams]

>
> I've got a python script that I'm using as a CGI and I want it to fork
> off another process which runs for a long time. Meanwhile I want the
> parent process to print some html and exit.
>
> All the above works except for one thing. I can't get the parent process
> to exit. (When it exits it shows up as a zombie on my ps listing).
> Because of this the browser sits there waiting for more data until the
> long-term process finally finishes which is exactly what I don't want to
> happen.

Forgive me if I'm barking up the wrong tree here, but another
consideration might be that the parent hangs around because it has
a child (or, conversely, that you want the child to be allowed
to continue after the parent dies). Perhaps you are already doing
this, but I believe you want to reassign the process group of the
child so it can run after the parent goes away. I notice that I am
currently just calling os.setpgrp() with no arguments, which I believe
sets the process group to that of the "controlling terminal".

sue

[Peter Haight]

>--- Guido van Rossum:

>Yes, this is the wrong tree -- the http server doesn't care about its
>grandchildren, but it does wait until it sees an EOF on the pipe from
>the child. And that will never happen as long as any of the
>descendants still has the write end of the pipe open.

Exactly. I got it to work in C by doing this:

if (!fork())
{
close(0);
...
close(24);
while(1)
{
}
}

The child would sit around forever, but my web page would close nicely. I
don't think I have to close all 0-24, but I do have to close some fd above
14 somewhere. The cgi launching program does some duping that causes this.

If I try the same thing in Python (using os.close()), it doesn't work. I
have also tried sys.stdout.close(), but I think the dups in the parent are
messing that up or something. Any suggestions?

An aside:
I noticed something wierd when I was doing this. I was trying to close all
the fds in the parent before forking because I don't need to write anything
at all in the child. I was using os.close() inside of try/except:

lock_file = open('lock', 'w)
lock_file.write('locked')
lock_file.close()
try:
os.close(0)
os.close(1)
...
os.close(7)
except:
pass
os.remove('lock')

Whenever I do this, the 'lock' file never gets deleted so somehow they
Python script is exiting or something. If I comment out just 'os.close(7)'
the 'lock' file gets removed.

[Guido van Rossum]

> Exactly. I got it to work in C by doing this:
>
> if (!fork())
> {
> close(0);
> ...
> close(24);
> while(1)
> {
> }
> }
>
> The child would sit around forever, but my web page would close nicely. I
> don't think I have to close all 0-24, but I do have to close some fd above
> 14 somewhere. The cgi launching program does some duping that causes this.
>
> If I try the same thing in Python (using os.close()), it doesn't work. I
> have also tried sys.stdout.close(), but I think the dups in the parent are
> messing that up or something. Any suggestions?

Strange. I do this routinely. Are you sure you put a try-except
around the os.close() call? It will raise os.error if the fd is in
fact not open.

> An aside:
> I noticed something wierd when I was doing this. I was trying to close all
> the fds in the parent before forking because I don't need to write anything
> at all in the child. I was using os.close() inside of try/except:
>
> lock_file = open('lock', 'w)
> lock_file.write('locked')
> lock_file.close()
> try:
> os.close(0)
> os.close(1)
> ...
> os.close(7)
> except:
> pass
> os.remove('lock')
>
> Whenever I do this, the 'lock' file never gets deleted so somehow they
> Python script is exiting or something. If I comment out just 'os.close(7)'
> the 'lock' file gets removed.

Are you sure this is exactly your code? I would put each os.close()
call in a separate try-except. Note that your "unqualified" except
clause is dangerous -- I would use "except os.error", so that if
there's another error (maybe a typo!) it doesn't get masked.

[Peter Haight]

>--- Guido van Rossum:
>>--- Peter Haight:

>> If I try the same thing in Python (using os.close()), it doesn't work. I
>> have also tried sys.stdout.close(), but I think the dups in the parent are
>> messing that up or something. Any suggestions?
>
>Strange. I do this routinely. Are you sure you put a try-except
>around the os.close() call? It will raise os.error if the fd is in
>fact not open.

Yeah. That was catching me at first so I put a try-except around it.

>> lock_file = open('lock', 'w)
>> lock_file.write('locked')
>> lock_file.close()
>> try:
>> os.close(0)
>> os.close(1)
>> ...
>> os.close(7)
>> except:
>> pass
>> os.remove('lock')
>>
>> Whenever I do this, the 'lock' file never gets deleted so somehow they
>> Python script is exiting or something. If I comment out just 'os.close(7)'
>> the 'lock' file gets removed.
>
>Are you sure this is exactly your code? I would put each os.close()
>call in a separate try-except. Note that your "unqualified" except
>clause is dangerous -- I would use "except os.error", so that if
>there's another error (maybe a typo!) it doesn't get masked.

Yeah. The thing is that with the "unqualified" except, it should always
continue, right? Short of a syntax error, if it hits the try when
evaluating, it will pop to the except and continue no matter what, right? I
think it isn't a syntax error, because that check happens before it starts
evaluating (?) and I know we get to the file creation line.

Anyway, I'm going to do some more tests this weekend to try and get a
better idea of what is going on. I'm beginning to suspect that part of the
problem is in the code from the web server that actually launches the
script. I don't think that fd 7 should be open in my C program when it gets
called. I have the source for the cgi launching program so I should be able
to track it down.

While I'm at it, I will also see if I can come up with what's actually
happening when I hit that os.close(7) in Python.