Why you sometimes get Windows Permission Denied errors
Colin Brown
I have been struggling to solve why an occasional "Permission Denied" error
popped up from the following code fragment in one of my Win2K program
threads:
...
input_file = preprocess(raw_file)
os.system('third_party input_file output_file > error_file')
if os.path.exists(saved_file):
os.remove(saved_file)
os.rename(input_file,saved_file)
...
The error occurs on the os.rename(). The third_party executable was closing
input_file before terminating and anyway the subshell process has finished
before the os.rename is called! Most baffling.
With the aid of handle.exe from www.sysinternals.com I finally resolved the
problem. To see the problem at first hand try the following script:
import time, thread, os
def rm(file):
print 'delete x.x'
os.remove(file)
def wait():
if os.name == 'nt':
os.system('pause')
elif os.name == 'posix':
os.system('sleep 3')
print 'end wait'
print 'create x.x'
f = open('x.x','w')
thread.start_new_thread(wait,())
time.sleep(1)
print '\nclose x.x'
f.close()
rm('x.x')
Although this works fine on Linux, I get a Permission Denied error on
Windows. I surmise that an os.system call uses a C fork to create the
subshell - an exact duplicate of the current process environment including
OPEN FILE HANDLES. Because the os.system call has not returned before the
os.remove is called a "Permission Denied" error occurs. Quite simple really.
Now going back to my original problem, I had other threads doing os.system
calls. When one of these calls happens during the preprocess file write of
my above thread AND takes longer to complete than the os.system call in the
above thread then the file will still be open and the os.rename will return
the Permission Denied error.
What can be done to prevent the problem? I am open to suggestions here. My
solution was to use a threading.Semaphore() lock around all my file
operations to ensure they could only happen sequentially and also around my
os.system calls as below:
mainline>
lock=threading.Semaphore()
threads>
lock.acquire()
try:
file_operation()
finally:
lock.release()
subprocess_call>
def child(cmd):
os.system(cmd)
...
lock.acquire()
try:
thread.start_new_thread(child,(cmd,))
time.sleep(1)
finally:
lock.release()
...
This is definitely not elegant. If you know that your subprocess call will
always finish quickly then maybe you could just trap the error, delay and
retry until the operation suceeds.
I hope this has been helpful.
Colin Brown
PyNZ
Chris Tavares
"Colin Brown" <cbr...@metservice.com> wrote in message
news:3cc88d2f$1...@news.nz.asiaonline.net...
Not quite what's happening - Windows doesn't do fork. CreateProcess (the
underlying system call) will start a new process, but it doesn't do the
virtual memory copy thing that Unix does on a fork. However, there is
something called file handle inheritance, so that could be what's happening
here.
> Now going back to my original problem, I had other threads doing os.system
> calls. When one of these calls happens during the preprocess file write of
> my above thread AND takes longer to complete than the os.system call in
the
> above thread then the file will still be open and the os.rename will
return
> the Permission Denied error.
>
> What can be done to prevent the problem? I am open to suggestions here. My
> solution was to use a threading.Semaphore() lock around all my file
> operations to ensure they could only happen sequentially and also around
my
> os.system calls as below:
>
[... snip ... ]
Hmm... one possible solution would be to use win32process.CreateProcess
(part of the win32 extensions) instead of os.system. Depending on how
complex your command lines are (do you do a lot of io redirection?) that may
be either trivial or very difficult. That way you can make sure handles
don't get inherited, and you won't have this problem.
I suspect that the underlying C library system call just isn't set up to be
called in this manner - os.system isn't the Windows Way.
-Chris