Follow-up: Win32/Posix file locking
Sheila King
topic of cross-platform file locking, I will share the code that I am
currently working with. I believe that I have solved my problem. I've
tested the code on both Win98 and Linux and it seems to work.
Ignacio suggested that I rename the module to something having to do
with Mutex objects, rather than file locking, and I have done so. The
idea is to acquire a lock on some Mutex before trying to write to a
datafile.
Here is the parent class:
-----------------------(begin MutexFile.py)--------------------------
'''MutexFile.py
wraps WinMutexFile.py and posixMutexFile.py to provide
a single interface for working with mutex files
across win32 and posix platforms.
Does not support fine-grained locking.
Recommended for getting a lock on a sentinel file
before trying to read/write data to a separate data file.
'''
import os
if os.name == 'nt':
from winMutexFile import mutexfile
elif os.name == 'posix':
from posixMutexFile import mutexfile
else:
raise ImportError, "MutexFile is not supported on your platform."
class MutexFile(mutexfile):
pass
-----------------------(end MutexFile.py)--------------------------
Below I present the two platform specific modules that implement the
file locking. I've used non-blocking locks with retries and time outs.
I've tried to handle a process crashing by coding a __del___ function
for the object. If anyone has feedback, I'm more than interested in
hearing it.
--
Sheila King
http://www.thinkspot.net/sheila/
http://www.k12groups.org/
----------------------(begin posixMutexFile.py)----------------------
''' posixMutexFile.py
class lockfile
supports the same function calls as
winLock.lockfile
export to LockFile.py: a wrapper module
around the win and posix mutexfiles
'''
import os, fcntl
from time import time
MAXTIME = 8 # number of secs to retry for a lock before timing out
class mutexfile:
def __init__(self, filename):
if os.access(filename, os.F_OK):
self.filename = filename
else:
errmssg = filename + \
" does not exist. Can't lock non-existent file."
raise IOError, errmssg
def __del__(self):
try:
self.unlock()
except:
pass
try:
self.f.close()
except:
pass
if hasattr(self, 'fd'):
del self.fd
def getReadLock(self):
start_time = time()
while time() - start_time < MAXTIME:
try:
self.f = open(self.filename, 'r')
self.fd = self.f.fileno()
fcntl.lockf(self.fd, fcntl.LOCK_SH | fcntl.LOCK_NB)
return 1
except:
self.f.close()
del self.fd
if not hasattr(self, 'fd'):
errmssg = self.filename + " temporarily unavailable"
raise IOError, errmssg
def getWriteLock(self):
start_time = time()
while time() - start_time < MAXTIME:
try:
self.f = open(self.filename, 'r+')
self.fd = self.f.fileno()
fcntl.lockf(self.fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
return 1
except:
self.f.close()
del self.fd
if not hasattr(self, 'fd'):
errmssg = self.filename + " temporarily unavailable"
raise IOError, errmssg
def unlock(self):
fcntl.lockf(self.fd, fcntl.LOCK_UN)
self.f.close()
del self.fd
def flock(self, flag):
'''flags are:
LOCK_UN - unlock
LOCK_SH - acquire a shared (or read) lock
LOCK_EX - acquire an exclusive (or write) lock
'''
if flag == 'LOCK_SH':
self.getReadLock()
elif flag == 'LOCK_EX':
self.getWriteLock()
elif flag == 'LOCK_UN':
self.unlock()
else:
errmssg = "The flag " + flag + \
" is not implemented for flock"
raise NotImplementedError, errmssg
----------------------(end posixMutexFile.py)----------------------
----------------------(begin winMutexFile.py)----------------------
''' winMutexFile.py
class lockfile
supports the same function calls as
posixLock.lockobject
export to lockobject.py: a wrapper module
around the win and posix lockobjects
'''
import os
from time import time
try:
import win32file
from win32con import GENERIC_READ, GENERIC_WRITE,\
FILE_SHARE_READ, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL
except ImportError, e:
print e
print "winMutexFile.py requires ActiveState.com Python win32
extensions"
MAXTIME = 8 # number of secs to retry for a lock before timing out
class mutexfile:
def __init__(self, filename):
if os.access(filename, os.F_OK):
self.filename = filename
else:
errmssg = filename + \
" does not exist. Can't lock non-existent file."
raise IOError, errmssg
def __del__(self):
try:
self.unlock()
except:
pass
try:
self.f.close()
except:
pass
if hasattr(self, 'fd'):
del self.fd
def getReadLock(self):
start_time = time()
while time() - start_time < MAXTIME:
try:
self.fd = win32file.CreateFile(self.filename,\
GENERIC_READ,\
FILE_SHARE_READ, None,\
OPEN_EXISTING,\
FILE_ATTRIBUTE_NORMAL, 0)
return 1
except:
pass
if not hasattr(self, 'fd'):
errmssg = self.filename + " temporarily unavailable"
raise IOError, errmssg
def getWriteLock(self):
start_time = time()
while time() - start_time < MAXTIME:
try:
self.fd = win32file.CreateFile(self.filename,\
GENERIC_READ,\
0, None, OPEN_EXISTING,\
FILE_ATTRIBUTE_NORMAL, 0)
return 1
except:
pass
if not hasattr(self,'fd'):
errmssg = self.filename + " temporarily unavailable"
raise IOError, errmssg
def unlock(self):
win32file.CloseHandle(self.fd)
del self.fd
def flock(self, flag):
'''flags are:
LOCK_UN - unlock
LOCK_SH - acquire a shared (or read) lock
LOCK_EX - acquire an exclusive (or write) lock
'''
if flag == 'LOCK_SH':
self.getReadLock()
elif flag == 'LOCK_EX':
self.getWriteLock()
elif flag == 'LOCK_UN':
self.unlock()
else:
errmssg = "The flag " + flag + \
" is not implemented for flock"
raise NotImplementedError, errmssg
----------------------(end winMutexFile.py)----------------------
Ignacio Vazquez-Abrams
> -----------------------(begin MutexFile.py)--------------------------
> ...
> import os
>
> if os.name == 'nt':
> from winMutexFile import mutexfile
> elif os.name == 'posix':
> from posixMutexFile import mutexfile
> else:
> raise ImportError, "MutexFile is not supported on your platform."
>
>
> class MutexFile(mutexfile):
> pass
> -----------------------(end MutexFile.py)--------------------------
Nice, clean interface. Can't get much better than this.
> ----------------------(begin posixMutexFile.py)----------------------
> ...
> import os, fcntl
> from time import time
>
> MAXTIME = 8 # number of secs to retry for a lock before timing out
Eight seconds might be a little long for CGI scripts, but it is writable so
that's fine.
> class mutexfile:
> def __init__(self, filename):
> if os.access(filename, os.F_OK):
> self.filename = filename
> else:
> errmssg = filename + \
> " does not exist. Can't lock non-existent file."
> raise IOError, errmssg
The only problem I can see here is if a program expects a certain filename and
someone goes and deletes it. You can't plan for every situation though, so
this risk is acceptable.
> def __del__(self):
> try:
> self.unlock()
> except:
> pass
> try:
> self.f.close()
> except:
> pass
> if hasattr(self, 'fd'):
> del self.fd
Once mutexfile is deleted, there are no longer any references to fd, so the
del can be dropped.
> def getReadLock(self):
> start_time = time()
> while time() - start_time < MAXTIME:
> try:
> self.f = open(self.filename, 'r')
> self.fd = self.f.fileno()
> fcntl.lockf(self.fd, fcntl.LOCK_SH | fcntl.LOCK_NB)
> return 1
> except:
> self.f.close()
> del self.fd
Ouch. I would put a short delay in here so that CPU usage doesn't go through
the roof by accident.
> if not hasattr(self, 'fd'):
> errmssg = self.filename + " temporarily unavailable"
> raise IOError, errmssg
>
> def getWriteLock(self):
> start_time = time()
> while time() - start_time < MAXTIME:
> try:
> self.f = open(self.filename, 'r+')
> self.fd = self.f.fileno()
> fcntl.lockf(self.fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
> return 1
> except:
> self.f.close()
> del self.fd
As above.
> if not hasattr(self, 'fd'):
> errmssg = self.filename + " temporarily unavailable"
> raise IOError, errmssg
>
> def unlock(self):
> fcntl.lockf(self.fd, fcntl.LOCK_UN)
> self.f.close()
> del self.fd
>
> def flock(self, flag):
> ...
> if flag == 'LOCK_SH':
> self.getReadLock()
> elif flag == 'LOCK_EX':
> self.getWriteLock()
> elif flag == 'LOCK_UN':
> self.unlock()
> else:
> errmssg = "The flag " + flag + \
> " is not implemented for flock"
> raise NotImplementedError, errmssg
> ----------------------(end posixMutexFile.py)----------------------
>
> ----------------------(begin winMutexFile.py)----------------------
> ...
> import os
> from time import time
>
> try:
> import win32file
> from win32con import GENERIC_READ, GENERIC_WRITE,\
> FILE_SHARE_READ, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL
> except ImportError, e:
> print e
> print "winMutexFile.py requires ActiveState.com Python win32
> extensions"
Hmm. I'm thinking that you need some sort of a raise here so that it stops.
Looks to me like Agent did a bit of a number on this file...
> return 1
> except:
> pass
As in the posixMutexFile module, a delay is probably called for here.
> if not hasattr(self, 'fd'):
> errmssg = self.filename + " temporarily unavailable"
> raise IOError, errmssg
>
> def getWriteLock(self):
> start_time = time()
> while time() - start_time < MAXTIME:
> try:
> self.fd = win32file.CreateFile(self.filename,\
> GENERIC_READ,\
> 0, None, OPEN_EXISTING,\
> FILE_ATTRIBUTE_NORMAL, 0)
> return 1
> except:
> pass
Another delay here.
> if not hasattr(self,'fd'):
> errmssg = self.filename + " temporarily unavailable"
> raise IOError, errmssg
>
> def unlock(self):
> win32file.CloseHandle(self.fd)
> del self.fd
>
> def flock(self, flag):
> ...
> if flag == 'LOCK_SH':
> self.getReadLock()
> elif flag == 'LOCK_EX':
> self.getWriteLock()
> elif flag == 'LOCK_UN':
> self.unlock()
> else:
> errmssg = "The flag " + flag + \
> " is not implemented for flock"
> raise NotImplementedError, errmssg
>
> ----------------------(end winMutexFile.py)----------------------
Looks good. Just two other things:
1) I'm wondering if assigning None to fd might be faster than deleting it and
recreating it.
2) The error messages use string concatenation when using a format string
might be cleaner.
Other than those few things it looks very good.
--
Ignacio Vazquez-Abrams <ign...@openservices.net>
Ignacio Vazquez-Abrams
> On Wed, 29 Aug 2001, Sheila King wrote:
> > MAXTIME = 8 # number of secs to retry for a lock before timing out
>
> Eight seconds might be a little long for CGI scripts, but it is writable so
> that's fine.
Hmm, that's not quite right, is it?
Okay, how about passing it as a parameter to __init__() with a default value
of 8? That would do it.
--
Ignacio Vazquez-Abrams <ign...@openservices.net>
Sheila King
<ign...@openservices.net> wrote in comp.lang.python in article
<mailman.999131050...@python.org>:
:On Wed, 29 Aug 2001, Sheila King wrote:
:> def getReadLock(self):
:> start_time = time()
:> while time() - start_time < MAXTIME:
:> try:
:> self.f = open(self.filename, 'r')
:> self.fd = self.f.fileno()
:> fcntl.lockf(self.fd, fcntl.LOCK_SH | fcntl.LOCK_NB)
:> return 1
:> except:
:> self.f.close()
:> del self.fd
:
:Ouch. I would put a short delay in here so that CPU usage doesn't go through
:the roof by accident.
What sort of time interval would you suggest? 0.1 seconds, perhaps? Or
is that too short?
:> ----------------------(begin winMutexFile.py)----------------------
:> ...
:> import os
:> from time import time
:>
:> try:
:> import win32file
:> from win32con import GENERIC_READ, GENERIC_WRITE,\
:> FILE_SHARE_READ, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL
:> except ImportError, e:
:> print e
:> print "winMutexFile.py requires ActiveState.com Python win32
:> extensions"
:
:Hmm. I'm thinking that you need some sort of a raise here so that it stops.
Yikes! Thanks for catching that. Duh.
:
:Looks good. Just two other things:
:
:1) I'm wondering if assigning None to fd might be faster than deleting it and
: recreating it.
:2) The error messages use string concatenation when using a format string
: might be cleaner.
:
:Other than those few things it looks very good.
Thanks for looking it over, and for the suggestions.
One other thing I've decided to change, is to move the file open out of
the loop in getReadLock in the posixMutexFile.py module. Somewhere in
the newsgroup someone posted in an article about a multi-thread
profiler, that file opening is expensive. So, I changed the way I
handled that a bit.
David Bolen
> What sort of time interval would you suggest? 0.1 seconds, perhaps? Or
> is that too short?
That should be more than enough - generally anything over 0 will at
least yield your timeslice to anything else that wants to run and
prevent you from hogging the CPU (even a time.sleep(0.01) under
Windows can be the difference between <1% CPU and 100% CPU).
Although since you're looping around I/O operations to the filesystem
it's unlikely you'd actually pin the CPU, but it's always a good
technique to follow anywhere you're looping anyway.
--
-- David
--
/-----------------------------------------------------------------------\
\ David Bolen \ E-mail: db...@fitlinxx.com /
| FitLinxx, Inc. \ Phone: (203) 708-5192 |
/ 860 Canal Street, Stamford, CT 06902 \ Fax: (203) 316-5150 \
\-----------------------------------------------------------------------/