Question: posix fcntl module
Sheila King
systems.
I was checking out the man pages on the Linux system that I use (my
webhost), and under man flock(2) it mentions (among other things):
NOTES
flock(2) does not lock files over NFS. Use fcntl(2)
instead: that does work over NFS, given a sufficiently
recent version of Linux and a server which supports locking.
flock(2) and fcntl(2) locks have different semantics with
respect to forked processes and dup(2).
Python docs for the fcntl module state:
lockf(fd, operation, [len, [start, [whence]]])
This is essentially a wrapper around the fcntl() locking calls.
fd is the file descriptor of the file to lock or unlock, and...
Am I correct, then, in interpreting these together, to mean, that I will
get greater portability/control if I use fcntl.lockf than if I use
fcntl.flock? Because, as I read this, fcntl.lockf may work in situations
where fcntl.flock does not (especially over NFS?).
Please advise. Thanks,
--
Sheila King
http://www.thinkspot.net/sheila/
http://www.k12groups.org/
Ignacio Vazquez-Abrams
Yes. Funny thing is, that if the OS doesn't have flock(), it'll use the
lockf()-equivalent fcntl anyways. If you didn't understand that, then don't
worry about it :)
--
Ignacio Vazquez-Abrams <ign...@openservices.net>
Sheila King
comp.lang.python in article <bo2ootg61vmposfvs...@4ax.com>:
:I have a question about file locking aspect of the fcntl module for Unix
:systems.
...<snipped>...
related to this question, I have something else that is really confusing me.
I've written the following module, to wrap two different
platform specific modules:
---------------(begin LockFile.py)--------------------------
'''LockFile.py
wraps WinLock.py and posixLock.py to provide
a single interface for working with locked files
across win32 and posix platforms.
Does not support fine-grained locking.
Recommended for getting a lock on a sentinel file
before trying to read/write data to a separate data file.
'''
import os
if os.name == 'nt':
from WinLock import lockfile
elif os.name == 'posix':
from posixLock import lockfile
else:
raise ImportError, "LockFile is not supported on your platform."
class LockFile(lockfile):
pass
---------------(end LockFile.py)--------------------------
And here is the posixLock.py module, that is imported on posix systems:
------------------(begin posixLock.py)------------------------
''' posixLock.py
class lockfile
supports the same function calls as
winLock.lockfile
export to LockFile.py: a wrapper module
around the win and posix lockfiles
'''
import os, fcntl
class lockfile:
def __init__(self, filename):
if os.access(filename, os.F_OK):
self.filename = filename
else:
errmssg = filename + " does not exist. Can't lock non-existent file."
raise IOError, errmssg
def getReadLock(self):
self.f = open(self.filename)
self.fd = self.f.fileno()
fcntl.flock(self.fd, fcntl.LOCK_SH)
def getWriteLock(self):
self.f = open(self.filename)
self.fd = self.f.fileno()
fcntl.flock(self.fd, fcntl.LOCK_EX)
def unlock(self):
fcntl.flock(self.fd, fcntl.LOCK_UN)
self.f.close()
del self.fd
def flock(self, flag):
'''flags are:
LOCK_UN - unlock
LOCK_SH - acquire a shared (or read) lock
LOCK_EX - acquire an exclusive (or write) lock
'''
if flag == 'LOCK_SH':
self.getReadLock()
elif flag == 'LOCK_EX':
self.getWriteLock()
elif flag == 'LOCK_UN':
self.unlock()
else:
errmssg = "The flag " + flag + " is not implemented for flock"
raise NotImplementedError, errmssg
-------------------(end posixLock.py)-------------------------
Now, the above seems to run alright on the Linux system that I have access to.
Here is a brief interactive session, which doesn't show much, except
that at least the calls to the function don't generate error messages:
Python 2.1.1 (#6, Aug 16 2001, 17:02:08)
[GCC egcs-2.91.66 19990314/Linux (egcs-1.1.2 release)] on linux2
Type "copyright", "credits" or "license" for more information.
>>> from LockFile import LockFile
>>> obj = LockFile('lock.txt')
>>> obj.flock('LOCK_SH')
>>> obj.flock('LOCK_UN')
>>> obj.flock('LOCK_EX')
>>> obj.flock('LOCK_UN')
>>>
Now, here's the kicker:
If I change the fcntl.flock calls in the posixLock.py file to fcntl.lockf
class, then I do get errors when I try the same interactive session.
Namely:
[GCC egcs-2.91.66 19990314/Linux (egcs-1.1.2 release)] on linux2
Type "copyright", "credits" or "license" for more information.
>>> from LockFile import LockFile
>>> obj = LockFile('lock.txt')
>>> obj.flock('LOCK_SH')
>>> obj.flock('LOCK_UN')
>>> obj.flock('LOCK_EX')
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "posixLock.py", line 44, in flock
self.getWriteLock()
File "posixLock.py", line 28, in getWriteLock
fcntl.lockf(self.fd, fcntl.LOCK_EX)
IOError: [Errno 9] Bad file descriptor
>>> obj2 = LockFile('datafile.txt')
>>> obj2.flock('LOCK_EX')
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "posixLock.py", line 44, in flock
self.getWriteLock()
File "posixLock.py", line 28, in getWriteLock
fcntl.lockf(self.fd, fcntl.LOCK_EX)
IOError: [Errno 9] Bad file descriptor
>>>
So, it appears to be the LOCK_EX flag that is precipitating the error
messages. It doesn't matter whether the call to Lock the file is
on a previously instantiated object, or a fresh, brand new on that
has previously not locked on the file, and the file has not been locked by
anyone else.
I'm quite puzzled, especially since the docs say that lockf is just a wrapped,
the function works fine for flock.
I thought it would be good to use lockf instead of flock, since it appears
to handle more cases than flock does. However, if it generates an error message
every time I try to get an exclusive lock it won't do me a whole bunch of good.
Does anyone have any ideas what I might be doing wrong?
Ignacio Vazquez-Abrams
>From the fcntl(2) man page:
"
F_SETLK
The lock is set (when l_type is F_RDLCK or F_WRLCK)
or cleared (when it is F_UNLCK). If the lock is
held by someone else, this call returns -1 and sets
errno to EACCES or EAGAIN.
"
>From <bits/fcntl.h>:
---
#define F_RDLCK 0 /* Read lock. */
#define F_WRLCK 1 /* Write lock. */
#define F_UNLCK 2 /* Remove lock. */
---
Those are the values you want to use. It looks like Python may have this one
wrong.
--
Ignacio Vazquez-Abrams <ign...@openservices.net>
Sheila King
<ign...@openservices.net> wrote in comp.lang.python in article
<mailman.999039163...@python.org>:
:>From the fcntl(2) man page:
:
:"
: F_SETLK
: The lock is set (when l_type is F_RDLCK or F_WRLCK)
: or cleared (when it is F_UNLCK). If the lock is
: held by someone else, this call returns -1 and sets
: errno to EACCES or EAGAIN.
:"
:
:>From <bits/fcntl.h>:
:---
:#define F_RDLCK 0 /* Read lock. */
:#define F_WRLCK 1 /* Write lock. */
:#define F_UNLCK 2 /* Remove lock. */
:---
:
:Those are the values you want to use. It looks like Python may have this one
:wrong.
Well, this sounded good to me. So, I tried this:
''' posixLock.py
class lockfile
supports the same function calls as
winLock.lockfile
export to LockFile.py: a wrapper module
around the win and posix lockfiles
'''
import os, fcntl
class lockfile:
def __init__(self, filename):
if os.access(filename, os.F_OK):
self.filename = filename
else:
errmssg = filename + " does not exist. Can't lock non-existent file."
raise IOError, errmssg
def getReadLock(self):
self.f = open(self.filename)
self.fd = self.f.fileno()
fcntl.lockf(self.fd, fcntl.F_RDLCK)
def getWriteLock(self):
self.f = open(self.filename)
self.fd = self.f.fileno()
fcntl.lockf(self.fd, fcntl.F_WRLCK)
def unlock(self):
fcntl.lockf(self.fd, fcntl.F_UNLCK)
self.f.close()
del self.fd
def flock(self, flag):
'''flags are:
LOCK_UN - unlock
LOCK_SH - acquire a shared (or read) lock
LOCK_EX - acquire an exclusive (or write) lock
'''
if flag == 'LOCK_SH':
self.getReadLock()
elif flag == 'LOCK_EX':
self.getWriteLock()
elif flag == 'LOCK_UN':
self.unlock()
else:
errmssg = "The flag " + flag + " is not implemented for flock"
raise NotImplementedError, errmssg
But, in an interactive session, I got:
Python 2.1.1 (#6, Aug 16 2001, 17:02:08)
[GCC egcs-2.91.66 19990314/Linux (egcs-1.1.2 release)] on linux2
Type "copyright", "credits" or "license" for more information.
>>> from LockFile import LockFile
>>> obj = LockFile('lock.txt')
>>> obj.flock('LOCK_SH')
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "posixLock.py", line 42, in flock
self.getReadLock()
File "posixLock.py", line 23, in getReadLock
fcntl.lockf(self.fd, fcntl.F_RDLCK)
AttributeError: 'fcntl' module has no attribute 'F_RDLCK'
>>>
I suppose I could just use the integer value equivalents, or possibly import the
constants from FCNTL. However, I'm not certain that the integer value equivalents
will be the same across all platforms.
Hmm. What to do, what to do...?
Well, I tried adding this to the top of my posixLock.py file:
from FCNTL import F_RDLCK, F_WRLCK, F_UNLCK
But then, I just got this error:
Python 2.1.1 (#6, Aug 16 2001, 17:02:08)
[GCC egcs-2.91.66 19990314/Linux (egcs-1.1.2 release)] on linux2
Type "copyright", "credits" or "license" for more information.
>>> from LockFile import LockFile
>>> obj = LockFile('lock.txt')
>>> obj.flock('LOCK_SH')
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "posixLock.py", line 43, in flock
self.getReadLock()
File "posixLock.py", line 24, in getReadLock
fcntl.lockf(self.fd, F_RDLCK)
ValueError: unrecognized flock argument
>>>
Maybe I just need to address the fcntl command, itself (i.e. fcntl.fcntl), however,
I feel that I'm going beyond where I feel comfortable (I'm not really good with
Linux/Unix stuff...)
Additional advice is welcome...
Ignacio Vazquez-Abrams
> [snip]
>
> Additional advice is welcome...
>
> --
> Sheila King
> http://www.thinkspot.net/sheila/
> http://www.k12groups.org/
Oh hell. fcntlmodule.so munges the parameter passed to it so that LOCK_UN
looks like F_UNLCK, LOCK_SH like F_RDLCK, and LOCK_EX like F_WRLCK. Whose
bright idea was that anyway?
But I think I have found the real reason this time. Passing no mode argument
to open() tells Python to open the file read-only. Guess what? By rejecting
LOCK_EX Linux is telling you "Write lock? Why the hell does a read-only file
need a write lock?". Same thing with 'w' and LOCK_SH. 'w+' and 'r+' can take
both.
So it seems that you need to take both a filename _and_ a mode in __init__(),
and then raise an exception in the appropriate locking methods based on
f.mode. You may also want to take a third parameter for initial lock mode,
just in case.
Also, you should make it so that the file is immediately opened in __init__(),
then close the file both in a close() method and in __del__(). In both methods
you should probably check f.closed to make sure that the file hasn't already
been closed.
Hopefully your problem has now been solved. Third (or fourth or whatever :) )
time's the charm, right?
--
Ignacio Vazquez-Abrams <ign...@openservices.net>
Sheila King
:Oh hell. fcntlmodule.so munges the parameter passed to it so that LOCK_UN
:looks like F_UNLCK, LOCK_SH like F_RDLCK, and LOCK_EX like F_WRLCK.
I was suspecting that might be what was going on.
:But I think I have found the real reason this time. Passing no mode argument
:to open() tells Python to open the file read-only. Guess what? By rejecting
:LOCK_EX Linux is telling you "Write lock? Why the hell does a read-only file
:need a write lock?". Same thing with 'w' and LOCK_SH. 'w+' and 'r+' can take
:both.
Great. Thanks. I will test more thoroughly, later tonight. But
preliminary testing seems to indicate this has solved the problem.
:So it seems that you need to take both a filename _and_ a mode in __init__(),
:and then raise an exception in the appropriate locking methods based on
:f.mode. You may also want to take a third parameter for initial lock mode,
:just in case.
Not at all. Based on whether the flag is LOCK_SH or LOCK_EX, I know
whether to open the file with 'r' or with 'w'.
:Also, you should make it so that the file is immediately opened in __init__(),
Why? I don't see any persuasive reason for that. The methods that I'm
using under windows don't allow for me to separate the action of opening
the file and obtaining the lock (there is a single command which does
both), and I wanted to keep the functions between the windows and the
posix versions as close to parallel as possible. In any case, I can't
see any compelling reason for the file opening to occur in the init
function.
:then close the file both in a close() method and in __del__(). In both methods
:you should probably check f.closed to make sure that the file hasn't already
:been closed.
Thanks. I probably do need to make a __del__ method. I hadn't thought of
that.
:Hopefully your problem has now been solved. Third (or fourth or whatever :) )
:time's the charm, right?
I really appreciate the help. The Python community is great!
Ignacio Vazquez-Abrams
> On Tue, 28 Aug 2001 20:02:51 -0400 (EDT), Ignacio Vazquez-Abrams
> <ign...@openservices.net> wrote in comp.lang.python in article
> <mailman.999043423...@python.org>:
>
> [snip]
>
> :Also, you should make it so that the file is immediately opened in __init__(),
>
> Why? I don't see any persuasive reason for that. The methods that I'm
> using under windows don't allow for me to separate the action of opening
> the file and obtaining the lock (there is a single command which does
> both), and I wanted to keep the functions between the windows and the
> posix versions as close to parallel as possible. In any case, I can't
> see any compelling reason for the file opening to occur in the init
> function.
I just finished talking with a friend who's done more recent Win32 development
than me, and he claims that you can reopen the file (without closing it) using
the aforementioned API function again in order to apply a lock. It sounds
fishy to me too, but it might be worth a shot.
--
Ignacio Vazquez-Abrams <ign...@openservices.net>
Sheila King
I'm sorry. You lost me here.
The windows function I'm using (which was mentioned elsewhere) is
CreateFile. Whether or not I can reopen the file without having closed
it is immaterial to my module, since my unlock function closes the file
and deletes the file handle.
What am I supposed to "give a shot" to???
So far, my windows code is all performing as I expect (not that I've
shared it here...). That may be partly why I'm confused by this segue
into discussing the Windows implementation.
Ignacio Vazquez-Abrams
Fair enough. Your code works how you expect it to. But does it work how
_others_ would expect it to? With normal files in Python, one call to open()
opens the file, and then you can immediately read() or write() with it.
There's no need to call a secondary method in order to "activate" it; it just
"works".
I don't know if CreateFile() is the correct API function (my friend couldn't
remember the name, MSDN is pooched as usual, the SDK will take a while to D/L,
and my memory fails me), but the class as written doesn't emulate normal
Python behaviour. If you give me time I can find the API function. It
shouldn't take me that long (and it would help if I could find my copy of VC++
6.0 :( ), so just wait a day or two then I can help you emulate normal files
perfectly.
--
Ignacio Vazquez-Abrams <ign...@openservices.net>
Mats Wichmann
:I have a question about file locking aspect of the fcntl module for Unix
while 1:
print "don't do locks over NFS!"
This position may POSSIBLY be able to be softened for NFS V4.
Mats Wichmann
Sheila King
in comp.lang.python in article <3b8c6396...@news.laplaza.org>:
:On Tue, 28 Aug 2001 21:25:51 GMT, Sheila King <she...@spamcop.net>
:wrote:
::Am I correct, then, in interpreting these together, to mean, that I will
::get greater portability/control if I use fcntl.lockf than if I use
::fcntl.flock? Because, as I read this, fcntl.lockf may work in situations
::where fcntl.flock does not (especially over NFS?).
:
:while 1:
: print "don't do locks over NFS!"
:
:
:This position may POSSIBLY be able to be softened for NFS V4.
:Mats Wichmann
I appreciate the input/feedback. However, simply "don't do locks over
NFS" is a bit brief. Could you elaborate? Why not? What problems does
this create? Why might the situation improve in NFS V4?
David Bolen
> while 1:
> print "don't do locks over NFS!"
>
>
> This position may POSSIBLY be able to be softened for NFS V4.
Fine grained file locking is horrid over NFS (even after you figure
out what custom independent communications path you want to use to
indicate locks), but sentinel file locking can still be done. If you
use a directory rather than a file (e.g., your "lock" operation is to
try to make a directory), then generally your request is atomic all
the way through the NFS RPC and the underlying filesystem. The major
con is trying to guarantee that you clean up since there's no live
handle, but it's still often better than nothing.
--
-- David
--
/-----------------------------------------------------------------------\
\ David Bolen \ E-mail: db...@fitlinxx.com /
| FitLinxx, Inc. \ Phone: (203) 708-5192 |
/ 860 Canal Street, Stamford, CT 06902 \ Fax: (203) 316-5150 \
\-----------------------------------------------------------------------/
David Bolen
> I appreciate the input/feedback. However, simply "don't do locks over
> NFS" is a bit brief. Could you elaborate? Why not? What problems does
> this create? Why might the situation improve in NFS V4?
It's been a while since I was deep in NFS messes, but the short of it
is that NFS is a stateless protocol, and file locking is inherently a
stateful activity. NFS separated out lock issues into a separate
protocol that was actually normally serviced by a completely separate
daemon than normal NFS operations. And the problem was that it just
didn't work that well (except perhaps in perfect conditions in
homogeneous environments). There were race conditions, locks could
get stranded due to system crashes, lock daemons didn't always work
properly and so on. The lock protocol also changed from NFS v2 to v3
creating further headaches for interoperability. And the NFS model
wasn't the greatest fit for non-Unix systems, complicating the attempt
to emulate lock locking facilities.
In most cases, anyone who tried to depend on locking over NFS (after
all, if you're using it it's probably important) eventually got burned
one too many times and just swore off of it.
Version 4 of NFS improves a lot on the handling of locks, moving them
(and the need to maintain their state on the server as a leased basis)
into the core protocol. I don't have any personal experience with v4
configurations (the RFC is from 12/2000 so I don't know how widely its
available yet), but at least in theory the protocol ought to be able
to do a much better job managing locks.
I'm sure doing some net searches for NFS and locking could point up
various horror stories :-)