How to lock files (the easiest/best way)?
Elmo Mäntynen
with a quick look.
Elmo Mäntynen
> Elmo Mäntynen enlightened us with:
>> Is there something better than using fnctl? It seems a bit
>> intimidating with a quick look.
>
> Locking files is a complex business. What do you want to lock? Why?
> Lock it with respect to what? It's easier to lock a file for local
> use, compared to when the file also has to be locked from, say, use
> via the network.
>
> Sybren
Only locally. I want to be able to read/write to a single file from
multiple possibly parallel processes. Would 'touch lock' (or something
like that) work reliably (this just occured to me)?
Elmo Mäntynen
> Elmo Mäntynen enlightened us with:
>> Only locally. I want to be able to read/write to a single file from
>> multiple possibly parallel processes. Would 'touch lock' (or
>> something like that) work reliably (this just occured to me)?
>
> I use a lock directory for that, os.mkdir('/var/lock/somedir').
> If you use a file, you need two steps:
> 1) Check whether the lock-file exists
> 2) Create the lock-file
>
> This is not atomic. With a directory, creating it will fail if it
> already exists. This means you can atomically check for the lock, and
> if it doesn't exist already, you've immediately created it too.
>
> Sybren
Thanks. Is that what atomic basically means?
Gerhard Fiedler
> Elmo Mäntynen enlightened us with:
>> Only locally. I want to be able to read/write to a single file from
>> multiple possibly parallel processes. Would 'touch lock' (or
>> something like that) work reliably (this just occured to me)?
>
> I use a lock directory for that, os.mkdir('/var/lock/somedir').
> If you use a file, you need two steps:
> 1) Check whether the lock-file exists
> 2) Create the lock-file
cvsnt for example used to use lock files. Now it uses a lock server: a
server app that just sits there and allows different processes to acquire,
check for and release locks on files. More implementation work, probably,
but more efficient.
Gerhard
Paul Rubin
> I use a lock directory for that, os.mkdir('/var/lock/somedir').
> If you use a file, you need two steps:
> 1) Check whether the lock-file exists
> 2) Create the lock-file
> already exists. This means you can atomically check for the lock, and
> if it doesn't exist already, you've immediately created it too.
The classic way in Unix was to make a link:
1) link('some_other_file', 'lockfile')
This was atomic and would fail if lockfile already existed.
I'm not sure whether it can still be reliable, in current environments
that can involve remote file systems etc. But I'm not sure if the
mkdir approach works either.
Maybe you want to use something like shm or posh?
Eric S. Johansson
> Is there something better than using fnctl? It seems a bit intimidating
> with a quick look.
try the portlocker wrapper from the active state cookbook. I have a
version which has been slightly updated for more modern pythons. I
really need to make my darcs repository visible so you could get it.
Alternatively, you can try the makedir trick. Creating directories are
atomic operations if it exists, you will get error message. If it
doesn't, you will get a directory and thereby capture the lock. You
delete the directory when you release the lock.
crude but effective. I used it in building a file based queue system.
You know, I really should learn how to build eggs because I have a whole
bunch of little pieces of software that would probably be useful to others.
---eric
Jim Segrave
Here's a more complete file locking scheme which uses a lockfile with
the O_CREAT and O_EXCL flags to make the creation atomic. If it gets
the lock, it writes its PID in readable form in the file. It also does
two other things:
If you know that no process should lock the file for more than a fixed
period of time, it will retry once/second as long as the lock file is
not older than that fixed period of time. If it is older, it will
report the problem, including the PID of the locking process and exit.
This caters for a process which has set the lock file and then
terminates without removing it (which can happen do to an application
or server crash).
------------------------------------------
import os
import errno
import sys
import time
import stat
# the maximum reasonable time for aprocesstobe
max_wait = 10
lockfile = "/tmp/mylock"
while True:
try:
fd = os.open(lockfile, os.O_EXCL | os.O_RDWR | os.O_CREAT)
# we created the lockfile, so we're the owner
break
except OSError, e:
if e.errno != errno.EEXIST:
# should not occur
raise
try:
# the lock file exists, try to stat it to get its age
# and read it's contents to report the owner PID
f = open(lockfile, "r")
s = os.stat(lockfile)
except OSError, e:
if e.errno != errno.EEXIST:
sys.exit("%s exists but stat() failed: %s" %
(lockfile, e.strerror))
# we didn't create the lockfile, so it did exist, but it's
# gone now. Just try again
continue
# we didn't create the lockfile and it's still there, check
# its age
now = int(time.time())
if now - s[stat.ST_MTIME] > max_wait:
pid = f.readline()
sys.exit("%s has been locked for more than " \
"%d seconds (PID %s)" % (lockfile, max_wait,
pid))
# it's not been locked too long, wait a while and retry
f.close()
time.sleep(1)
# if we get here. we have the lockfile. Convert the os.open file
# descriptor into a Python file object and record our PID in it
f = os.fdopen(fd, "w")
f.write("%d\n" % os.getpid())
f.close()
--
Jim Segrave (j...@jes-2.demon.nl)
Jim Segrave
this should read:
if e.errno != errno.ENOENT:
(it was left with EEXIST from testing this code by forcing an error,
as the code for this failure requires a very tight race condition to test)
> sys.exit("%s exists but stat() failed: %s" %
> (lockfile, e.strerror))
> # we didn't create the lockfile, so it did exist, but it's
--
Jim Segrave (j...@jes-2.demon.nl)
Donn Cave
Elmo Mäntynen <elm...@jippii.fi> wrote:
Yes, and also "race condition". That's why Jim Segrave's
example code uses O_EXCL with open(2).
Donn Cave, do...@u.washington.edu
Fuzzyman
Sybren Stuvel wrote:
> Elmo Mäntynen enlightened us with:
> > multiple possibly parallel processes. Would 'touch lock' (or
> > something like that) work reliably (this just occured to me)?
>
> If you use a file, you need two steps:
> 1) Check whether the lock-file exists
> 2) Create the lock-file
>
> This is not atomic. With a directory, creating it will fail if it
> already exists. This means you can atomically check for the lock, and
> if it doesn't exist already, you've immediately created it too.
`Pathutils <http://www.voidspace.org.uk/python/pathutils.html>`_ has a
simple interface to a system like this.
Apparently not all platforms guarantee that attempts to create a
directory will fail if the directory already exists - so pathutils does
*slightly* more, but it's the same idea.
All the best,
Fuzzyman
http://www.voidspace.org.uk/python/index.shtml
>
> Sybren
> --
> The problem with the world is stupidity. Not saying there should be a
> capital punishment for stupidity, but why don't we just take the
> safety labels off of everything and let the problem solve itself?
> Frank Zappa
pavlove...@gmail.com
> Is there something better than using fnctl? It seems a bit intimidating
> with a quick look.
Although fcntl is pretty opaque, it's quite easy to use if all you want
is a simple exclusive lock for all your data. The thing to keep in
mind is, you don't have to lock every file you want exclusive access
to. In fact, doing that provides no added security (since it's an
advisory lock and programs are free to ignore it). For simple cases,
locking one file and only accessing your data if you have that lock
works, and is very simple.
What I usually do is touch an empty file (say, "lockfile") in my data
directory. I don't access any files in that directory unless I have a
lock to the lockfile. This is done simply with (untested):
import fcntl
f = open("/path/to/data/directory/lockfile","r")
try:
fcntl.flock(f.fileno(),fcntl.LOCK_EX)
...access data freely here...
finally:
f.close()
Closing the file should release the lock (unless you have a truly
horrible operating system).
Carl Banks
Carl J. Van Arsdall
>
>> ith a quick look.
>>
>
>
> f = open("/path/to/data/directory/lockfile","r")
> try:
> fcntl.flock(f.fileno(),fcntl.LOCK_EX)
> ...access data freely here...
> finally:
> f.close()
>
> Closing the file should release the lock (unless you have a truly
> horrible operating system).
>
>
problems using the python fcntl module and nfs - could be that horrible
operating system, but doing things like that over nfs can be tricky).
-carl
--
Carl J. Van Arsdall
cvana...@mvista.com
Build and Release
MontaVista Software
Carl Banks
Carl J. Van Arsdall wrote:
> pavlove...@gmail.com wrote:
> >
> > f = open("/path/to/data/directory/lockfile","r")
> > try:
> > fcntl.flock(f.fileno(),fcntl.LOCK_EX)
> > ...access data freely here...
> > finally:
> > f.close()
> >
> > Closing the file should release the lock (unless you have a truly
> > horrible operating system).
> >
> >
> I also find that fcntl has problems with NFS (or at least, *I* had
> problems using the python fcntl module and nfs - could be that horrible
> operating system, but doing things like that over nfs can be tricky).
Ah, that's a tough one.
Apparently, you could lock files on NFS if both the client OS and NFS
server are up to the task (i.e., sufficiently recent), but good luck
getting that to fly. And, with NFS, even some of the seemingly
foolproof methods like "lock directories" aren't necessarily going to
work. Cross your fingers and hope you have a solid NFS server and
well-behaved clients.
Carl Banks