File Lock in TCL
Jean-Claude Wippler
>
> Is it possible to obtain a lock on a file using TCL ?
Here's a mechanism I once got from Alexandre Ferrieux:
Lock: open $file {CREAT EXCL WRONLY} ;# you can even
# write info into it
Unlock: file delete $file
It works well in the most common cases. It gets (a lot) hairier for
solutions which do not use a separate lock file.
Some Unix systems use flock, others lockf, others ioctl, and Windows has
yet another library function for it (sopen). When used with NFS, you
have to accept that:
- some systems will ignore locks, or ignore only across the network
- some systems have no advisory locks, others only advisory locks
- some systems can fatally lock up multiple machines (trhough lockd)
So I guess the answer is: in pure Tcl, you can use the lines shown above
for basic locking, but the more genreal case is very complex (and not a
Tcl-specific problem).
I ended up doing different things on Windows, Linux, and Solaris, and
accepting the fact that locks over NFS were impossible to get right.
The code I came up with is buried in this source file, around line 300:
http://www.equi4.com/metakit/mk4tcl/mk4tcl.cpp
Some good locking reference material I found on the net a while back,
were the sources of IMAP (http://www.washington.edu/imap/) and procmail
(ftp://ftp.informatik.rwth-aachen.de/pub/packages/procmail/).
-- Jean-Claude
P.S. My server is being moved, it/I may be inaccessible for a day or so.
Darren New
> - some systems will ignore locks, or ignore only across the network
> - some systems have no advisory locks, others only advisory locks
> - some systems can fatally lock up multiple machines (trhough lockd)
And you forgot "subject to live-lock". At least under any UNIX I've
used.
Another way to do it is to create a separate lock file if it doesn't
exist, write your process-ID into it, close the file, then reopen the
lock file and read to make sure your PID is still there. This eliminates
most race conditions, but it's rather high overhead. It also allows a
separate job to find and potentially unlock lock-files left over from
crashed processes.
--
Darren New / Senior Software Architect / MessageMedia, Inc.
San Diego, CA, USA (PST). Cryptokeys on demand.
Help outlaw Dihydrogen monoxide, a major component of acid rain!
Alexandre Ferrieux
>
> Jean-Claude Wippler wrote:
> > - some systems will ignore locks, or ignore only across the network
> > - some systems have no advisory locks, others only advisory locks
> > - some systems can fatally lock up multiple machines (trhough lockd)
>
> And you forgot "subject to live-lock". At least under any UNIX I've
> used.
>
> Another way to do it is to create a separate lock file if it doesn't
> exist, write your process-ID into it, close the file, then reopen the
> lock file and read to make sure your PID is still there. This eliminates
> most race conditions, but it's rather high overhead. It also allows a
> separate job to find and potentially unlock lock-files left over from
> crashed processes.
Yes. Re-read Jean-Claude's message, it was the very first method
mentioned :).
-Alex
lvi...@cas.org
According to Darren New <dn...@messagemedia.com>:
:Jean-Claude Wippler wrote:
:> - some systems will ignore locks, or ignore only across the network
:> - some systems have no advisory locks, others only advisory locks
:> - some systems can fatally lock up multiple machines (trhough lockd)
:
:And you forgot "subject to live-lock". At least under any UNIX I've
:used.
:
:Another way to do it is to create a separate lock file if it doesn't
:exist, write your process-ID into it, close the file, then reopen the
:lock file and read to make sure your PID is still there. This eliminates
:most race conditions, but it's rather high overhead. It also allows a
:separate job to find and potentially unlock lock-files left over from
:crashed processes.
There are so many possible uses of file locking that trying to describe
an algorithm seems difficult. Another thing to consider is adding the
machine name - in cases of NFS, multiple machines will see a lock file,
check to see 'if the process id is still there' not find it and blow
away the lock file otherwise.
Seems like this might be a good topic for a Wiki page...
--
<URL: mailto:lvi...@cas.org> Quote: Saving the world before bedtime.
<*> O- <URL: http://www.purl.org/NET/lvirden/>
Unless explicitly stated to the contrary, nothing in this posting
should be construed as representing my employer's opinions.
Jean-Claude Wippler
following code ought to work correctly on Unix, Windows, and Mac:
LOCK:
close [open $file {CREAT WRONLY EXCL}]
UNLOCK:
file delete $file
The open will throw an exception if the lock was denied.
On Unix, the lock file should not be on a remote/NFS mount point.
-- Jean-Claude
P.S. Thanks to Cameron Laird for spotting the errors in the last post.
Kenneth H. Cox
> On Unix, the lock file should not be on a remote/NFS mount point.
Why is that? It is much less useful if this is true. In many
environments, shared files are on NFS, and the only way to get
a lock on a shared file is to put the lock file in the same
directory as the file (on NFS).
Regards,
Ken
Jean-Claude Wippler
> Why is that? It is much less useful if this is true. In many
> environments, shared files are on NFS, and the only way to get
> a lock on a shared file is to put the lock file in the same
> directory as the file (on NFS).
If you want to lock with a separate lock file ("dotlocking", see below),
then there is no problem, although there can be a race condition (which,
on a network, slow link, etc, can be a serious issue). In this case,
using the Tcl-only locking approach is perhaps most suitable.
If you want to use fcntl/lockf/flock system calls, then locking may or
may not work - AFAIK, some platform combinations simply don't lock,
while some others can cause the underlying faulty lock daemon to hang.
Here's an excerpt on a note I found through Alta Vista:
========================================================================
A brief overview regarding locking methods would be:
dotlocking (*.lock) If you want to lock "testfile", try and
create a *new* "testfile.lock" in the same
directory. Repeat (with ample sleep) until
this succeeds. To unlock, simply remove the
file. This method is supported by almost
all configurations.
fcntl() (POSIX) Usually implemented over the lockd program
(thus NFS aware).
lockf() (SysV) Usually implemented over fcntl().
flock() (BSD) On SunOS this only locks locally, on other
systems it might use the lockd.
========================================================================
It's by Stephen van den Berg, Procmail's author:
http://members.xoom.com/procmail/best-of-procmail-list/3
Another place to look for tips is Mailman (Python, http://www.list.org),
see the file mailman-1.0rc1/Mailman/flock.py - which says:
"""Portable (?) file locking with timeouts.
This code should work with all versions of NFS.
The algorithm was suggested by the GNU/Linux open() man page. Make
sure no malicious people have access to link() to the lock file.
"""
It sure would be nice if there were a package which jumps through all
known hoops to implement locking, on most Unix'es, Windows, Mac, etc.
IMHO, this is an example of where the split between Tcl, Python, Perl
and others is hampering the generation of a general package.
-- Jean-Claude
Bruce S. O. Adams
Jean-Claude Wippler wrote:
Tk is already accessible to both Python and Perl, though I don't know how
easy it is to use it? Perhaps there should be a general discussion on how
to
share extensions between the three platforms. I only know the TCL mechanism
well at the momment, but I got the general impression that it was the best.
Perhaps, that's just because Perl & Python are less advocated on
comp.lang.tcl
than tcl is :-). Is anyone going to raise this topic at the forthcoming
scripting conference (which I believe originally centred around perl)? What
would
we need for a generalised extension architecture? I'm sure it would involve
more
than just swapping Tcl_CreateCommand for Perl_CreateCommand or whatever
(especially as we are now encouraged to use Tcl_Obj's in our interfaces.
A general solution is to have in each language a routine allowing cross
language
calls to be made e.g. in perl have an InvokeTCLCommand function and in
TCL an InvokePerlCommand function. Of course some extensions would
require access to the languages interpreter and others wouldn't. This
solution
would also come at the cost of some efficiency. Would people be prepared to
pay that cost? Any thoughts?
Regards,
Bruce A.
lvi...@cas.org
According to Jean-Claude Wippler <j...@equi4.com>:
:dotlocking (*.lock) If you want to lock "testfile", try and
: create a *new* "testfile.lock" in the same
: directory. Repeat (with ample sleep) until
: this succeeds. To unlock, simply remove the
: file. This method is supported by almost
: all configurations.
Problems with this simple implementation include:
unable to determine whether presence of lock file means a process is truly
updating the file, or if a process crashed after having locked the file
any NFS implementation which includes any sort of caching to 'improve'
streamlining introduces a period of time during which the cache is out of
date....
Alexandre Ferrieux
>
> any NFS implementation which includes any sort of caching to 'improve'
> streamlining introduces a period of time during which the cache is out of
> date....
Yes, it seems to be currently the case with NFS implementations. No I'm
wondering why it should be so: the O_EXCL flag is sufficiently
characteristic of the locking situation to be handled specially by the
cache software. Why doesn't it simply synchronize with the server when
an O_EXCL is requested ?
-Alex
Matt Newman
>wondering why it should be so: the O_EXCL flag is sufficiently
>characteristic of the locking situation to be handled specially by the
>cache software. Why doesn't it simply synchronize with the server when
>an O_EXCL is requested ?
The only truly reliable locking that I have found for Unix is the rename
option - file renames are always atomic, one of the few things that is in
NFS.
E.g.
create tmpfile
rename tmpfile mailfile ;# sucess means I aquried the lock, failure that it
is already locked.
This also has the advantage that it works from shell scripts/tcl etc.
MN