Help: using msvcrt for file locking
Sheila King
hope that the methods in the msvcrt module will work OK on any win32?
I thought I understood what I was doing. I had tested a bit of file
locking stuff.
I opened up two separate MSDOS console windows, and tried acquiring
locks and releasing locks, etc...
In one window:
E:\Apache\cgi-bin\pylinks>python
Python 2.1.1 (#20, Jul 20 2001, 01:19:29) [MSC 32 bit (Intel)] on win32
Type "copyright", "credits" or "license" for more information.
>>> import msvcrt, os, sys
>>> f = open('lock.txt')
>>> fd = f.fileno()
>>> size = os.path.getsize('lock.txt')
>>> from time import localtime
>>> msvcrt.locking(fd, msvcrt.LK_RLCK, size)
>>> msvcrt.locking(fd, msvcrt.LK_UNLCK, size)
>>> localtime()
(2001, 8, 25, 12, 14, 19, 5, 237, 1)
>>> msvcrt.locking(fd, msvcrt.LK_RLCK, size)
>>>
In the other window:
E:\Apache\cgi-bin\pylinks>python
Python 2.1.1 (#20, Jul 20 2001, 01:19:29) [MSC 32 bit (Intel)] on win32
Type "copyright", "credits" or "license" for more information.
>>> import msvcrt, os, sys
>>> f = open('lock.txt')
>>> fd = f.fileno()
>>> size = os.path.getsize('lock.txt')
>>> from time import localtime
>>> msvcrt.locking(fd, msvcrt.LK_RLCK, size)
Traceback (most recent call last):
File "<stdin>", line 1, in ?
IOError: [Errno 36] Resource deadlock avoided
>>> localtime()
(2001, 8, 25, 12, 13, 24, 5, 237, 1)
>>> msvcrt.locking(fd, msvcrt.LK_RLCK, size)
>>> msvcrt.locking(fd, msvcrt.LK_UNLCK, size)
>>>
When one process had the lock on the file first, the other process would
either time out and given an IOError, or else wait until the first
process unlocked the file, and then it appeared to acquire the lock.
Even very simply, in a single window, I can do this and at least get no
error messages:
>>> f = open(r'e:\apache\cgi-bin\pylinks\lock.txt')
>>> fd = f.fileno()
>>> size = os.path.getsize(r'e:\apache\cgi-bin\pylinks\lock.txt')
>>> msvcrt.locking(fd, msvcrt.LK_RLCK, size)
>>> msvcrt.locking(fd, msvcrt.LK_UNLCK, size)
>>>
I realize that this doesn't prove I had a lock on anything, but I've
tested that out separately, and I believe I am acquiring a lock and
releasing it without error.
NOW, why doesn't this class that I've written work without error?
Here is the class:
---------------(begin winLock module)--------------------
''' winLock.py
class lockobject
supports the same function calls as
posixLock.lockobject
export to lockobject.py: a wrapper module
around the win and posix lockobjects
'''
import os, msvcrt
modes = {'write':msvcrt.LK_RLCK, 'read':msvcrt.LK_NBLCK,\
'unlock':msvcrt.LK_UNLCK}
class lockobject:
def _lockoperation(self, lockfilename, mode):
size = os.path.getsize(lockfilename)
f = open(lockfilename, 'r')
f.seek(0)
msvcrt.locking(f.fileno(), modes[mode], size)
def getReadLock(self, lockfilename):
self._lockoperation(lockfilename, 'read')
def getWriteLock(self, lockfilename):
self._lockoperation(lockfilename, 'write')
def unlock(self, lockfilename):
self._lockoperation(lockfilename, 'unlock')
--------------(end of winLock module)-------------------------
Here is an interactive session with the module:
>>> import winLock
>>> from winLock import lockobject
>>> one = lockobject()
>>> one.getWriteLock(r'e:\apache\cgi-bin\pylinks\lock.txt')
>>> one.unlock(r'e:\apache\cgi-bin\pylinks\lock.txt')
Traceback (most recent call last):
File "<interactive input>", line 1, in ?
File "e:\apache\cgi-bin\pylinks\winLock.py", line 28, in unlock
self._lockoperation(lockfilename, 'unlock')
File "e:\apache\cgi-bin\pylinks\winLock.py", line 19, in
_lockoperation
msvcrt.locking(f.fileno(), modes[mode], size)
IOError: [Errno 13] Permission denied
>>>
Now, why am I getting this Permission denied error? I'm only doing the
same things I did before without the class? (OK, I did put a seek(0)
command in there. It didn't used to be there and I was getting the same
error, so I though maybe I should set the file pointer to the beginning
of the file each time? Anyhow, doesn't work either way.)
If someone can give me some tips here, I'd really appreciate it.
--
Sheila King
http://www.thinkspot.net/sheila/
http://www.k12groups.org/
Ignacio Vazquez-Abrams
I have no experience with msvcrt, but I'm guessing that it might be that
msvcrt.locking() requires the descriptor to be the same when unlocking as when
locking.
Also, where is this posixLock so that I can look at it?
--
Ignacio Vazquez-Abrams <ign...@openservices.net>
Sheila King
<ign...@openservices.net> wrote in comp.lang.python in article
<mailman.99877065...@python.org>:
:I have no experience with msvcrt, but I'm guessing that it might be that
:msvcrt.locking() requires the descriptor to be the same when unlocking as when
:locking.
Thanks. That sounds like a good tip. I will try to work on that. I think
I can figure out a way to handle it.
:Also, where is this posixLock so that I can look at it?
Well, it didn't have any bearing on this question, so I didn't post it.
But it is my own code (not yet tested). Since you ask, here it is:
''' posixLock.py
class lockobject
supports the same function calls as
winLock.lockobject
export to lockobject.py: a wrapper module
around the win and posix lockobjects
'''
import fcntl
class lockobject:
def getReadLock(self, lockfilename):
fcntl.flock(lockfilename.fileno(), fcntl.LOCK_SH)
def getWriteLock(self, lockfilename):
fcntl.flock(lockfilename.fileno(), fcntl.LOCK_EX)
def unlock(self, lockfilename):
fcntl.flock(lockfilename.fileno(), fcntl.LOCK_UN)
And then there is the wrapper class, lockobject:
''' lockobject.py
a class for supporting file locking across
posix and windows platforms. This class is an abstraction
that wraps the locking details between these two
platforms.
Wrapped methods:
getReadLock(lockfilename)
getWriteLock(lockfilename)
unlock(lockfilename)
A LockObject is an object which will be read from and
written to. However, there may be multiple concurrent
processes trying to access the object. In order to prevent
data corruption, write operations must have exclusive
access. Read operations can occur simultaneously with no ill-effects.
The lockfilename is the name of a file, different from the
LockObject, which the LockObject must obtain an appropriate
type of lock on (a shared lock for reading, an exclusive
lock for writing), before any attempts at reading data from or
writing data to the Lock Object occur.
It is important that when a read or write process has finished,
that the LockObject unlock the lockfile so that other processes
can proceed. (All other processes attempting to read or write
may be blocked, depending on the type of lock.)
Note: On the Windows platforms, when a lock is requested (either
a read or write lock), there will be a maximum of ten attempts
made in one second intervals to obtain a lock on the lockfile. If
all of these attempts fail, the operation will time out and raise
an IOError.
'''
import os
osname = os.name
if osname == "nt":
from winLock import lockobject
elif osname == "posix":
from posixLock import lockobject
else:
raise ImportError, "locking not supported on your platform"
class LockObject(lockobject):
pass
if __name__ == '__main__':
print "Testing..."
lockfile = raw_input("Enter a complete path/filename to test for
locking")
one = LockObject()
print "created a LockObject"
I'm just really dissatisfied with the poor file locking support in
Python, unless one is writing code exclusively for posix platforms.
About a week ago, I was composing a rant on that topic to post here. I
did a lot of research on the topic, and a lot of web searches
(google.com, groups.google.com, activestate.com, mail.python.org and so
forth...)
In the end, I decided not to post the rant (I deleted it), since I
didn't think I'd gain anything by it. I decided to just rather try to
solve the problem myself.
Really, though, I do think the problem should be solved by someone in
the Python developers circle and included in the standard library
(something along the lines of anydbm). But, that's just my opinion. I
have a number of reasons why I feel that way, but I won't go into it
right now.
Anyhow, Ignacio, thanks for the idea about the file descriptor. I think
you may have something there. I will try it out later tonight and see if
that helps.
Ignacio Vazquez-Abrams
> Well, it didn't have any bearing on this question, so I didn't post it.
> But it is my own code (not yet tested). Since you ask, here it is:
>
> [snip]
>
> And then there is the wrapper class, lockobject:
>
> [snip]
>
I like. The only suggestion I'm going to make is that instead of passing the
filename to methods of the instantiated object, perhaps passing it when
instantiating it might be better. That way you can keep all the information
about the file on-hand when the locking methods are called. You could also
pass the mode and have a complete replacement for the file object, with
locking added.
> I'm just really dissatisfied with the poor file locking support in
> Python, unless one is writing code exclusively for posix platforms.
> About a week ago, I was composing a rant on that topic to post here. I
> did a lot of research on the topic, and a lot of web searches
> (google.com, groups.google.com, activestate.com, mail.python.org and so
> forth...)
>
> In the end, I decided not to post the rant (I deleted it), since I
> didn't think I'd gain anything by it. I decided to just rather try to
> solve the problem myself.
Good for you. I'm on several mailing lists and there are always people
complaining about a certain feature not being supported, yet they aren't
willing to look into the problem.
> Really, though, I do think the problem should be solved by someone in
> the Python developers circle and included in the standard library
> (something along the lines of anydbm). But, that's just my opinion. I
> have a number of reasons why I feel that way, but I won't go into it
> right now.
One of the problems with freely-available open-source software is that the
developers frequently aren't paid. That puts a limiting factor on how much
time they can spend on a project, because the rest of their time is spent
working and dealing with family.
One of the benefits of open-source software (and of scripting/programming
languages in general) is that the circle of developers is usually not limited
to a small number of people who have to be bugged for everything (even though
they usually are ;) ).
> Anyhow, Ignacio, thanks for the idea about the file descriptor. I think
> you may have something there. I will try it out later tonight and see if
> that helps.
>
> --
> Sheila King
> http://www.thinkspot.net/sheila/
> http://www.k12groups.org/
--
Ignacio Vazquez-Abrams <ign...@openservices.net>
Tim Peters
> I'm trying to do some file locking on Windows. I'm running Win98, but I
> hope that the methods in the msvcrt module will work OK on any win32?
msvcrt directly exposes some library functions supplied by, and unique to,
Microsoft. So, like it or not (I don't like it myself <wink>), Microsoft is
the only entity who can answer questions about them with authority. FWIW,
the docs MS supplied with MSVC 6 say their _locking function works under
"Win 95, Win NT", which *usually* means "everything after Win 3.1".
Also as usual, the functions MS provides as part of their C library are a
feeble subset of what you can do by using the Win32 API directly, in this
case by using the Win32 LockFile and LockFileEx functions. I don't know
whether win32all exposes those specific functions; core Python does not.
> I thought I understood what I was doing. I had tested a bit of file
> locking stuff.
>
> I opened up two separate MSDOS console windows, and tried acquiring
> locks and releasing locks, etc...
>
> [seemingly successful stuff snipped]
> NOW, why doesn't this class that I've written work without error?
>
> Here is the class:
> ---------------(begin winLock module)--------------------
> ''' winLock.py
>
> class lockobject
> supports the same function calls as
> posixLock.lockobject
> export to lockobject.py: a wrapper module
> around the win and posix lockobjects
> '''
> import os, msvcrt
>
> modes = {'write':msvcrt.LK_RLCK, 'read':msvcrt.LK_NBLCK,\
> 'unlock':msvcrt.LK_UNLCK}
>
> class lockobject:
> def _lockoperation(self, lockfilename, mode):
> size = os.path.getsize(lockfilename)
> f = open(lockfilename, 'r')
> f.seek(0)
> msvcrt.locking(f.fileno(), modes[mode], size)
My guess is that's why: in your interactive sessions, you kept the file
open. In the _lockoperation method above, the file will be closed
implicitly under CPython the instant _lockoperation returns (because the
refcount on f falls to 0 then, and an open file object is closed by the file
object's destructor). The MS _locking docs say:
Regions should be locked only briefly and should be unlocked before
closing a file or exiting the program.
So if you leave locks sitting around when a file is closed, the behavior is
undefined. I happen to get the same error from this program:
import os, msvcrt
NAME = 'empty.txt'
size = os.path.getsize(NAME)
f = open(NAME, 'r')
msvcrt.locking(f.fileno(), msvcrt.LK_UNLCK, size)
C:\Code\python\PCbuild>python mlock.py
Traceback (most recent call last):
File "mlock.py", line 7, in ?
msvcrt.locking(f.fileno(), msvcrt.LK_UNLCK, size)
IOError: [Errno 13] Permission denied
In other words, it's acting the same as if the file had never been locked.
> ...
> Now, why am I getting this Permission denied error?
The MS docs aren't clear, but what you're trying to do goes beyond even the
little they do say will work.
> ...
> (OK, I did put a seek(0) command in there. It didn't used to be there
> and I was getting the same error, so I though maybe I should set the
> file pointer to the beginning of the file each time?
The file pointer is always 0 immediately after an open for read, so the seek
wasn't needed; but it shouldn't hurt either.
> ...
> If someone can give me some tips here, I'd really appreciate it.
Cross-platform file-locking is extremely difficult, in part because Python
is coded in C and C defines no facilities for doing it. I don't know why
you're trying to lock files, but the evidence suggests you're trying to use
msvcrt.locking for something it was never intended to do (it was intended
for programs like databases, where multiple threads and even processes
cooperatively both read and write to a single file, and sometimes need
exclusive access for brief periods of time to small regions of the file --
that's why there's a "size" argument).
Sheila King
wrote in comp.lang.python in article
:[Sheila King]
:> I'm trying to do some file locking on Windows. I'm running Win98, but I
:> hope that the methods in the msvcrt module will work OK on any win32?
:
:msvcrt directly exposes some library functions supplied by, and unique to,
:Microsoft. So, like it or not (I don't like it myself <wink>), Microsoft is
:the only entity who can answer questions about them with authority. FWIW,
:the docs MS supplied with MSVC 6 say their _locking function works under
:"Win 95, Win NT", which *usually* means "everything after Win 3.1".
Thanks. I don't have a copy of MSVC 6, so it is hard for me to find the
information on this. This past week I spent a good hour or so at the
microsoft.com's developers site, searching for more definitive info. I
really didn't find anything that helped *me*, but that could be just
because it's *me*.
:Also as usual, the functions MS provides as part of their C library are a
:feeble subset of what you can do by using the Win32 API directly, in this
:case by using the Win32 LockFile and LockFileEx functions. I don't know
:whether win32all exposes those specific functions; core Python does not.
I've been pouring over the Active State site, as well. The win32
extensions do, apparently, provide access to the LockFile and LockFileEx
functions.
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/65203
It's just that I was hoping to accomplish this without requiring the use
of the win32 extensions. (Maybe it's silly of me, but I tend to want to
avoid requiring others to install stuff to use the code.)
...<snipped>...
:> class lockobject:
OK, first of all, thank you for the discussion above. It gives me a
tremendous insight into what I've been doing wrong and how to handle the
problem.
Second of all, I feel that I am at a tremendous disadvantage, as I don't
have access to the MS documentation that you keep referring to. Here's
what I have access to:
http://www.python.org/doc/current/lib/msvcrt-files.html
Of course, I was only intending to leave the files locked very briefly,
but I didn't realize that having them close after being locked would be
a problem. The Python documentation on this module is even more brief
than the MS documentation that you keep referring to as overly brief and
inadequate. (Is everyone who tries to use Python under Windows expected
to have access to the MS VC 6 documentation? If so, where is it
available?)
:Cross-platform file-locking is extremely difficult, in part because Python
:is coded in C and C defines no facilities for doing it. I don't know why
:you're trying to lock files, but the evidence suggests you're trying to use
:msvcrt.locking for something it was never intended to do (it was intended
:for programs like databases, where multiple threads and even processes
:cooperatively both read and write to a single file, and sometimes need
:exclusive access for brief periods of time to small regions of the file --
:that's why there's a "size" argument).
I am intending to use this for programs that will access databases.
Specifically, CGI scripts using the Berkeley data base hash. The
Berkeley Sleepy Cat that I have access to doesn't support concurrent
write access. I have to handle the file locking myself.
What you see above is the confusion of someone who is trying to learn
about file locking and handling concurrent access at the same time that
she is attempting to learn how Python handles it AND how to do it cross
platform. No more. No less.
Anyhow, I thank you for your examples. And your quoting from the MS VC 6
documentation. It has helped me immensely.
Tim Peters
> Thanks. I don't have a copy of MSVC 6, so it is hard for me to find the
> information on this.
It's almost impossible, alas. Here are the MS _locking docs (you'll
probably have to paste this URL back together by hand):
<http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vccore98/h
tml/_crt__locking.asp>
> This past week I spent a good hour or so at the microsoft.com's
> developers site, searching for more definitive info. I really didn't
> find anything that helped *me*, but that could be just because it's
> *me*.
I'm afraid an hour is nothing, Sheila -- there are gigabytes of info
available there, and it can take months just to become comfortable finding
your way around it.
> I've been pouring over the Active State site, as well. The win32
> extensions do, apparently, provide access to the LockFile and LockFileEx
> functions.
> http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/65203
>
> It's just that I was hoping to accomplish this without requiring the use
> of the win32 extensions. (Maybe it's silly of me, but I tend to want to
> avoid requiring others to install stuff to use the code.)
Sure! You need to understand, though, that Python is coded in C, and C is
of no help here: C defines *nothing* about file locking. That means
there's nothing Python can build on cross-platform either. Even if the only
thing you cared about were Linux systems, you could spend many weeks
learning all the subtleties involved with file-locking on it alone. It's a
mess.
You're trying to solve a very difficult problem. I don't know whether the
recipe you found is reliable, but years of experience tell me that anything
that short won't work as intended in all cases (for example, I see the
recipe builds on flock on Unixish boxes, and that may be useless if a
network file system gets involved).
> ...
> OK, first of all, thank you for the discussion above. It gives me a
> tremendous insight into what I've been doing wrong and how to handle the
> problem.
>
> Second of all, I feel that I am at a tremendous disadvantage, as I don't
> have access to the MS documentation that you keep referring to. Here's
> what I have access to:
> http://www.python.org/doc/current/lib/msvcrt-files.html
You'll find that the MS page I linked to above doesn't say much more than
that. And we're in an impossible bind here: the MS docs are copyrighted,
so we can't just reproduce them in the Python docs. In addition, we don't
get to see MS's source code, so the only way we have to fill in the gaps is
the same way *you* have: try things at random, see what happens, and try to
*guess* whether that's reliable. This is impossible, so we shuffle the
MS-specific C functions into their own module, give it an unlikely name, and
say "here it is, if you want it -- but you're on your own". And you are,
and there's no realistic alternative.
> ...
> (Is everyone who tries to use Python under Windows expected
> to have access to the MS VC 6 documentation?
Of course not. But "msvcrt" means MicroSoft Visual C++ RunTime, and if you
use *that* obscure module, then, yes, you had better know what you're doing.
msvcrt is a module for people comfortable with MSVC; it's not suitable for
others. Indeed, there's *no* function in that module I'd recommend for
non-expert use.
> ...
> What you see above is the confusion of someone who is trying to learn
> about file locking and handling concurrent access at the same time that
> she is attempting to learn how Python handles it AND how to do it cross
> platform. No more. No less.
Python doesn't handle file locking. Perl pretends to <wink>. If I were you
I'd try another approach (for example, if your program has control over
access to the files of interest, you can invent your own locking
*abstraction* in software and leave painful OS-specific gimmicks out of it
entirely).
Bengt Richter
>[Sheila King]
>> I'm trying to do some file locking on Windows. I'm running Win98, but I
>> hope that the methods in the msvcrt module will work OK on any win32?
>
>msvcrt directly exposes some library functions supplied by, and unique to,
>Microsoft. So, like it or not (I don't like it myself <wink>), Microsoft is
>the only entity who can answer questions about them with authority. FWIW,
>the docs MS supplied with MSVC 6 say their _locking function works under
>"Win 95, Win NT", which *usually* means "everything after Win 3.1".
>
>Also as usual, the functions MS provides as part of their C library are a
>feeble subset of what you can do by using the Win32 API directly, in this
>case by using the Win32 LockFile and LockFileEx functions. I don't know
>whether win32all exposes those specific functions; core Python does not.
>
I'm wondering if _locking requires using _sopen? The Python builtin open uses
fopen as part of creating the file object, I believe. Example code in
the msvc++6.0 help re _locking uses _sopen, which provides shared access control.
Bengt Richter
>On Sat, 25 Aug 2001 23:25:42 -0400, "Tim Peters" <tim...@home.com>
>wrote in comp.lang.python in article
><mailman.998796398...@python.org>:
>
>:[Sheila King]
>:> I'm trying to do some file locking on Windows. I'm running Win98, but I
>:> hope that the methods in the msvcrt module will work OK on any win32?
>:
>:msvcrt directly exposes some library functions supplied by, and unique to,
>:Microsoft. So, like it or not (I don't like it myself <wink>), Microsoft is
>:the only entity who can answer questions about them with authority. FWIW,
>:the docs MS supplied with MSVC 6 say their _locking function works under
>:"Win 95, Win NT", which *usually* means "everything after Win 3.1".
>
http://starship.python.net/crew/jjkunce/python/ntposixfile.py
might be useful?
Note from my other post: Microsoft example code for _locking uses
_sopen rather than fopen (or _open). The s in _sopen is for shared
file access, so I wonder if python's builtin open is affected, since
it uses fopen.
The ntposixfile.py seems to go to __builtin__.open() as well, though,
so if _sopen does something different and required, the same problem
should appear. Hard to believe there wouldn't be more noise on google
about it though, if _locking didn't work with fopen, unless people
just quietly give up and go on to the win32 api...
Sheila King
wrote in comp.lang.python in article
:http://starship.python.net/crew/jjkunce/python/ntposixfile.py
:
:might be useful?
:Note from my other post: Microsoft example code for _locking uses
:_sopen rather than fopen (or _open). The s in _sopen is for shared
:file access, so I wonder if python's builtin open is affected, since
:it uses fopen.
:
:The ntposixfile.py seems to go to __builtin__.open() as well, though,
:so if _sopen does something different and required, the same problem
:should appear. Hard to believe there wouldn't be more noise on google
:about it though, if _locking didn't work with fopen, unless people
:just quietly give up and go on to the win32 api...
I did download and try that code a couple of days ago. (Note that it
says, it hasn't really been tested thoroughly.)
Anyhow, when I tried to run the master/slave __main__ code in two
separate DOS console windows to test it, I got the "Permission denied"
errors using that module.
Sheila King
wrote in comp.lang.python in article
:[Sheila King]
:> I've been pouring over the Active State site, as well. The win32
:> extensions do, apparently, provide access to the LockFile and LockFileEx
:> functions.
:> http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/65203
:>
:> It's just that I was hoping to accomplish this without requiring the use
:> of the win32 extensions. (Maybe it's silly of me, but I tend to want to
:> avoid requiring others to install stuff to use the code.)
:
:Sure! You need to understand, though, that Python is coded in C, and C is
:of no help here: C defines *nothing* about file locking.
I understand that.
: That means
:there's nothing Python can build on cross-platform either. Even if the only
:thing you cared about were Linux systems, you could spend many weeks
:learning all the subtleties involved with file-locking on it alone. It's a
:mess.
<sigh> Thanks for the encouragement. No, I know you're just trying to be
honest about the whole thing.
:You're trying to solve a very difficult problem. I don't know whether the
:recipe you found is reliable, but years of experience tell me that anything
:that short won't work as intended in all cases (for example, I see the
:recipe builds on flock on Unixish boxes, and that may be useless if a
:network file system gets involved).
OK, well I guess I'm not going to worry about the networking thing. For
one, it's beyond me. And for two, I don't expect my code to need to
access files remotely from the machine it is running on.
:Of course not. But "msvcrt" means MicroSoft Visual C++ RunTime, and if you
:use *that* obscure module, then, yes, you had better know what you're doing.
:msvcrt is a module for people comfortable with MSVC; it's not suitable for
:others. Indeed, there's *no* function in that module I'd recommend for
:non-expert use.
OK, so if we went with the win32all Win API extensions, are these better
documented and better understood ? Would it be possible to do more
reliable file locking with those libraries?
As you will see in this article (when it becomes available on Google):
http://groups.google.com/groups?as_umsgid=%3C0q4iotg9ld08s7850vv3dqfi2hnbefv015%404ax.com%3E
apparently msvcrt doesn't allow one to acquire a non-blocking lock?
:> ...
:> What you see above is the confusion of someone who is trying to learn
:> about file locking and handling concurrent access at the same time that
:> she is attempting to learn how Python handles it AND how to do it cross
:> platform. No more. No less.
:
:Python doesn't handle file locking. Perl pretends to <wink>. If I were you
:I'd try another approach (for example, if your program has control over
:access to the files of interest, you can invent your own locking
:*abstraction* in software and leave painful OS-specific gimmicks out of it
:entirely).
For CGI, I need file locking. The alternatives are (as I understand
them):
1. Have a server running, which controls access to the files, thus
ensuring that only one process can write to the files at a time.
2. Something dealing with sockets, where the CGI script tries to see if
a certain socket is listening to determine whether it has permission to
write to the file...details are hazy and I'm not going to look this one
up again right now.
3. Check whether a certain file exists to determine whether or not the
process can have permission to write to the files.
The problems with # 1 and 2, is that it requires a process running on
the web server all the time, and I am not allowed to do that on the
shared, community server where I host my website. (I don't think most
people who do not run a dedicated server are allowed to have processes
running in the background on the server.)
The problem with #3, is I've heard that is unreliable as well. Problems
with whether the file will be deleted or written if the program crashes.
Then, if the file is still there/missing, subsequent calls to the CGI
script will think that they have permission to write when they don't.
Or, the process may think that it can't have permission, when it can.
Etc...
Since in a CGI situation, you may have many processes running in
parallel, then "my program" doesn't control access to the files, since
there may be many instances of the program. Actually, this is what I
thought to do originally, but After going over # 1, 2, 3 above, I
thought...gee, this isn't going to work. Then, I found a really nifty
example in Mark Lutz Programming Python, Chapter 14 where he does a
shelf database implementation for his Book Errata website. He discusses
file locking in there, but only for a Unix-type platform.
I really don't see any way around file locking for the type of
application I'm trying to write. However, I guess after seeing all the
trouble with the msvcrt module, and how it doesn't even seem to allow
shared locking access to a file, that I will go ahead and use the
win32all extensions. I'm supposing that will allow better control?
Sheila King
wrote in comp.lang.python in article
:I'm wondering if _locking requires using _sopen? The Python builtin open uses
:fopen as part of creating the file object, I believe. Example code in
:the msvc++6.0 help re _locking uses _sopen, which provides shared access control.
Hmm. This may be the case???
In an interactive session, when I try to acquire two non-blocking locks:
>>> import os
>>> filename = r'e:\apache\cgi-bin\pylinks\lock.txt'
>>> size = os.path.getsize(filename)
>>> f = open(filename)
>>> fd = f.fileno()
>>> msvcrt.locking(fd, msvcrt.LK_NBLCK, size)
>>> g = open(filename)
>>> gd = g.fileno()
>>> msvcrt.locking(gd, msvcrt.LK_NBLCK, size)
Traceback (most recent call last):
File "<interactive input>", line 1, in ?
IOError: [Errno 13] Permission denied
>>>
--
Ignacio Vazquez-Abrams
> I am intending to use this for programs that will access databases.
> Specifically, CGI scripts using the Berkeley data base hash. The
> Berkeley Sleepy Cat that I have access to doesn't support concurrent
> write access. I have to handle the file locking myself.
I'm just curious as to why you're not using a database server for this. Any
specific reason?
--
Ignacio Vazquez-Abrams <ign...@openservices.net>
Sheila King
<ign...@openservices.net> wrote in comp.lang.python in article
<mailman.998845240...@python.org>:
:On Sun, 26 Aug 2001, Sheila King wrote:
:
:> I am intending to use this for programs that will access databases.
:> Specifically, CGI scripts using the Berkeley data base hash. The
:> Berkeley Sleepy Cat that I have access to doesn't support concurrent
:> write access. I have to handle the file locking myself.
:
:I'm just curious as to why you're not using a database server for this. Any
:specific reason?
Because I don't have access to one. (Not everyone does, you know.)
In my case, I could spend extra $$$ (on the order of $20 extra per month
over what I'm spending for hosting now) to get a mySQL database, but I
can't do that right now. (Probably some day, I will.)
I looked at Gadfly, but it doesn't support concurrent access, and
requires running a server. I can't have a process running all the time
in the background (as I mentioned in another post that you may not have
seen yet), so running it all the time is out. And starting up the server
every time I want to run a CGI script seems prohibitive (too much time
and CPU usage).
Even if I managed to get access to a database server, there'd be someone
else who wanted to do what I'm doing. It's quite common in CGI scripts.
I'm guessing, because of the frequent references to the fact that Perl
only *appears* to provide cross-platform file locking, that many of the
cgi scripts that are out there don't actually work as the author
intended.
Ignacio Vazquez-Abrams
> On Sun, 26 Aug 2001 12:59:09 -0400 (EDT), Ignacio Vazquez-Abrams
> <ign...@openservices.net> wrote in comp.lang.python in article
> <mailman.998845240...@python.org>:
>
> :On Sun, 26 Aug 2001, Sheila King wrote:
> :
> :> I am intending to use this for programs that will access databases.
> :> Specifically, CGI scripts using the Berkeley data base hash. The
> :> Berkeley Sleepy Cat that I have access to doesn't support concurrent
> :> write access. I have to handle the file locking myself.
> :
> :I'm just curious as to why you're not using a database server for this. Any
> :specific reason?
>
> Because I don't have access to one. (Not everyone does, you know.)
That's fine; I understand. I was just curious, that's all.
> In my case, I could spend extra $$$ (on the order of $20 extra per month
> over what I'm spending for hosting now) to get a mySQL database, but I
> can't do that right now. (Probably some day, I will.)
>
> I looked at Gadfly, but it doesn't support concurrent access, and
> requires running a server. I can't have a process running all the time
> in the background (as I mentioned in another post that you may not have
> seen yet), so running it all the time is out. And starting up the server
> every time I want to run a CGI script seems prohibitive (too much time
> and CPU usage).
>
> Even if I managed to get access to a database server, there'd be someone
> else who wanted to do what I'm doing. It's quite common in CGI scripts.
> I'm guessing, because of the frequent references to the fact that Perl
> only *appears* to provide cross-platform file locking, that many of the
> cgi scripts that are out there don't actually work as the author
> intended.
>
> --
> Sheila King
> http://www.thinkspot.net/sheila/
> http://www.k12groups.org/
I was just poking around in the Sleepycat documentation, and apparently they
have some sort of locking scheme via their C API. Which version on Sleepycat
do you currently have access to, and how are you accessing it? You'll have to
excuse my naïveté when it comes to file-based databases.
--
Ignacio Vazquez-Abrams <ign...@openservices.net>
Sheila King
:I was just poking around in the Sleepycat documentation, and apparently they
:have some sort of locking scheme via their C API. Which version on Sleepycat
:do you currently have access to, and how are you accessing it? You'll have to
:excuse my naïveté when it comes to file-based databases.
SleepyCat 2.4.x, or something like that.
I'm accessing it via Python. Doesn't have file locking, like I mentioned
(somewhere I mentioned that...to many messages on this topic now...)
Here are the docs on how to access it via Python:
http://www.python.org/doc/current/lib/module-dbhash.html
Notice that you can append an optional 'l' to the mode, if your
installation supports locking.
Here is an interactive session:
Python 2.1.1 (#6, Aug 16 2001, 17:02:08)
[GCC egcs-2.91.66 19990314/Linux (egcs-1.1.2 release)] on linux2
Type "copyright", "credits" or "license" for more information.
>>> import dbhash
>>> db = dbhash.open('rec', 'rl')
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "/big/dom/xthinkspot/lib/python2.1/dbhash.py", line 16, in open
return bsddb.hashopen(file, flag, mode)
bsddb.error: locking not supported on this platform
Sheila King
in comp.lang.python in article
<pm5iotkp8d0q00kpq...@4ax.com>:
:
:I really don't see any way around file locking for the type of
:application I'm trying to write. However, I guess after seeing all the
:trouble with the msvcrt module, and how it doesn't even seem to allow
:shared locking access to a file, that I will go ahead and use the
:win32all extensions. I'm supposing that will allow better control?
It appears, after a bit more research, that this will not work for all
win32 platforms, anyway. Apparently unsupported on Win95/98/Me:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/hh/winbase/filesio_39h4.asp
(scroll down to just below where it says "requirements")
Sheila King
in comp.lang.python in article
<0q4iotg9ld08s7850...@4ax.com>:
:On Sun, 26 Aug 2001 10:23:22 GMT, bo...@accessone.com (Bengt Richter)
:>>>
Upon reading the Python docs for the msvcrt module, yet another time,
and thinking about this more carefully...
(The docs say, in part:
"
LK_LOCK
LK_RLCK
Locks the specified bytes. If the bytes cannot be locked, the program
immediately tries again after 1 second. If, after 10 attempts, the bytes
cannot be locked, IOError is raised.
LK_NBLCK
LK_NBRLCK
Locks the specified bytes. If the bytes cannot be locked, IOError is
raised.
"
)
It appears the LK_NBLCK tries for a blocking lock. Tries once, gives up
if not successful. Kind of strange they named it NBLCK. Seems like
non-blocking lock, but doesn't behave that way.
Whereas, LK_LOCK is a non-blocking lock. Well, tries 10 times, anyhow.
Appears that, via the msvcrt module, only exclusive locks are available,
no shared locks. From other reading I've done,
LK_LOCK and LK_RLOCK are the same. (One might think LK_RLOCK would be
fore a read lock, thus a shared lock, but that is not the case.)
OK, I think I'm OK, with this. I just won't try for a lock for reading
processes. I'll have some file there, that as long as it exists, reading
the file can proceed. Writers will delete that file before trying to
obtain an exclusive lock on the file. Then, when they unlock the file,
they can create that reading permission file again.
Richard van de Stadt
[...]
>
> For CGI, I need file locking. The alternatives are (as I understand
> them):
>
> 1. Have a server running, which controls access to the files, thus
> ensuring that only one process can write to the files at a time.
I missed the start of this thread, but doesn't then this work for you:
def oneAtTheTime():
f = posixfile.open (someFilename, 'a')
f.lock ('w|')
f.write ('something') # or do something else that no other process may do at this time
f.lock ('u')
f.close()
I've been using this for over 5 years. It controls access to a particular
file for a system that collects submissions for conferences, where most
submissions have to be dealt with very close to the deadline. (Designed
in order to deals with multiple webform submissions, it also saved me a
couple of times when a fileserver broke down. The submissions where
simply blocked on the webserver until the fileserver came up again a
few days later :-)
Richard.
Ignacio Vazquez-Abrams
> Notice that you can append an optional 'l' to the mode, if your
> installation supports locking.
>
> Here is an interactive session:
>
> Python 2.1.1 (#6, Aug 16 2001, 17:02:08)
> [GCC egcs-2.91.66 19990314/Linux (egcs-1.1.2 release)] on linux2
> Type "copyright", "credits" or "license" for more information.
> >>> import dbhash
> >>> db = dbhash.open('rec', 'rl')
> Traceback (most recent call last):
> File "<stdin>", line 1, in ?
> File "/big/dom/xthinkspot/lib/python2.1/dbhash.py", line 16, in open
> return bsddb.hashopen(file, flag, mode)
> bsddb.error: locking not supported on this platform
> >>>
>
> --
> Sheila King
> http://www.thinkspot.net/sheila/
> http://www.k12groups.org/
Hmmm...
The only thing I can think of is to get a copy of libdb.dll 3.3.1 from
somewhere (I checked Google and couldn't find it, and I left my copy of
VC++ 6.0 in my other pants...), put it in your local directory, and try again.
DLLs, like Python modules, check the current directory before trying the other
standard directories.
--
Ignacio Vazquez-Abrams <ign...@openservices.net>
Sheila King
<st...@cs.utwente.nl> wrote in comp.lang.python in article
<3B8965A9...@cs.utwente.nl>:
:Sheila King wrote:
:[...]
:>
:> For CGI, I need file locking. The alternatives are (as I understand
:> them):
:>
:> 1. Have a server running, which controls access to the files, thus
:> ensuring that only one process can write to the files at a time.
:[...]
:
:I missed the start of this thread, but doesn't then this work for you:
:
:def oneAtTheTime():
: f = posixfile.open (someFilename, 'a')
: f.lock ('w|')
: f.write ('something') # or do something else that no other process may do at this time
: f.lock ('u')
: f.close()
Well, maybe it would if I were content to run only on a posix system. I
was trying to do something cross-platform (Unix/Windows).
:I've been using this for over 5 years. It controls access to a particular
:file for a system that collects submissions for conferences, where most
:submissions have to be dealt with very close to the deadline. (Designed
:in order to deals with multiple webform submissions, it also saved me a
:couple of times when a fileserver broke down. The submissions where
:simply blocked on the webserver until the fileserver came up again a
:few days later :-)
Maybe you need to update your code. The docs on the posix module (here:
http://www.python.org/doc/current/lib/module-posixfile.html )
state:
"
Note: This module will become obsolete in a future release. The locking
operation that it provides is done better and more portably by the
fcntl.lockf() call.
Sheila King
:On Sun, 26 Aug 2001, Sheila King wrote:
:
:> Notice that you can append an optional 'l' to the mode, if your
:> installation supports locking.
:>
:> Here is an interactive session:
:>
:> Python 2.1.1 (#6, Aug 16 2001, 17:02:08)
:> [GCC egcs-2.91.66 19990314/Linux (egcs-1.1.2 release)] on linux2
:> Type "copyright", "credits" or "license" for more information.
:> >>> import dbhash
:> >>> db = dbhash.open('rec', 'rl')
:> Traceback (most recent call last):
:> File "<stdin>", line 1, in ?
:> File "/big/dom/xthinkspot/lib/python2.1/dbhash.py", line 16, in open
:> return bsddb.hashopen(file, flag, mode)
:> bsddb.error: locking not supported on this platform
:> >>>
:
:Hmmm...
:
:The only thing I can think of is to get a copy of libdb.dll 3.3.1 from
:somewhere
libdb.dll 3.3.1 for what...SleepyCat? No, I don't want to install
another version on top of the system version. I am going to stick with
the default system install for the bsddb. (I've already had plenty of
fun installing my own version of Python instead of using the system
default of 1.5.2.)
bsddb works fine on this system, if I don't try to do any file locking:
Python 2.1.1 (#6, Aug 16 2001, 17:02:08)
[GCC egcs-2.91.66 19990314/Linux (egcs-1.1.2 release)] on linux2
Type "copyright", "credits" or "license" for more information.
>>> import dbhash
>>> db = dbhash.open('rec', 'r')
>>> for key in db.keys():
... print key
...
num
name
age
>>> db.close()
>>>
You will notice that SleepyCat has three versions:
http://sleepycat.com/products.html
and I believe our system just has the lowest of these installed.
: (I checked Google and couldn't find it, and I left my copy of
:VC++ 6.0 in my other pants...),
LOL. Heh, now I'm getting confused. VC++ isn't going to help me with
using bsddb on my webhost. That's a Linux machine.
What I want is to write a script that I can run on my webhost (Linux)
and also on my home machine (Win98) for testing/development.
Anyhow, I think I've got things figured out now.
:put it in your local directory, and try again.
:DLLs, like Python modules, check the current directory before trying the other
:standard directories.
Thanks,
Ignacio Vazquez-Abrams
> LOL. Heh, now I'm getting confused. VC++ isn't going to help me with
> using bsddb on my webhost. That's a Linux machine.
Oh heck, you shoulda just said that from the outset. find out what version of
glibc ('ls /lib/libc-*') and what processor ('uname -a') they have and I'll
build you a bsddbmodule.so against Sleepycat 3.3.1 that you can use.
--
Ignacio Vazquez-Abrams <ign...@openservices.net>
Mark Hammond
It appears that LockFileEx is designed specifically for files that use
"overlapped IO". Indeed, the "portalocker" recipe explicitly uses
overlapped IO structures.
Try using LockFile - the MSDN documentation implies that it is supported
on all Win32 platforms, and the error you get:
pywintypes.api_error: (120, 'LockFileEx', 'This function is only valid
in Win32 mode.')
implies you are running on Win9x.
A quick look at portalocker implies you will also want to drop all
references to "_overlapped" - replace them with "None".
Mark.
Bengt Richter
[...]
I'll assume that when you say 'file locking', you mean the file as
a whole, rather than a region of the file, as the parameters for _locking
specify (I think an example of yours had parameters for the whole
file length).
Have you considered using os.rename() to implement your method #3 above?
Barring a system crash, renaming should be atomic, and even if there
is a crash after renaming, you should be able to detect orphan files
if you put appropriate info in the new name.
I'll stick my neck out here with a pretty untested sketch:
Here is a sample interactive sequence (there's a file 'xxx'
in the current directory) ...
Python 2.1 (#15, Apr 16 2001, 18:25:49) [MSC 32 bit (Intel)] on win32
Type "copyright", "credits" or "license" for more information.
>>> from LockedFile import LockedFile
>>> flock=LockedFile('xxx','r')
>>> if flock.open():
... flock.f.read()
... else:
... print flock.lasterror
...
'This is xxx with no extension\n'
>>> glock=LockedFile('xxx','r')
>>> if glock.open():
... glock.f.read()
... else:
... print glock.lasterror
...
[Errno 13] Permission denied trying to lock "xxx.998881229"
>>> del flock
>>> if glock.open():
... glock.f.read()
... else:
... print glock.lasterror
...
'This is xxx with no extension\n'
>>> glock.lockedName
'xxx.998881516'
>>> glock.f
<open file 'xxx.998881516', mode 'r' at 007F74F0>
>>> del glock
That Errno 13 was because I was slow, and the locked
file looked orphaned after 10 seconds, but it was still
open via the other open.
Note that calling glock.open() doesn't do anything if
it's already open, except clear the last error. If it
was locked but not orphaned yet, it should return None,
but with no lasterror (needs testing). Thus you might
be able to do glock.open() and sleep a little to let others free
the file. 10 seconds is a long time though, but be careful
with 1-second resolution, since a difference of 1 can mean
a microsecond or a microsecond short of two seconds, depending
on when the ticks happened in reality.
There's a mutex module that might be of interest too.
And you might want to change this thing to use all lowlevel
io instead of the builtin open and file object. You might
be able to get shared reading without error.
I don't know how/whether this incomplete workaround thing
fits your use, but HTH get you closer.
________________________________________________________
# LockedFile.py -- a class for locking by renaming, then opening a file
# Copyright (C) 2001 Bengt Richter. All rights reserved.
# Released under Python license, with NO WARRANTIES EXPRESSED OR IMPLIED.
# Version 0.01 alpha -- just a sketch, rework to suit ;-)
#
MAXLOCKSECONDS = 10 # max time to hold lock, in seconds (resolution 1 sec)
import os, glob, time
class LockedFile:
def __init__(self, fullPath, fmode='r',maxhold=MAXLOCKSECONDS):
self.lasterror = None
self.f = None
self.fdir = fullPath.split(os.sep)
self.fname = self.fdir[-1]
self.fdir = os.sep.join(self.fdir[:-1])
if self.fdir:
self.fdir += os.sep
self.fmode = fmode
self.open()
def open(self):
self.lasterror = None
if self.f: return self.f
self.lockedName = self.fname+'.'+str(long(time.time()))
fpath = self.fdir+self.fname #pathname for unlocked file
if not os.access(fpath,os.F_OK): # just checks existence
# check for orphan over MAXLOCKSECONDS seconds old
oldDir = os.getcwd()
if self.fdir: os.chdir(self.fdir) # want names w/o paths
orphan = glob.glob(self.fname+'.*')
os.chdir(oldDir)
if len(orphan) != 1:
self.lasterror = '%d items matched "%s.*' % (len(orphan),self.fname)
return None
orphan = orphan[0]
torph = orphan.split('.')[-1]
if not torph.isdigit():
self.lasterror = '"%s" has bad lock suffix' % orphan
return None
torph = long(torph)
tnow = long(time.time())
if tnow-torph > MAXLOCKSECONDS:
# assume ok to take over orphan locked file
fpath = self.fdir + orphan
else:
return None #no, error, just locked, so f remains None
# now try to lock the file at fpath
try:
os.rename(fpath, self.lockedName)
self.f = open(self.fdir+self.lockedName, self.fmode)
return self.f
except OSError,e:
self.lasterror = str(e) +' trying to lock "%s"' % fpath
def close(self):
if self.f:
self.f.close() # close the locked file
self.f = None
# try to rename it to its unlocked name
try:
os.rename(self.fdir+self.lockedName, self.fname)
except OSError,e:
self.lasterror = e.errmsg
def __del__(self):
self.close()
________________________________________________________
Sheila King
wrote in comp.lang.python in article
:I'll assume that when you say 'file locking', you mean the file as
:a whole, rather than a region of the file, as the parameters for _locking
:specify (I think an example of yours had parameters for the whole
:file length).
Yes, this is what I was going to do. I wasn't going to lock the file I
want to read to/write from itself, but a 'lockfile' and then, when I
obtain the lock on the lockfile, I would read/write to the datafile
(because underlying implementations of databases vary, and databases are
sometimes comprised of more than one file).
:Have you considered using os.rename() to implement your method #3 above?
:Barring a system crash, renaming should be atomic, and even if there
:is a crash after renaming, you should be able to detect orphan files
:if you put appropriate info in the new name.
Actually, I had thought of that. I had been discussing it with my
husband this afternoon. He didn't think it was a great idea, but he
thought it had possibilities. It makes me feel a bit better, to hear
someone else suggest it.
Actually, I think what happens with Windows98, is that there is only
exclusive lock. No shared locking (which is what one would want for
reading). So, I could change the name of the file when a write process
wants to acquire it. Otherwise, if the un-renamed file is there, it is
OK to read from it. Do you think this type of approach would work for
the reading part only?
:I'll stick my neck out here with a pretty untested sketch:
Thanks. I will look this code over. I really appreciate the suggestions.
[code snipped]
Alex Martelli
news:mailman.998814219...@python.org...
...
> so we can't just reproduce them in the Python docs. In addition, we don't
> get to see MS's source code, so the only way we have to fill in the gaps
is
The source code for MSVCRT.DLL comes with every installation
of VC++6 (perhaps not with the cheap one for beginners, but
surely Professional and up). You don't get to see the sources
for the underlying operating systems (quite likely very
different ones for the 95/98/ME lineage vs the NT/2000/XP
one, of course:-), but the C runtime's sources often help.
Alex
David Bolen
> I'm trying to do some file locking on Windows. I'm running Win98, but I
> hope that the methods in the msvcrt module will work OK on any win32?
After reading this thread, it appears that you probably want more fine
grained locking, but just in case you do end up falling back to some
sort of sentinel file or more global lock approach...
An alternative to the msvcrt lock stuff for a full file lock that may
be more portable among Windows platforms would be to just open your
lock file using the native Win32 CreateFile call (wrapped in
win32file), and ask for exclusive access (no sharing mode). If you
need to do I/O to the lock file you'll probably need to use the Win32
functions (although you could wrap that in a file-like object), but if
you're just using that file to restrict access to other files it can
be a convenient method.
--
-- David
--
/-----------------------------------------------------------------------\
\ David Bolen \ E-mail: db...@fitlinxx.com /
| FitLinxx, Inc. \ Phone: (203) 708-5192 |
/ 860 Canal Street, Stamford, CT 06902 \ Fax: (203) 316-5150 \
\-----------------------------------------------------------------------/
Sheila King
comp.lang.python in article <uu1ytr...@ctwd0143.fitlinxx.com>:
:Sheila King <she...@spamcop.net> writes:
:
:> I'm trying to do some file locking on Windows. I'm running Win98, but I
:> hope that the methods in the msvcrt module will work OK on any win32?
:
:After reading this thread, it appears that you probably want more fine
:grained locking, but just in case you do end up falling back to some
:sort of sentinel file or more global lock approach...
No, I wasn't trying to do fine grained locking. I was going to use a
sentinel file.
:An alternative to the msvcrt lock stuff for a full file lock that may
:be more portable among Windows platforms would be to just open your
:lock file using the native Win32 CreateFile call (wrapped in
:win32file), and ask for exclusive access (no sharing mode).
Thanks. I'm looking at that, and it looks like it might be a very good
approach.
:If you need to do I/O to the lock file you'll probably need to use the Win32
:functions (although you could wrap that in a file-like object),
Fortunately, I'm not planning any I/O on the locked file.
:but if you're just using that file to restrict access to other files it can
:be a convenient method.
This is exactly what I'm trying to do. Thanks, again, for the
suggestion.
Tim Peters
> The source code for MSVCRT.DLL comes with every installation
> of VC++6 (perhaps not with the cheap one for beginners, but
> surely Professional and up). You don't get to see the sources
> for the underlying operating systems (quite likely very
> different ones for the 95/98/ME lineage vs the NT/2000/XP
> one, of course:-), but the C runtime's sources often help.
Not much in this case -- _locking gets mapped into LockFile, but restricted
to 32-bit offsets and lengths. Since we don't have the source for LockFile,
no progress.
Bengt Richter
>exclusive lock. No shared locking (which is what one would want for
>reading). So, I could change the name of the file when a write process
>wants to acquire it. Otherwise, if the un-renamed file is there, it is
>OK to read from it. Do you think this type of approach would work for
>the reading part only?
>
2+n states of a file: (1) not open for anything (2) open for readonly
n>0 times (3) open for exclusive writing.
A prospective writer must be able to distinguish (1) from (2).
Renaming only by writer would be ok to stop *additional* prospective
readers from opening the file but the prospective writer must also
have a way of knowing when all *current* readers have closed the file.
BTW, note that renaming will fail with permission denied on win32
if the file is open, but not so on unix with its separation
of names vs inode file info, so on unix a file could be in state
(2) and you'd have no indication if you leave the name unchanged
for (1) and (2), and don't have another indication.
A writer has to be able watch for the count of readers to go to zero
one way or another. Perhaps you could have readers create individual
expiring read-lock files named by appending their os.pid() numbers
plus time tag to the name of the file to lock. Then the writer could
watch for all those to disappear or expire.
This only applies for high traffic on win32, but since on win32 the
writer can't assume ability to rename as a mechanism for preventing
additional readers when the file is already open (it get permission
denied, unlike unix), perhaps if the writer process creates its own
write-lock file as a first step, then potential readers could defer
creating their read-lock files if they see that, and existing ones
would cycle out or expire, and the writer could rename and open the file.
For unix the write-lock file would be a waste, since renaming will succeed
unless the file is in orphan form. And unless you are fighting a humongous
swarm of readers all interested in one file on win32, it shouldn't hurt
to leave it out there either.
Note that the atomicity of renaming is critical to making this work. E.g.,
reader writer
-------- ------------
[ck wlock] . <-not really needed except high traffic win32?
. [set wlock] <-ditto
. rename <-not safe to eliminate
. ck rlocks
. open for w
set rlock .
open for r .
. write
read ??
If writer renames or attempts to rename the base file just before
checking rlocks, then the reader's open will fail or succeed depending on
whether it happened before the rename. If the open for r succeeds, it means
that it has also set rlock before the rename, so the writer will see that and
not open for w. If it fails, then the reader failed to complete access and
should delete its rlock (which the writer would be monitoring). In the
open failure case, it would not matter if the failing reader's rlock was seen
or not by the writer. However, any prior reader's rlock that actually
succeeded in opening the file would be seen.
On win32 the rename would raise an exception if the reader had succeeded,
so the rlock check could effectively be accomplished just with that, but
as mentioned, that's not true on unix, so it needs to be in the waiting
loop for win32 with a try/except wrapper, and the rename should not
be repeated if it succeeds, as it normally will on unix.
It'd be nice if the builtin open had an option for exclusive access within
a specified timeout for trying ;-)
HTH
David Bolen
> Thanks. I'm looking at that, and it looks like it might be a very good
> approach.
Ah, well just to kick start things - here's a lock function that one
of my backup scripts uses to lock the database backup files in a
directory before beginning retrieval of the backup to a central
location (to avoid the remote host starting a new backup). Nothing
fancy, just some periodic attempts. Since CreateFile doesn't have the
concept of a timeout to obtain the lock, you have to handle your own
retries. Under heavy load, this does mean in theory it's possible for
one process to starve and never get the lock, depending on the retry
time.
When the process is done, just "win32file.CloseHandle(lock_file)" on
the handle returned from this function.
Since it's an open file lock, if the process owning it happens to die,
the file will still be around, but the next process to try will be
able to open it without problems. To be nice, my backup task does
explicitly try to remove the lock file when done (inside a try/except
clause in case someone else immediately grabs the lock).
- - - - - - - - - - - - - - - - - - - - - - - - -
def LockDB(LockName):
lock_file = None
# Attempt to open lock file to ensure access. We keep trying
# to obtain lock for up to 60 seconds, trying every 10
lock_start = time.time()
lock_current = lock_start
while (1):
try:
lock_file = win32file.CreateFile(LockName,
win32file.GENERIC_WRITE,
0, # Exclusive access
None,
win32file.OPEN_ALWAYS,
0,0)
except:
if (lock_current == lock_start):
traceback.print_exc(0)
print "Waiting for database backup lock file \"%s\"" % \
LockName
if ((lock_current - lock_start) < 60):
time.sleep(10)
lock_current = time.time()
else:
print "Unable to obtain access to database backup lock file"
break
else:
break
return lock_file
- - - - - - - - - - - - - - - - - - - - - - - - -
Bengt Richter
>On 27 Aug 2001 18:08:56 -0400, David Bolen <db...@fitlinxx.com> wrote in
>comp.lang.python in article <uu1ytr...@ctwd0143.fitlinxx.com>:
>
>:Sheila King <she...@spamcop.net> writes:
>:
>:> I'm trying to do some file locking on Windows. I'm running Win98, but I
>:> hope that the methods in the msvcrt module will work OK on any win32?
>:
Looks like I mistook your goal. I thought you were trying to do something
that would work on both windows and linux. Or are you testing os.name and
doing different things?
Sheila King
wrote in comp.lang.python in article
I want something that will work on both posix and win32.
However, I expect to have to code them differently, and then put a
wrapper around that, so it appears that they both have the same
interface.
On the method used for Windows, I wanted to make it especially clear,
that it had to work for all win32 platforms, since sometimes people
suggest methods that only work for NT/2000.
Anyhow, I want to thank everyone who has made suggestions. All of the
suggestions have been helpful. I'm leaning right now to the
win32file.CreateFile method for the windows code, suggested by David
Bolen. I played around with it a bit this afternoon, and it looks
hopeful.
I expect to use the fcntl.flock (or should I use fcntl.lockf ?) from the
fcntl module for the posix platforms. I understand that it won't work in
some situations where the files are across a network instead of on the
same machine as where the script is running, but I'm not going to worry
about that.
So, I will write two separate modules, one for posix, one for windows,
and then write a module that wraps them both, so that it *appears* to be
cross-platform.
Sheila King
wrote in comp.lang.python in article
:On Mon, 27 Aug 2001 06:14:56 GMT, Sheila King <she...@spamcop.net> wrote:
:[...]
:>
:>Actually, I think what happens with Windows98, is that there is only
:>exclusive lock. No shared locking (which is what one would want for
:>reading). So, I could change the name of the file when a write process
:>wants to acquire it. Otherwise, if the un-renamed file is there, it is
:>OK to read from it. Do you think this type of approach would work for
:>the reading part only?
:>
:Yes, but there's a problem, because you need to be able to distinguish
:2+n states of a file: (1) not open for anything (2) open for readonly
:n>0 times (3) open for exclusive writing.
Yes, I had been thinking about this last night and the good part of this
morning, and hadn't come up with a solution, yet. However, thanks to
David Bolen's suggestion, I think I won't have to handle it this way,
after all.
...<snippage>>..
:A writer has to be able watch for the count of readers to go to zero
:one way or another. Perhaps you could have readers create individual
:expiring read-lock files named by appending their os.pid() numbers
:plus time tag to the name of the file to lock. Then the writer could
:watch for all those to disappear or expire.
Right. It would be tricky to code. I had not thought of a good way, yet.
:This only applies for high traffic on win32, but since on win32 the
:writer can't assume ability to rename as a mechanism for preventing
:additional readers when the file is already open (it get permission
:denied, unlike unix), perhaps if the writer process creates its own
:write-lock file as a first step, then potential readers could defer
:creating their read-lock files if they see that, and existing ones
:would cycle out or expire, and the writer could rename and open the file.
:
:For unix the write-lock file would be a waste, since renaming will succeed
:unless the file is in orphan form. And unless you are fighting a humongous
:swarm of readers all interested in one file on win32, it shouldn't hurt
:to leave it out there either.
This is very useful and interesting information for me. I did play a bit
and discover that I could not rename open files on win32, but I could on
Linux. So, I can see that there is quite a bit of difference between how
the file systems work.
:Note that the atomicity of renaming is critical to making this work. E.g.,
Would someone please explain the concept of "atomic" or "atomicity" with
respect to file operations. Do you mean, that you are (below) showing in
detail, step-by-step what would happen at each point in the process that
you suggest?
:reader writer
:-------- ------------
:[ck wlock] . <-not really needed except high traffic win32?
: . [set wlock] <-ditto
: . rename <-not safe to eliminate
: . ck rlocks
: . open for w
:set rlock .
:open for r .
: . write
:read ??
:
...<other good info snipped>...
:
:On win32 the rename would raise an exception if the reader had succeeded,
:so the rlock check could effectively be accomplished just with that, but
:as mentioned, that's not true on unix, so it needs to be in the waiting
:loop for win32 with a try/except wrapper, and the rename should not
:be repeated if it succeeds, as it normally will on unix.
:
:It'd be nice if the builtin open had an option for exclusive access within
:a specified timeout for trying ;-)
Actually, the win32file.CreateFile does have an option for exclusive
access. I don't believe there is a specified timeout, but it does have
the exclusive access part.
Thanks again for all these tips and info. It really helps.
David Bolen
> :Note that the atomicity of renaming is critical to making this work. E.g.,
>
> Would someone please explain the concept of "atomic" or "atomicity" with
> respect to file operations. Do you mean, that you are (below) showing in
> detail, step-by-step what would happen at each point in the process that
> you suggest?
Atomicity or ensuring that an operation is "atomic" refers to the fact
that the operation in question can be considered a single
uninterruptible unit of activity, at some level of granularity. The
jargon file says it comes from the Greek "atomos" or indivisible.
For a CPU, atomic often means a single instruction since anything more
than a single instruction can be interrupted. But you can do stuff
like disable all processor interrupts, and then execute a series of
instructions as a single atomic activity. Higher level operations can
be atomic as well, at least as when compared to other operations at
the same general level and affecting the same resource.
Particularly when synchronization is involved, the operations being
used as part of the synchronization must guarantee that they cannot be
broken into separate steps or else they may fail to provide proper
synchronization.
This can not always be deduced simply by the top level entry points
into operations (e.g., the functions called, or the instructions
executed), since the operations being called may be formed by lower
level operations that themselves provide separate steps that might be
interrupted.
For example, take a really simple manual synchronization primitive of a
binary value used as a flag to control access to a shared resource.
If the value is 0, then nobody is using the resource, while if 1, then
the resource is in use. You might write code like:
if flag == 0:
# Mark that we're working with the data
flag = 1
# ...
# Use protected information
# ...
# Mark that we're done
flag = 0
The problem, of course, is that something could change in between your
querying the value to see if it's clear (0) and then resetting it to
1. That operation is therefore not atomic. Even if a compiler were
to offer a function that could test and set a value in one call, if it
compiled to separate processor instructions to load the current
contents from memory, test it and then write it back, an interrupt
during that sequence of operations could change the memory location.
Since this concept (testing and setting a value) is so common, many
systems - even down at the chip level - implement specific operations
for an atomic test-and-set operation.
In this case, instead of a flag, a file might be getting used as a
sentinel, and the rename operation on the file treated as a high level
way to change the state of that sentinel. What is important is that
the rename operation not be implemented in such a way as to permit its
activity to be interrupted or interfered with by other filesystem
activity related to the files in question. Otherwise you couldn't be
sure that after you renamed the file that the result hadn't been
influenced by other filesystem activity (equivalent to changing the
flag value in the above example) and thus it would no longer be
suitable as a consistent sentinel status.
In many (most? I'm not sure) filesystems, rename is an atomic (through
other internal locks or procedures) operation to help ensure the
integrity of the filesystem, at least when renaming is an integral
part of the filesystem interface. There are other times when it
doesn't hold true though, many of which occur when using networked
filesystems. Another good odds atomic operation is making a directory
- that's normally my preference if I have to have a cross-platform
sentinel file (it should even work with NFS). The big problem with
any non-active-filehandle approach (like a file or directory) is that
cleanup is critical if you don't want to block other tasks. So if
there's any chance the process managing the lock might die, you need
some way to remove the lock if that happens.
> Actually, the win32file.CreateFile does have an option for exclusive
> access. I don't believe there is a specified timeout, but it does have
> the exclusive access part.
It doesn't have the timeout, thus as my suggested code in another post
does, you have to implement a retry mechanism yourself. This leaves a
small risk of starvation (depending on load and retry mechanism) but
in most cases it's not a major problem.
Sheila King
Thanks for a very clear and thorough explanation on the idea of atomic
operations in the file system. (I finally took the time today to read
through this thoroughly.) I have some additional questions below.
On 28 Aug 2001 03:13:11 -0400, David Bolen <db...@fitlinxx.com> wrote in
comp.lang.python in article <uheusl...@ctwd0143.fitlinxx.com>:
:Atomicity or ensuring that an operation is "atomic" refers to the fact
:that the operation in question can be considered a single
:uninterruptible unit of activity, at some level of granularity.
...<excellent explanations snipped>...
:In many (most? I'm not sure) filesystems, rename is an atomic (through
:other internal locks or procedures) operation to help ensure the
:integrity of the filesystem, at least when renaming is an integral
:part of the filesystem interface. There are other times when it
:doesn't hold true though, many of which occur when using networked
:filesystems. Another good odds atomic operation is making a directory
:- that's normally my preference if I have to have a cross-platform
:sentinel file (it should even work with NFS).
So, you're saying that the way you usually handle exclusive access to
objects, is by creating/removing directories?
: The big problem with
:any non-active-filehandle approach (like a file or directory) is that
:cleanup is critical if you don't want to block other tasks. So if
:there's any chance the process managing the lock might die, you need
:some way to remove the lock if that happens.
Yes, I discovered that while testing, today. Ignacio had suggested that
I needed a __del__ function for my classes, and I found out today, that
on Linux, if an object had the lock and was deleted but that process
still kept running, I could not recover the lock. I've written a __del__
function, now, that seems to work. It unlocks the file and does cleanup.
:> Actually, the win32file.CreateFile does have an option for exclusive
:> access. I don't believe there is a specified timeout, but it does have
:> the exclusive access part.
:
:It doesn't have the timeout, thus as my suggested code in another post
:does, you have to implement a retry mechanism yourself. This leaves a
:small risk of starvation (depending on load and retry mechanism) but
:in most cases it's not a major problem.
I will post my code in another thread, I think. (This one is sooo long
already.) I think I have something that works fairly well, now.
David Bolen
> So, you're saying that the way you usually handle exclusive access to
> objects, is by creating/removing directories?
If I need to have a reliable means (or at least about as reliable as
possible) for gross-level locking compatibly cross-platform and
especially if NFS is involved (see my other recent post), then yes,
creating/removing directories is something I've used successfully in
the past. If my target domain is more limited, then the range of
possibilities expand.
> Yes, I discovered that while testing, today. Ignacio had suggested that
> I needed a __del__ function for my classes, and I found out today, that
> on Linux, if an object had the lock and was deleted but that process
> still kept running, I could not recover the lock. I've written a __del__
> function, now, that seems to work. It unlocks the file and does cleanup.
You're still somewhat at risk if your process is forcibly killed but
without permitting cleanup. Or if for example, your lock is across
the network but the source system for the lock crashes.
One approach that can be used is that in addition to creating the
sentinel file, once you "have the lock" create a PID file that
contains the PID of the process that has the lock.
When attempting to get the lock, if it fails, read the PID file and
then see if there is a process with that id on the system. If so,
assume the lock is valid and wait. You might be wrong if the actual
process died but the id got reused, but it's the more conservative
choice. (If you know that all processes using the lock are a
particular executable you can add cross-checks for that).
If not, log nasty warnings in your log files or consoles, then go
ahead and remove the lock and then try to re-acquire. Depending on
the timing requirements for a blocked system in the presence of
crashes, you can have this cleanup performed by an independent task at
some regular interval.