What is the standard workaround for the fact that rename() isn't atomic
on Win32? Do we need to create our own locking around the
reading/writing of files that are normally updated in place using
rename()?
Second, when you unlink() a file on Win32, do applications continue
accessing the old file contents if they had the file open before the
unlink?
--
Bruce Momjian | http://candle.pha.pa.us
pg...@candle.pha.pa.us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073
---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to majo...@postgresql.org
I'm pretty sure it errors with 'file in use'. Pretty ugly, huh?
Ross
Yeah - the windows filesystem is pretty poor when it comes to multiuser
access. That's why even as administrator I cannot delete borked files and
people's profiles and stuff off our NT server - the files are always 'in
use'. Even if you kick all users off, reboot the machine, do whatever.
It's terrible.
Chris
Visual C++ comes with the source to Microsoft's C library:
rename() calls MoveFile() which will error if:
1. The target file exists
2. The source file is in use
MoveFileEx() (not available on 95/98) can overwrite the target
file if it exists. The Apache APR portability library uses
MoveFileEx() to rename files if under NT/XP/2K vs. a sequence of :
1. CreateFile() to test for target file existence
2. DeleteFile() to remove the target file
3. MoveFile() to rename the old file to new
under Windows 95/98. Of course, some other process could create
the target file between 2 and 3, so their rename() would just
error out in that situation. I haven't tested it, but I recall
reading somewhere that MoveFileEx() has the ability to rename an
opened file. I'm 99% sure MoveFile() will fail if the source
file is open.
>
> Second, when you unlink() a file on Win32, do applications continue
> accessing the old file contents if they had the file open before the
> unlink?
>
unlink() just calls DeleteFile() which will error if:
1. The target file is in use
CreateFile() has the option:
FILE_FLAG_DELETE_ON_CLOSE
which might be able to be used to simulate traditional unlink()
behavior.
Hope that helps,
Mike Mascari
mas...@mascari.com
---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?
Yep. That's why often it requires rebooting to uninstall
software. How can the installer remove itself? Under Windows
95/98/ME, you have to manually add entries to WININIT.INI. With
Windows NT/XP/2K, MoveFileEx() with a NULL target and the
MOVEFILE_DELAY_UNTIL_REBOOT flag will add the appropriate
entries into the system registry so that the next time the
machine reboots it will remove the files specified. Its a real
pain and a real hack of an OS.
Mike Mascari
mas...@mascari.com
---------------------------(end of broadcast)---------------------------
OK, I downloaded APR and see in apr_file_rename():
if (MoveFileEx(frompath, topath, MOVEFILE_REPLACE_EXISTING |
MOVEFILE_COPY_ALLOWED))
Looking at the entire APR function, they have lots of tests so it works
on Win9X and wide characters. I think we will just use the APR as a
guide in implementing the things we need. I think MoveFileEx() is the
proper way to go; any other solution requires loop tests for rename.
I see the MoveFileEx manual page at:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/base/movefile.asp
> > Second, when you unlink() a file on Win32, do applications continue
> > accessing the old file contents if they had the file open before the
> > unlink?
> >
>
> unlink() just calls DeleteFile() which will error if:
>
> 1. The target file is in use
>
> CreateFile() has the option:
>
> FILE_FLAG_DELETE_ON_CLOSE
>
> which might be able to be used to simulate traditional unlink()
> behavior.
No, that flag isn't going to help us. I wonder what MoveFileEx does if
the target file exists _and_ is open by another user? I don't see any
loop in that Win32 rename() routine, and I looked at the Unix version of
apr_file_rename and its just a straight rename() call.
--
Bruce Momjian | http://candle.pha.pa.us
pg...@candle.pha.pa.us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073
---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majo...@postgresql.org so that your
message can get through to the mailing list cleanly
This says that if the target is in use, it is overwritten:
http://support.microsoft.com/default.aspx?scid=KB;EN-US;q140570&
While I think that is good news, does it open the problem of other
readers reading partial updates to the file and therefore seeing
garbage. Not sure how to handle that, nor am I even sure how I would
test it.
--
Bruce Momjian | http://candle.pha.pa.us
pg...@candle.pha.pa.us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073
---------------------------(end of broadcast)---------------------------
I read the article and did not come away with that conclusion.
The article describes using the MOVEFILE_DELAY_UNTIL_REBOOT
flag, which was created for the express purpose of allowing a
SETUP.EXE to remove itself, or rather tell Windows to remove it
on the next reboot. Also, if you want the Win32 port to run in
95/98/ME, you can't rely on MoveFileEx(), you have to use
MoveFile().
I will do some testing with concurrency and let you know. But
don't get your hopes up. This is one of the many advantages that
TABLESPACEs have when more than one relation is stored in a
single DATAFILE. There was Oracle for MS-DOS, after all..
Mike Mascari
mas...@mascari.com
I was focusing on handling of pg_pwd and other config file that are
written by various backend while other backends are reading them. The
actual data files should be OK because we have an exclusive lock when we
are adding/removing them.
--
Bruce Momjian | http://candle.pha.pa.us
pg...@candle.pha.pa.us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073
---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?
OK. So you want to test:
1. Process 1 opens "foo"
2. Process 2 opens "foo"
3. Process 1 renames "foo" to "bar"
4. Process 2 can safely read from its open file handle
Is that what you want tested? I have a small Win32 app ready to
test. Just let me know the scenarios...
Mike Mascari
mas...@mascari.com
---------------------------(end of broadcast)---------------------------
Actually, looking at the pg_pwd code, you want to determine a
way for:
1. Process 1 opens "foo"
2. Process 2 opens "foo"
3. Process 1 creates "bar"
4. Process 1 renames "bar" to "foo"
5. Process 2 can continue to read data from the open file handle
and get the original "foo" data.
Is that correct?
Mike Mascari
mas...@mascari.com
---------------------------(end of broadcast)---------------------------
Yep, that's it.
--
Bruce Momjian | http://candle.pha.pa.us
pg...@candle.pha.pa.us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073
---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to majo...@postgresql.org)
So far, MoveFileEx("foo", "bar", MOVEFILE_REPLACE_EXISTING)
returns "Access Denied" when Process 1 attempts the rename. But
I'm continuing to investigate the possibilities...
Mike Mascari
mas...@mascari.com
---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster
> Bruce Momjian wrote:
> > Mike Mascari wrote:
> >
> >>Actually, looking at the pg_pwd code, you want to determine a
> >>way for:
> >>
> >>1. Process 1 opens "foo"
> >>2. Process 2 opens "foo"
> >>3. Process 1 creates "bar"
> >>4. Process 1 renames "bar" to "foo"
> >>5. Process 2 can continue to read data from the open file handle
> >>and get the original "foo" data.
> >
> >
> > Yep, that's it.
> >
>
> So far, MoveFileEx("foo", "bar", MOVEFILE_REPLACE_EXISTING)
> returns "Access Denied" when Process 1 attempts the rename. But
> I'm continuing to investigate the possibilities...
Does a sequence like
Process 1 opens "foo"
Process 2 opens "foo"
Process 1 creates "bar"
Process 1 renames "foo" to <something>
- where something is generated to not overlap an existing file
Process 1 renames "bar" to "foo"
Process 2 continues reading
let you do the replace and keep reading (at the penalty that
you've now got to have a way to know when to remove the
various <something>s)
Yes! Indeed that does work.
Mike Mascari
mas...@mascari.com
---------------------------(end of broadcast)---------------------------
> Stephan Szabo wrote:
> > On Fri, 20 Sep 2002, Mike Mascari wrote:
> >>So far, MoveFileEx("foo", "bar", MOVEFILE_REPLACE_EXISTING)
> >>returns "Access Denied" when Process 1 attempts the rename. But
> >>I'm continuing to investigate the possibilities...
> >
> >
> > Does a sequence like
> > Process 1 opens "foo"
> > Process 2 opens "foo"
> > Process 1 creates "bar"
> > Process 1 renames "foo" to <something>
> > - where something is generated to not overlap an existing file
> > Process 1 renames "bar" to "foo"
> > Process 2 continues reading
> > let you do the replace and keep reading (at the penalty that
> > you've now got to have a way to know when to remove the
> > various <something>s)
>
> Yes! Indeed that does work.
Thinking back, I think that may still fail on Win95 (using MoveFile).
Once in the past I had to work on (un)installers for Win* and I
vaguely remember Win95 being more strict than Win98 but that may just
have been with moving the executable you're currently running.
Well, here's the test:
foo.txt contains "This is FOO!"
bar.txt contains "This is BAR!"
Process 1 opens foo.txt
Process 2 opens foo.txt
Process 1 sleeps 7.5 seconds
Process 2 sleeps 15 seconds
Process 1 uses MoveFile() to rename "foo.txt" to "foo2.txt"
Process 1 uses MoveFile() to rename "bar.txt" to "foo.txt"
Process 1 uses DeleteFile() to remove "foo2.txt"
Process 2 awakens and displays "This is FOO!"
On the filesystem, we then have:
foo.txt containing "This is BAR!"
The good news is that this works fine under NT 4 using just
MoveFile(). The bad news is that it requires the files be opened
using CreateFile() with the FILE_SHARE_DELETE flag set. The C
library which ships with Visual C++ 6 ultimately calls
CreateFile() via fopen() but with no opportunity through the
standard C library routines to use the FILE_SHARE_DELETE flag.
And the FILE_SHARE_DELETE flag cannot be used under Windows
95/98 (Bad Parameter). Which means, on those platforms, there
still doesn't appear to be a solution. Under NT/XP/2K,
AllocateFile() will have to modified to call CreateFile()
instead of fopen(). I'm not sure about ME, but I suspect it
behaves similarly to 95/98.
Mike Mascari
mas...@mascari.com
I don't understand FILE_SHARE_DELETE. I read the description at:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/base/createfile.asp
but I don't understand it:
FILE_SHARE_DELETE - Windows NT/2000/XP: Subsequent open operations on
the object will succeed only if delete access is requested.
---------------------------------------------------------------------------
--
Bruce Momjian | http://candle.pha.pa.us
pg...@candle.pha.pa.us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073
---------------------------(end of broadcast)---------------------------
That is the hard part. Mike's description omitted one crucial step:
6. The old "foo" goes away when the last open file handle for it is
closed.
I doubt there is any practical way for Postgres to cause that to happen
if the OS itself does not have any support for it.
regards, tom lane
I think that's a rather poor description. I think it just means
that if the file is opened once via CreateFile() with
FILE_SHARE_DELETE, then any subsequent CreateFile() calls will
fail unless they too have FILE_SHARE_DELETE. In other words, if
one of us can delete this file while its open, any of us can.
Mike Mascari
mas...@mascari.com
---------------------------(end of broadcast)---------------------------
> Bruce Momjian wrote:
> > I don't think we are not going to be supporting Win9X so there isn't an
> > issue there. We will be supporting Win2000/NT/XP.
> >
> > I don't understand FILE_SHARE_DELETE. I read the description at:
> >
> > http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/base/createfile.asp
> >
> > but I don't understand it:
> >
> > FILE_SHARE_DELETE - Windows NT/2000/XP: Subsequent open operations on
> > the object will succeed only if delete access is requested.
>
> I think that's a rather poor description. I think it just means
> that if the file is opened once via CreateFile() with
> FILE_SHARE_DELETE, then any subsequent CreateFile() calls will
> fail unless they too have FILE_SHARE_DELETE. In other words, if
> one of us can delete this file while its open, any of us can.
The question is, what happens if two people have the file open
and one goes and tries to delete it? Can the other still read
from it?
I don't understand what that gets us.
--
Bruce Momjian | http://candle.pha.pa.us
pg...@candle.pha.pa.us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073
---------------------------(end of broadcast)---------------------------
> instead of fopen(). I'm not sure about ME, but I suspect it
> behaves similarly to 95/98.
I just checked with Katie and the good news (tm) is that the Win32 port
we did here at PeerDirect doesn't support 95/98 and ME anyway. It does
support NT4, 2000 and XP. So don't bother.
Jan
--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanW...@Yahoo.com #
Yes. I just tested it and it worked. I'll test Bruce's scenario
as well:
foo contains: "FOO"
bar contains: "BAR"
1. Process 1 opens "foo"
2. Process 2 opens "foo"
3. Process 1 calls MoveFile("foo", "foo2");
4. Process 3 opens "foo" <- Successful?
5. Process 1 calls MoveFile("bar", "foo");
6. Process 4 opens "foo" <- Successful?
7. Process 1 calls DeleteFile("foo2");
8. Process 1, 2, 3, 4 all read from their respective handles.
I think the thing to worry about is a race condition between the
two MoveFile() attempts. A very ugly hack would be to loop in a
CreateFile() in an attempt to open "foo", giving up if the error
is not a NOT EXISTS error code.
Mike Mascari
mas...@mascari.com
---------------------------(end of broadcast)---------------------------
Process 1: "FOO"
Process 2: "FOO"
Process 3: Error - File does not exist
Process 4: "BAR"
Its interesting in that it allows for Unix-style rename() and
unlink() behavior, but with a race condition. Without Stephan's
two MoveFile() trick and the FILE_SHARE_DELETE flag, however,
the result would be Access Denied. Are the places in the backend
that use rename() and unlink() renaming and unlinking files that
are only opened for a brief moment by other backends?
Mike Mascari
mas...@mascari.com
---------------------------(end of broadcast)---------------------------
Yes, those files are only opened for a brief moment. They are not held
open.
--
Bruce Momjian | http://candle.pha.pa.us
pg...@candle.pha.pa.us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073
---------------------------(end of broadcast)---------------------------
I think we are better off looping over
MoveFileEx(MOVEFILE_REPLACE_EXISTING) until the file isn't opened by
anyone. That localizes the changes to rename only and not out to all
the opens.
The open failure loops when the file isn't there seem much worse.
I am a little concerned about starving the rename when there is a lot of
activity but I don't see a better solution.
--
Bruce Momjian | http://candle.pha.pa.us
pg...@candle.pha.pa.us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073
---------------------------(end of broadcast)---------------------------