I want my script to generate a ~1KB status file several times a second.
The script may be terminated at any time but the status file must not
be corrupted.
When the script is started next time the status file will be read to
check what needs to be done.

My initial solution was a thread that writes status to a tmp file
first and then renames:

open(tmp_file, 'w').write(status)
os.rename(tmp_file, status_file)

This works well on Linux but Windows raises an error when status_file
already exists.
http://docs.python.org/library/os.html#os.rename

I guess I could delete the status file:

open(tmp_file, 'w').write(status)
if os.path.exists(status_file):
    os.remove(status_file)
os.rename(tmp_file, status_file)

and then on startup read from tmp_file if status_file does not exist.
But this seems awkward.

Is there a better way? Or do I need to use a database?

with open(tmp_file, "w") as f:
    f.write(status)
    # flush buffer and write data/metadata to disk
    f.flush()
    os.fsync(f.fileno())

# now rename the file
os.rename(tmp_file, status_file)

# finally flush metadata of directory to disk
dirfd = os.open(os.path.dirname(status_file), os.O_RDONLY)
try:
    os.fsync(dirfd)
finally:
    os.close(dirfd)

if os.name == "posix":
    rename = os.rename
else:
    def rename(a, b):
        try:
            os.rename(a, b)
        except OSError, e:
            if e.errno != 183:
                raise
            os.unlink(b)
            os.rename(a, b)

Michael

But even if the rename operation is atomic, there is still a race
condition. Your program can be terminated after the original status file
has been deleted, and before the temp file was renamed. In this case,
you will be missing the status file (although your program already did
something just it could not write out the new status).

Here is an algorithm that can always write and read a status (but it
might not be the latest one). You can keep the last two status files.

Writer:
*    create temp file, write new status info
* create lock file if needed
* flock it
* try:
*    delete older status file
*   rename temp file to new status file
* finally: unlock the lock file

Reader:

* flock the lock file
* try:
*    select the newer status file
*   read status info
* finally: unlock the lock file

It is guaranteed that you will always have a status to read, and in most
cases this will be the last one (because the writer only locks for a
short time). However, it is still questionable, because your writer may
be waiting for the reader to unlock, so the new status info may not be
written immediatelly.

It would really help if you could tell us what are you trying to do that
needs status.

Best,

You don't need locks or any other fancy stuff. You just need to make
sure that you flush the data and metadata correctly to the disk and
force a re-write of the directory inode, too. It's a standard pattern on
POSIX platforms and well documented in e.g. the maildir RFC.

You can use the same pattern on Windows but it doesn't work as good and
doesn't guaranteed file integrity for two reasons:

1) Windows's rename isn't atomic if the right side exists.

2) Windows locks file when a program opens a file. Other programs can't
rename or overwrite the file. (You can get around the issue with some
extra work, though.)

import sqlite3

con = sqlite3.connect('status.db')

...
with con:
    cur = con.cursor()
    cur.execute('UPDATE ...', ...)

and similar code to restore the status or create required tables on
startup.

http://msdn.microsoft.com/en-us/library/aa365512%28VS.85%29.aspx

     Renaming files is the wrong way to synchronize a
crawler.  Use a database that has ACID properties, such as
SQLite.  Far fewer I/O operations are required for small updates.
It's not the 1980s any more.

     I use a MySQL database to synchronize multiple processes
which crawl web sites.  The tables of past activity are InnoDB
tables, which support transactions.  The table of what's going
on right now is a MEMORY table.  If the database crashes, the
past activity is recovered cleanly, the MEMORY table comes back
empty, and all the crawler processes lose their database
connections, abort, and are restarted.  This allows multiple
servers to coordinate through one database.

There are existing solutions for managing job queues, so you can
choose what you like.  If you're unfamiliar, maybe start by looking at
celery.

Michael

See http://linux.die.net/man/2/rename, section BUGS.

"The ReplaceFile function combines several steps within a single
function. An application can call ReplaceFile instead of calling
separate functions to save the data to a new file, rename the original
file using a temporary name, rename the new file to have the same name
as the original file, and delete the original file."

About the best you can get in Windows, I think, is MoveFileTransacted,
but you need to be running Vista or later:

http://msdn.microsoft.com/en-us/library/windows/desktop/aa365241(v=vs...

POSIX has no operation that actually deletes a file. It just has an
unlink() syscall that removes an associated name from an inode. As soon
as an inode has no names and is not references by a file descriptor, the
file content and inode is removed by the operating system. rename() is
more like a link() followed by an unlink() wrapped in a system wide
global lock.

With POSIX semantics the reading process will either see the full
content before the rename op or the full content after the rename op.
The writing process can replace the name (rename op) while the reading
process reads the status file because its file descriptor still points
to the old status file.

On Windows, things are very different.

Hope this helps,

                                        Ross Ridge

"Good" case:

process #1:  unlink(old status file)
process #1: rename(new status file)
process#2: open(new status file)
process#2: read(new status file)

"Bad" case:

process #1:  unlink(old status file)
process#2: open(???) -- there is no file on disk here, this system call
returns with an error!
process #1: rename(new status file)

 >>> f = open("test.txt","wb+")
 >>> f.close()
 >>> f2 = open("test2.txt","wb+")
 >>> f2.close()
 >>> import os
 >>> os.rename("test2.txt","test.txt")
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
WindowsError: [Error 183] File already exists
 >>>

I have also tried this on FreeBSD and it worked.

Now, let's go back to the original question: