hg repositories and Windows DFS Replication

[Robert Altman]

Has anyone ever used mercurial on Windows servers with DFS replicating the repositories to another server.  (This would essentially server to "mirror" the repositories to another computer.)

As long as the changes are only made on one server (the web server), is there any risk to the data itself?  (I wouldn't think so, but I'm curious if anyone has tried it.)

Thanks.

[Matt Mackall]

First, Mercurial repos are just files; there's no intangible spirit that
needs to be copied. The only magic you need to be aware of is that
copies made during a write may not be completely valid. This generally
isn't an issue for backup scenarios.

I'm not sure how DFS compares to industrial-strength distributed
filesystems like GFS and Lustre, but if it provides similar coherency
guarantees, it should be more or less invisible to Mercurial, even with
multiple writers. But given the reliability we've observed over CIFS,
I'd try to keep things simple.

--
Mathematics is the supreme nostalgia of our time.

_______________________________________________
Mercurial mailing list
Merc...@selenic.com
http://selenic.com/mailman/listinfo/mercurial

[Matt Mackall]

[replying back to list]

On Tue, 2011-05-31 at 21:56 -0500, Robert Altman wrote:
> The only problems I have seen with DFS are when copying large files (>100's
> MB). If you try moving or renaming a large files while it is being
> replicated, the operation may fail with file-in-use error.

Oh. That's quite unfortunate. I guess that answers the "is DFS in the
same class with GFS and Lustre" with a resounding no.

If I had to guess, I would say that the above problem is encountered
with a probability proportional to size, and you will eventually
encounter it on a small file given enough time/use.

> Most version control files are small, so this seems unlikely to be an issue
> but I like to be certain when it comes to my own repositories.
>
> How would hg handle this?

If Mercurial encounters an filesystem error, it aborts. Filesystem
operations are supposed to be maximally reliable, not a best-effort
try-again sort of affair; if the operating system gives up, Mercurial
assumes the situation is hopeless.

You may be in luck here though: the one exception Mercurial makes is for
deletion of files that are being renamed on Windows, because the
semantics here on Windows are a bit magical.

[Adrian Buehlmann]

> On Tue, 2011-05-31 at 21:56 -0500, Robert Altman wrote:
>> The only problems I have seen with DFS are when copying large files (>100's
>> MB). If you try moving or renaming a large files while it is being
>> replicated, the operation may fail with file-in-use error.

Then this is completely unsuitable for use with Mercurial on Windows and
might lead to repository corruption - not just for large files. [1]

Mercurial was designed for POSIX-like file systems and Windows isn't. We
(or rather Matt and contributors) mimicked POSIX-like file access on
Windows by using a wrapping layer. At the core of this is a file pattern
that renames a file to a temporary name before re-writing the original
file. [2]

So, locking a filename while replicating it is a big NO for Mercurial
repositories. It breaks the assumptions Mercurial makes about a file system.

If you intend to ponder deeply about file system access on Windows and
Mercurial see:
http://mercurial.selenic.com/wiki/UnlinkingFilesOnWindows
http://mercurial.selenic.com/wiki/HardlinkedClones

[1]
Luckily, thanks to the decent architecture of Mercurial, we have a 'hg
verify' command that finishes in reasonable time, so, at least, 'hg
verify' can be used to detect such corruption -- other DVCS' either
don't yet have a verify command - cough, hi Veracity - or are slow like
hell so almost nobody ever checks the integrity of their repos - cough,
hi Bazaar.

[2]
People may argue that re-writing isn't used because they heard someone
saying that Mercurial does "append only" inside the store (.hg/store).
But Mercurial needs to break up hardlinks inside the store, and there,
that pattern is absolutely needed on Windows. Besides that, operations
that switch an inlined *.i file to a *.i / *.d file pair in the store
are not 'append only' at the files layer either.

[Matt Brown - nyc]

Guys, before misinfo spreads further, DFS-R does deal with large files like any other file.

But... you have to keep the method DFS-R uses in mind considering:

1) a file is written to a folder monitored for replication on SERVER A

2) SERVER A then produces a delta check sum of the already generated check sum of the file (which is a version of the originally generated checksum of the file when DFS-R was configured to monitor this folder).

3) SERVER A communicates to all receiving servers (we'll call this entity simply SERVER B) to figure out if SERVER B has any data "blocks" in its existing cache that will match this file delta... If so, then it replicates a small amount, if not, it replicates a large amount, up to the total size of the delta.

4) SERVER B receives the delta data

5) SERVER B "stages" the file==writes the ENTIRE contents of the file in to a staging area [this is where the bottleneck exists in the instance of pre-existing large files that have many little changes taking place]

6) If during the staging on SERVER B, SERVER A sends another change to the same file, SERVER B cancels the current staging operation on CHANGE 1 and starts processing CHANGE 2... this "loop" can introduce huge problems... because... you got it... ORIGINAL FILE will always be ORIGINAL FILE until CHANGE N*94 is allowed to finish (see step 7)

7) When SERVER B is finally allowed to finish writing the file completely, it simply moves the file (aka changes the block location(s) entry in the file system table for the file) to it's final production location (and "staging" is finished.

http://mbrownnyc.wordpress.com/2011/08/24/method-to-really-monitor-dfs-replication/  

Due to these necessary steps performed, generally, it is NOT recommended to use DFS-R to replicate extremely large files, but as you already gathered, that truly relies on what your receiving servers' write capabilities.