How to let Mercurial treat local repositories as ordinary files

[Markus]

Hello HGlers,

what would I have to do to advise "hg" to treat every file in the
directory tree, it is responsible for, as an ordinary file,
irrespective if some of these files happen to have something to do
with more local hg tree structures?

The reason, why I'm asking, is, that I'm about to use Mercurial as an
incremental backup solution and I can not tolerate, that files are
omitted from the backup only because some of my users also use
Mercurial locally somewhere.


Best regards,

Markus


_______________________________________________
Mercurial mailing list
Merc...@selenic.com
http://selenic.com/mailman/listinfo/mercurial

[Kevin Bullock]

On Jul 11, 2012, at 2:23 PM, Markus wrote:

Hello HGlers,

what would I have to do to advise "hg" to treat every file in the
directory tree, it is responsible for, as an ordinary file,
irrespective if some of these files happen to have something to do
with more local hg tree structures?

The reason, why I'm asking, is, that I'm about to use Mercurial as an
incremental backup solution and I can not tolerate, that files are
omitted from the backup only because some of my users also use
Mercurial locally somewhere.

My primary recommendation is: Don't do that. Mercurial is designed for versioning source code, and because of several design decisions related to that is not well-suited to being cajoled into a backup solution.

* It doesn't handle large non-compactly-diffable binary file formats well.

* It doesn't reconstruct file permissions, except for the execute bit.

* It deliberately doesn't descend into hg repos contained within the managed tree, unless you add them as subrepos.

* …and if you do add them as subrepos, you get a whole bunch of other constraints along for the ride.

You could maybe rig up something using subrepos, but it would be duct tape and bailing wire. There are innumerable better incremental backup solutions available.

pacem in terris / мир / शान्ति / ‎‫سَلاَم‬ / 平和

Kevin R. Bullock

[Markus]

Hello Kevin + *,

thanks for your time to think about my "problems" ;-)

> On Jul 11, 2012, at 2:23 PM, Markus wrote:
>
> > Hello HGlers,
> >
> > what would I have to do to advise "hg" to treat every file in the
> > directory tree, it is responsible for, as an ordinary file,
> > irrespective if some of these files happen to have something to do
> > with more local hg tree structures?
> >
> > The reason, why I'm asking, is, that I'm about to use Mercurial as an
> > incremental backup solution and I can not tolerate, that files are
> > omitted from the backup only because some of my users also use
> > Mercurial locally somewhere.
>
> My primary recommendation is: Don't do that. Mercurial is designed for versioning source code, and because of several design decisions related to that is not well-suited to being cajoled into a backup solution.
>

I'm not [yet?] convinced by the next two arguments.
And the third and forth are exactly my question.

Thus, I'm not yet such far to take my question back.

> * It doesn't handle large non-compactly-diffable binary file formats well.

... and the better alternative wrtt?

> * It doesn't reconstruct file permissions, except for the execute bit.

This is relatively simple to add.

> * It deliberately doesn't descend into hg repos contained within the managed tree, unless you add them as subrepos.

> * …and if you do add them as subrepos, you get a whole bunch of other constraints along for the ride.
>
What I'm looking for is something equivalent to a switch, which tells
"hg" to treat every directory equally -- be there a subdirectory named
".hg" in it or not. {hg is perfectly working the way I want, if all
".hg" subdirectories are renamed to something else during the time hg
sees them; but I'm reluctant to add a stack of wrapping code to get
something, which should more easily to be got by just leaving
something off: the special handling of directories, which contain a
subdirectory named ".hg".}

> You could maybe rig up something using subrepos, but it would be
> duct tape and bailing wire.

I hope, my last statement is to be understood as my desire to just
avoid such duct tape {nice metaphor}.

> There are innumerable better incremental backup solutions available.
>

Ok -- and these better incremental backup solutions were? {You may
answer by pm, if you fear, this runs off topic of this list.}

[Michael Diamond]

On Wed, Jul 11, 2012 at 5:57 PM, Markus <mp.l...@free.fr> wrote:

Hello Kevin + *,

thanks for your time to think about my "problems" ;-)

 > On Jul 11, 2012, at 2:23 PM, Markus wrote:
 >
 > > Hello HGlers,
 > >
 > > what would I have to do to advise "hg" to treat every file in the
 > > directory tree, it is responsible for, as an ordinary file,
 > > irrespective if some of these files happen to have something to do
 > > with more local hg tree structures?
 > >
 > > The reason, why I'm asking, is, that I'm about to use Mercurial as an
 > > incremental backup solution and I can not tolerate, that files are
 > > omitted from the backup only because some of my users also use
 > > Mercurial locally somewhere.
 >
 > My primary recommendation is: Don't do that. Mercurial is designed for versioning source code, and because of several design decisions related to that is not well-suited to being cajoled into a backup solution.
 >

I'm not [yet?] convinced by the next two arguments.
And the third and forth are exactly my question.

Thus, I'm not yet such far to take my question back.

 > * It doesn't handle large non-compactly-diffable binary file formats well.

... and the better alternative wrtt?

Compact diffs of binary files are, essentially by definition, difficult and application specific.  Kevin's point is that Mercurial will do poorly as a backup tool for binary files.

 > * It doesn't reconstruct file permissions, except for the execute bit.

This is relatively simple to add.

Unlikely.  You don't grasp the scope of this issue.  Needless to say, it's not a feature likely to be added to Mercurial any time soon, and doing so would involve quite a lot of fairly severe changes, all of which would have to be backwards compatible.

 > * It deliberately doesn't descend into hg repos contained within the managed tree, unless you add them as subrepos.

 > * …and if you do add them as subrepos, you get a whole bunch of other constraints along for the ride.
 >
What I'm looking for is something equivalent to a switch, which tells
"hg" to treat every directory equally -- be there a subdirectory named
".hg" in it or not.  {hg is perfectly working the way I want, if all
".hg" subdirectories are renamed to something else during the time hg
sees them; but I'm reluctant to add a stack of wrapping code to get
something, which should more easily to be got by just leaving
something off: the special handling of directories, which contain a
subdirectory named ".hg".} 

For anyone using Mercurial for its intended purpose (source control) this is undesirable.  Having a super-repository track both working directory contents and repository contents is a recipe for trouble down the line (and, the .hg repository consists of binary files, which, as mentioned, aren't ideal to track anyways).

 > You could maybe rig up something using subrepos, but it would be
 > duct tape and bailing wire.

I hope, my last statement is to be understood as my desire to just
avoid such duct tape {nice metaphor}.

And Kevin is telling you that no, your desire is both impossible and a poor practice.  Making the necessary changes to core Mercurial is both simply not going to happen, and trying to do it yourself will be dramatically more painful than you imagine.

 > There are innumerable better incremental backup solutions available.
 >

Ok -- and these better incremental backup solutions were?  {You may
answer by pm, if you fear, this runs off topic of this list.}

Best bet is to use Google, asking us is like walking into a bakery and asking the baker to recommend a cut of meat.

[Dave S]

On Wed, Jul 11, 2012 at 5:57 PM, Markus <mp.lists at free.fr> wrote:


> > There are innumerable better incremental backup solutions available.
> >
> Ok -- and these better incremental backup solutions were? {You may
> answer by pm, if you fear, this runs off topic of this list.}
>

I can recommend Beyond Compare from
<http://www.scootersoftware.com/moreinfo.php>

It's not free, although you get full features for a 30-day trial.

BC3 is a source compare tool that has grown up very well. It works
very well on Windows for incremental backups; it runs on Linux, but I
haven't tried any incremental backup techniques with it there. It's
normally set up to be run by a user for his or her own files, but with
the proper account I imagine you could take care of multiple users.

Of course, there are true backup solutions that support incremental
backup. I believe Amanda is a Linux tool that can do that. And at
one time, a lot of people were using scripts based on rsynch.

HTH.

/dps

[Robert Figura]

"Markus" <mp.l...@free.fr> schrieb:

> > There are innumerable better incremental backup solutions available.
> >
> Ok -- and these better incremental backup solutions were? {You may
> answer by pm, if you fear, this runs off topic of this list.}

Nowadays rsync is the classic. It has a small network footprint and it
can hardlink unchanged files to reduce disk usage.

Kind Regards
- Robert Figura

--
http://teslawm.org/

[Jeff Baird]

While it would be nice to use the same setup for version control as
backups, they are really not the same. A simple backup system like
rdiff-backup would probably be better.

Here is a little script that makes clever use of hardlinks (and is even
better if the target is on a compressing file system...)

#!/bin/sh
# backup_incremental
# by jc...@cs.cmu.edu with inspiration from josu...@cs.cmu.edu and
jslt...@cs.cmu.edu
# Released under GPL: http://www.fsf.org/copyleft/gpl.html
#
# script for incremental backup
# $1 = location of directory structure to backup
# $2 = location of backup directory tree
#
export to=$2
export from=$1

export start_time=`date '+%Y_%m_%d_%H_%M_%S'`
mkdir -p $to/$from/current
cp -al $to/$from/current $to/$from/$start_time
rsync -azHS --delete -e ssh $from $to/$from/current


--
Jeff Baird <Jeff_...@cs.cmu.edu>

[Markus]

Hi Micheal and all,

Wed, 11 Jul 2012 19:08:51 -0400, Michael wrote:

> On Wed, Jul 11, 2012 at 5:57 PM, Markus <mp.l...@free.fr> wrote:
>
> > Hello Kevin + *,
> >
> > thanks for your time to think about my "problems" ;-)
> >
> >
> > > On Jul 11, 2012, at 2:23 PM, Markus wrote:
> > >
> > > > Hello HGlers,
> > > >

...

> >
> > ... and the better alternative wrtt?
> >
>
> Compact diffs of binary files are, essentially by definition, difficult and
> application specific. Kevin's point is that Mercurial will do poorly as a
> backup tool for binary files.
>

If somebody can point us to a better binary differ, may be it could
be implemented in a future release of Mercurial.
Btw. binary diffing is not such bad, if the diff is empty.

> >
> >
> > > * It doesn't reconstruct file permissions, except for the execute bit.
> >
> > This is relatively simple to add.
> >
>
> Unlikely. You don't grasp the scope of this issue. Needless to say, it's
> not a feature likely to be added to Mercurial any time soon, and doing so
> would involve quite a lot of fairly severe changes, all of which would have
> to be backwards compatible.
>

I'm not asking to put file metadata management into Mercurial {I would
have posted it to the developer list then}.
When I said "relatively simple", I had in mind, doing it in a similar
way, metastore does it, but with something like the better algorithm
of e.g. "tar".

> >
> >
> > > * It deliberately doesn't descend into hg repos contained within the
> > managed tree, unless you add them as subrepos.
> > > * …and if you do add them as subrepos, you get a whole bunch of other
> > constraints along for the ride.
> > >
> > What I'm looking for is something equivalent to a switch, which tells
> > "hg" to treat every directory equally

...

>
> For anyone using Mercurial for its intended purpose (source control) this
> is undesirable.

I never, said, "my" switch should be the default for anybody else.

> Having a super-repository track both working directory
> contents and repository contents is a recipe for trouble down the line
> (and, the .hg repository consists of binary files, which, as mentioned,
> aren't ideal to track anyways).
>

hg already does it perfectly [as far as tested yet] as long as ".hg"
sub directories are named differently.


...

>
> And Kevin is telling you that no, your desire is both impossible

I don't think, he said this, and I don't think, it is.

> and a poor
> practice. Making the necessary changes to core Mercurial is both simply
> not going to happen, and trying to do it yourself will be dramatically more
> painful than you imagine.
>

If not already there in some form {that's, what I asked for,
originally}, it can not be such hard, just to switch off some special
behaviour of hg, in a situation, where this is requested.

> >
> >
> > > There are innumerable better incremental backup solutions available.
> > >
> > Ok -- and these better incremental backup solutions were? {You may
> > answer by pm, if you fear, this runs off topic of this list.}
> >
>
> Best bet is to use Google, asking us is like walking into a bakery and
> asking the baker to recommend a cut of meat.
>

Kevin already was such constructive to explain, what he meant.
I think, Google wouldn't have been able to do this.

[Markus]

Hello all,

thanks to all, who threw in their knowledge on the topic.

If I try to sum the discussion up so far, I finally see it as a matter
of taste:
- a version management system
- offers me
- support for a lot of distributed data handling scenaria
{eg. my alter ego working on its laptop}
- tell me, what changed when, down to the line level {log, annotate}
- built-in expertise in redundancy reduction
- efficient access to any {as far as tracked} status of a file in the past
- doesn't offer me
- file metadata management
- truth wrt subdirectory content
- a conventional [bare bones] backup system {eg. rsync {thanks to Jeff's tutorial}}
- offers me
- file metadata management
- truth wrt subdirectory content
- does not offer me
- built-in expertise in redundancy reduction
{I have to fiddle out myself, how to handle hardlinks in a
clever way}
- efficient access to the change history
{I have to write the algorithms to find out, what changed
when myself}
.

Somehow, I have the feeling to work on the Assembler level, when using
bare bones backup support systems {eg. rsync}: I have more control and
I'm forced to do it. With a [distributed] version managment system,
I'm on a higher level and get more expertise [eg. wrt distribution,
workflow, redundancy mgmt.] built-in, but also some magic and a
certain price to pay for hardware ressources.

Special purpose backup solutions in the best case can be the most
comforable [when I found the product, which exactly does, what I
want], but certainly the most restricting in possible (usage
scenaria)|workflows.


Up to now, it was an illuminative discussion on backup solutions, but
I still only see very few light wrt my original simple little
question.

[Kevin Bullock]

On 12 Jul 2012, at 12:47 PM, Markus wrote:

 - a conventional [bare bones] backup system  {eg. rsync  {thanks to Jeff's tutorial}}
     - offers me
         - file metadata management
         - truth wrt subdirectory content
     - does not offer me
         - built-in expertise in redundancy reduction
           {I have to fiddle out myself, how to handle hardlinks in a
            clever way}

Where on earth did you get that impression? _Any_ decent backup tool (even GNU cp) can use hardlinks to reduce the size of the backup pool.

         - efficient access to the change history
           {I have to write the algorithms to find out, what changed
            when myself}

Again, any decent backup system can tell you at the file level exactly what changed. If you want source-level change tracking, well, that's why you're using hg for source control, right? And a backup tool will back up Mercurial repos just as well as any other file.

Somehow, I have the feeling to work on the Assembler level, when using
bare bones backup support systems {eg. rsync}: I have more control and
I'm forced to do it.  With a [distributed] version managment system,
I'm on a higher level and get more expertise [eg. wrt distribution,
workflow, redundancy mgmt.] built-in, but also some magic and a
certain price to pay for hardware ressources.

Special purpose backup solutions in the best case can be the most
comforable [when I found the product, which exactly does, what I
want], but certainly the most restricting in possible (usage
scenaria)|workflows.


Up to now, it was an illuminative discussion on backup solutions, but
I still only see very few light wrt my original simple little
question.  

I'm still not sure what your exact requirements are that an existing backup solution wouldn't fit.

[Lester Caine]

Markus wrote:
> Up to now, it was an illuminative discussion on backup solutions, but
> I still only see very few light wrt my original simple little
> question.

I think you partially miss the key point here. hg uses a set of hidden files and
folders to store information on all of the changes made to that data. It works
most efficiently with text files as it can simply store the 'patches' required
to create each version of the file. If it can't create a 'patch' then it has to
keep a complete copy of each binary file, which is where there has been some
moves to support the sort of thing you asked for. However what you are asking to
do does not make any sense given the way hg actually stores the history data.
Storing that data in a totally unrelated area of the disk could be practical,
you would still need something to tell hg where to look?

Now if the whole area you want to manage is contained one level down, you would
not have to worry about the overhead .hg directory. That would work, but is
still not directly addressing your 'backup' requirements. (And BeyondCompare
does not become part of this either ... that simply allows one to identify files
which have changed, not back them up).

At the end of the day are you simply trying to back up all the changed files on
a disk to another location? Which is rsync ... Or are you wanting to record each
version of a file change and be able to select a particular version? In which
case the history has to be stored somewhere ...

--
Lester Caine - G8HFL
-----------------------------
Contact - http://lsces.co.uk/wiki/?page=contact
L.S.Caine Electronic Services - http://lsces.co.uk
EnquirySolve - http://enquirysolve.com/
Model Engineers Digital Workshop - http://medw.co.uk
Rainbow Digital Media - http://rainbowdigitalmedia.co.uk

[Markus]

Hi Kevin,

2012-07-12 18:13, Kevin wrote:

> On 12 Jul 2012, at 12:47 PM, Markus wrote:
>
> > - a conventional [bare bones] backup system {eg. rsync {thanks to Jeff's tutorial}}
> > - offers me
> > - file metadata management
> > - truth wrt subdirectory content
> > - does not offer me
> > - built-in expertise in redundancy reduction
> > {I have to fiddle out myself, how to handle hardlinks in a
> > clever way}
>
> Where on earth did you get that impression? _Any_ decent backup tool (even GNU cp) can use hardlinks to reduce the size of the backup pool.
>

At least such a simple thing as "cp" only does something with
hardlinks, when I tell it to. It's up to _me_ to implement the
"if a equals b ; then cp -l ... ; else cp ... ; fi "-logic.
With a vcs, I don't have to think about such implementation details to
minimise redundancy.

> > - efficient access to the change history
> > {I have to write the algorithms to find out, what changed
> > when myself}
>
> Again, any decent backup system can tell you at the file level exactly what changed. If you want source-level change tracking, well, that's why you're using hg for source control, right? And a backup tool will back up Mercurial repos just as well as any other file.

At least rsync [as an example of the bare bones systems] offers a way
to perform snapshot transport in a clever way, but [as far as I
understand it] requires the _user_ to
- implement a snapshot data organisation
- search [when investigating file changes eg.] the point in the
snapshot sequence, where a status change happened, and to make a
diff afterwards, if he is interested in the status change details.

I have too few experience with "any decent backup system" to generalize
too much further.


>
...

>
> I'm still not sure what your exact requirements are that an existing backup solution wouldn't fit.
>

I just asked, which possibilities there are to tell "hg" not to do
anything special with directories, that contain ".hg" directories
[and, sorry, I gave a bit of context for my question].

[Markus]

Hi Lester,

> Markus wrote:
> > Up to now, it was an illuminative discussion on backup solutions, but
> > I still only see very few light wrt my original simple little
> > question.
>

...

> Storing that data in a totally unrelated area of the disk could be practical,
> you would still need something to tell hg where to look?
>

Presently, I have the root ".hg" directory {the one for backup
purposes} symlinked to a different disk.
This is not elegant {is there a better way?}, but not the problem.

The problem is "hg"'s habit to treat directories below, which contain
a ".hg" directory[, such are assumed to be hg spaces {repository +
workspace} themselves,] in a particular way: [depending on
configuration] it ignores them or assumes, they are sub repositories.
And this habit spoils my intention to get a true snapshot of
everything below the root. Thus [on the backup level], I'd like to
"switch it off".


...

> At the end of the day are you simply trying to back up all the changed files on
> a disk to another location? Which is rsync ... Or are you wanting to record each
> version of a file change and be able to select a particular version? In which
> case the history has to be stored somewhere ...
>

Both: I want to have a fine-grained history [for restore comfort]
at another location [for robustness].

[Michael Diamond]

On Thu, Jul 12, 2012 at 4:56 PM, Markus <mp.l...@free.fr> wrote:

Hi Michael,

Michael Diamond writes:
 > On Thu, Jul 12, 2012 at 3:41 PM, Markus <mp.l...@free.fr> wrote:
 >

...

 >
 > Normally one would take these issues as a sign that they're trying to fit a
 > square peg in a round hole, and conclude that Mercurial is not designed to
 > recursively track instances of itself, and isn't the right solution for the
 > problem at hand.  You seem quite dead set on using Mercurial as a backup
 > tool, so trying to convince you otherwise is obviously futile.  I wish you
 > the best, and may God have mercy on whoever has to take over the
 > infrastructure you build after you.

Ok, and when comes the constructive part of your contribution?

 

Meant to send that to the list.  We are all trying to be constructive and helpful by telling you this is a bad plan.  You aren't listening to us, so we're at an impasse.  I sincerely apologize that I cannot provide you with a better alternative, but we're not experts on large scale data backup solutions, so asking us isn't likely to net very useful responses.  If you really want, I use SyncBackSE on my personal machines, and my company uses rsync without trouble, though I know nothing about the intricacies of how we use it.

http://www.2brightsparks.com/syncback/sbse.html
https://www.google.com/search?q=incremental+backup+solutions
http://serverfault.com/search?q=incremental+backup

[Waldemar Augustyn]

Well, I guess, what's noteworthy is that VCS and backup are close enough in function and mission that one might wonder if we'd see convergence at some point.  I bet most of us have seen the use of VCS as a kind of a specialized backup systems.  Wouldn't it be nice to keep /etc under VCS? Didn't we hear sysadmins musing thousands of times?

Mercurial, certainly, is not going in that direction. Incidentally, the much bashed CVS is a lot better suited for this. Its versioning is per file, doesn't complain too much about binaries or large files.  Pretty stable, one can dump just about anything onto it.  As it happened in our very large organization, CVS has survived mercurial invasion by serving as a more capable ftp/file server of sorts.  In fact, it strives in this role.  There are just some things which are best left to CVS over mercurial or git.  Perhaps it could be made into a nice backup system (with mercurial bindings!!).

[Dave S]

Lester Caine lester at lsces.co.uk
wrote on Thu Jul 12 13:27:51 CDT 2012

> That would work, but is
> still not directly addressing your 'backup' requirements. (And BeyondCompare
> does not become part of this either ... that simply allows one to identify files
> which have changed, not back them up).

BeyondCompare has various means of copying the files identified.
Check out snapshots, and in BC3 (but not BC2) the Synch command.

These capabilities may or may not be useful to Markus, but are
definitely designed to support backing up files.

/dps

--
test signature -- please apply at front gate on Tuesdays only.