Cloning mercurial repo with subrepo without network connection

[Sergey Kishchenko]

Hi,

I have a problem with mercurial subrepos that prevents me from using all its
advantages. The problem is described below:

Let's assume we have a repo /tmp/repos/test1 and a parent repo /tmp/repos/parent
with .hgsub as follows:
test1 = /tmp/repos/test1

Cloning works fine in this case:
$ cd /tmp/repos
$ hg clone parent parent2
updating to branch default
cloning subrepo test1 from /tmp/repos/test1
3 files updated, 0 files merged, 0 files removed, 0 files unresolved

But if there is no access to the test1 repo, the clone will fail:
$ hg clone parent2 paren3
updating to branch default
abort: repository /tmp/repos/test1 not found!

That means it's not possible to use mercurial as DVCS with subrepos, mercurial
needs centralized repo for the subrepos. The problem with non working update was
already discussed in
http://article.gmane.org/gmane.comp.version-control.mercurial.general/23378/ but
the problem above is different: mercurial doesn't use original repo as a source
for subrepo cloning. Is it possible to override its behavior?

_______________________________________________
Mercurial mailing list
Merc...@selenic.com
http://selenic.com/mailman/listinfo/mercurial

[Matt Mackall]

On Tue, 2011-10-11 at 11:35 +0000, Sergey Kishchenko wrote:
> Hi,
>
> I have a problem with mercurial subrepos that prevents me from using all its
> advantages. The problem is described below:
>
> Let's assume we have a repo /tmp/repos/test1 and a parent repo /tmp/repos/parent
> with .hgsub as follows:
> test1 = /tmp/repos/test1
>
> Cloning works fine in this case:
> $ cd /tmp/repos
> $ hg clone parent parent2
> updating to branch default
> cloning subrepo test1 from /tmp/repos/test1
> 3 files updated, 0 files merged, 0 files removed, 0 files unresolved
>
> But if there is no access to the test1 repo, the clone will fail:
> $ hg clone parent2 paren3
> updating to branch default
> abort: repository /tmp/repos/test1 not found!
>
> That means it's not possible to use mercurial as DVCS with subrepos, mercurial
> needs centralized repo for the subrepos.

If you put your subrepo at an absolute path, then yes, you're absolutely
right: it absolutely has to be accessible. There are relatively easy
ways around this.

--
Mathematics is the supreme nostalgia of our time.

[Isaac Jurado]

On Tue, Oct 11, 2011 at 1:35 PM, Sergey Kishchenko <voi...@gmail.com> wrote:
> Hi,
>
> I have a problem with mercurial subrepos that prevents me from using
> all its advantages. The problem is described below:
>
> Let's assume we have a repo /tmp/repos/test1 and a parent repo /tmp/repos/parent
> with .hgsub as follows:
> test1 = /tmp/repos/test1
>
> Cloning works fine in this case:
> $ cd /tmp/repos
> $ hg clone parent parent2
> updating to branch default
> cloning subrepo test1 from /tmp/repos/test1
> 3 files updated, 0 files merged, 0 files removed, 0 files unresolved
>
> But if there is no access to the test1 repo, the clone will fail:
> $ hg clone parent2 paren3
> updating to branch default
> abort: repository /tmp/repos/test1 not found!
>
> That means it's not possible to use mercurial as DVCS with subrepos,
> mercurial needs centralized repo for the subrepos. The problem with
> non working update was already discussed in
> http://article.gmane.org/gmane.comp.version-control.mercurial.general/23378/
> but the problem above is different: mercurial doesn't use original
> repo as a source for subrepo cloning. Is it possible to override its
> behavior?

Have you read the "subpaths" section from the hgrc(5) man page?

--
Isaac Jurado

"The noblest pleasure is the joy of understanding"
Leonardo da Vinci

[Sergey Kishchenko]

The example above with the absolute paths in the file system was only
intended to demonstrate the problem. That is, when subrepo is
inaccessible, why it can't be cloned from the parent's subrepo that
still exists in filesystem(in example above parent/test1 is a full
clone that is accessible)? In real world subrepos urls usually are
http urls and they become inaccessible when there is no network
connection. But the full clone of the subrepo exists and is
accessible. Why mercurial doesn't use this subrepo clone? Are there
any ways around this restriction?

2011/10/11 Matt Mackall <m...@selenic.com>:

[Sergey Kishchenko]

Hi, thanks for the answer. It seems the subpaths rewriting rules can
be a way around the issue. But these rewriting rules just allow to
replace the centralized repo url with new one without affecting the
.hgsub. It doesn't make the mercurial a distributed VCS once again. I
can't find any discussion or document that describes why subrepos work
in the way they work now.

2011/10/11 Isaac Jurado <dipt...@gmail.com>:

[Matt Mackall]

On Tue, 2011-10-11 at 21:24 +0300, Sergey Kishchenko wrote:
> The example above with the absolute paths in the file system was only
> intended to demonstrate the problem. That is, when subrepo is
> inaccessible, why it can't be cloned from the parent's subrepo that
> still exists in filesystem(in example above parent/test1 is a full
> clone that is accessible)?

(I see my last punning hint was too subtle, sorry.)

A URL is an absolute path. If you tell Mercurial to use an absolute
path, Mercurial will do just that. Whether you have a local copy is
irrelevant. If you want Mercurial to use a relative path for a subrepo,
then do:

lib = lib

Then when you clone http://server/foo, mercurial will clone the subrepo
http://server/foo/lib too. When you clone /home/sergey/foo, it'll
clone /home/sergey/foo.

--
Mathematics is the supreme nostalgia of our time.

--
Mathematics is the supreme nostalgia of our time.

[Isaac Jurado]

On Tue, Oct 11, 2011 at 8:30 PM, Sergey Kishchenko <voi...@gmail.com> wrote:
>
> Hi, thanks for the answer. It seems the subpaths rewriting rules can
> be a way around the issue. But these rewriting rules just allow to
> replace the centralized repo url with new one without affecting the
> .hgsub. It doesn't make the mercurial a distributed VCS once again. I
> can't find any discussion or document that describes why subrepos work
> in the way they work now.

You can define theese rules on a per-repository basis if you define them
in the repo/.hg/hgrc file. In fact, they are applied sequentially as
the RC files are processed: first /etc/mercurial/hgrc, then $HOME/.hgrc
and then repo/.hg/hgrc.

I don't know how much more magic are you asking for. Not all other DVCS
implement a subrepository equivalent functionality, and those that do
approach the problem very similarly to Mercurial.

I'm sure if you have a better idea on how to be more distribute while
maintaining a reasonable command API, the developers will be glad to
read it.

Cheers.

[Arne Babenhauserheide]

Am Dienstag, 11. Oktober 2011, 08:29:47 schrieb Matt Mackall:
> If you put your subrepo at an absolute path, then yes, you're absolutely
> right: it absolutely has to be accessible. There are relatively easy
> ways around this.

I have that problem, too. I’d love to have an “ignore subrepo clone errors”
option for that.

Best wishes,
Arne

[Matt Mackall]

You'd love a "get a broken checkout resulting in a future broken commit"
option? How odd.

If you can get by without a subrepo in your clone, then I suggest to you
that you rethink having it as a subrepo.

Also, everyone working with subrepos should carefully read everything on
this page, but especially 3.1 and 3.2:

http://mercurial.selenic.com/wiki/Subrepository#Recommendations

[Arne Babenhauserheide]

Am Dienstag, 11. Oktober 2011, 19:11:13 schrieb Matt Mackall:
> On Wed, 2011-10-12 at 00:08 +0200, Arne Babenhauserheide wrote:
> > Am Dienstag, 11. Oktober 2011, 08:29:47 schrieb Matt Mackall:
> > > If you put your subrepo at an absolute path, then yes, you're
> > > absolutely right: it absolutely has to be accessible. There are
> > > relatively easy ways around this.
> >
> > I have that problem, too. I’d love to have an “ignore subrepo clone
> > errors” option for that.
>
> You'd love a "get a broken checkout resulting in a future broken commit"
> option? How odd.

I’d love a “it’s not there, darn it, we’ll treat it as unchanged when
committing” option.

I use subrepos for including story-repos in a roleplaying-ruleset. They are
not essential, but they should be there for new developers, and I want to be
able to define a consistent state, so rules and stories match each other.

Best wishes,
Arne

[Franklin M. Siler]

On Tue, Oct 11, 2011 at 20:47, Arne Babenhauserheide <arne...@web.de> wrote:
> Am Dienstag, 11. Oktober 2011, 19:11:13 schrieb Matt Mackall:
>> On Wed, 2011-10-12 at 00:08 +0200, Arne Babenhauserheide wrote:
>> > Am Dienstag, 11. Oktober 2011, 08:29:47 schrieb Matt Mackall:
>> > > If you put your subrepo at an absolute path, then yes, you're
>> > > absolutely right: it absolutely has to be accessible. There are
>> > > relatively easy ways around this.
>> >
>> > I have that problem, too. I’d love to have an “ignore subrepo clone
>> > errors” option for that.
>>

I would like to second this motion, or perhaps have a different
implementation: why not simply allow a subpath to be "!" which means
to ignore that repo?

>> You'd love a "get a broken checkout resulting in a future broken commit"
>> option? How odd.
>

Depending on use case, that is not at all what he's proposing. I can
envision plenty of use cases where a suprepo isn't mandatory or even
really related to the parent; it's merely a convenience. The point is
that having the feature to optionally ignore a subrepo in some clones
does not break suprepo functionality for those who would like it to
keep working as it does now.

> I’d love a “it’s not there, darn it, we’ll treat it as unchanged when
> committing” option.
>

my suggestion is slightly different- simply let the user specify to
ignore a given suprepo using subpaths, as opposed to a command-line
option to "just this one time, ignore the subrepo".

> I use subrepos for including story-repos in a roleplaying-ruleset. They are
> not essential, but they should be there for new developers, and I want to be
> able to define a consistent state, so rules and stories match each other.
>

My use case is somewhat different: I have a repo for my home
directory, then have subrepos for projects, archive, etc.; the thing
is I want to be able to select when I clone projects and archive
because they are large- I'd like to be able to clone my dotfiles
without copying over everything else.

Cheers,

Frank

[Angel Ezquerra]

On Wed, Oct 12, 2011 at 1:43 AM, Isaac Jurado <dipt...@gmail.com> wrote:
> On Tue, Oct 11, 2011 at 8:30 PM, Sergey Kishchenko <voi...@gmail.com> wrote:
>>
>> Hi, thanks for the answer. It seems the subpaths rewriting rules can
>> be a way around the issue. But these rewriting rules just allow to
>> replace the centralized repo url with new one without affecting the
>> .hgsub. It doesn't make the mercurial a distributed VCS once again. I
>> can't find any discussion or document that describes why subrepos work
>> in the way they work now.
>
> You can define theese rules on a per-repository basis if you define them
> in the repo/.hg/hgrc file.  In fact, they are applied sequentially as
> the RC files are processed: first /etc/mercurial/hgrc, then $HOME/.hgrc
> and then repo/.hg/hgrc.
>
> I don't know how much more magic are you asking for.  Not all other DVCS
> implement a subrepository equivalent functionality, and those that do
> approach the problem very similarly to Mercurial.
>
> I'm sure if you have a better idea on how to be more distribute while
> maintaining a reasonable command API, the developers will be glad to
> read it.

Not long ago there was some discussion (on the dev mailing list, I
believe) where Martin Geisler suggested that the local subrepo clones
should be used as a sort of "cache" so that performing a local clone
would use the local subrepo repositories as a source by default. I
think that would be some neat magic :-)

Angel

[Sergey Kishchenko]

It seems like a solution for me. Thank you!

2011/10/11 Matt Mackall <m...@selenic.com>:

[Arne Babenhauserheide]

Am Dienstag, 11. Oktober 2011, 23:58:38 schrieb Franklin M. Siler:
> > I’d love a “it’s not there, darn it, we’ll treat it as unchanged when
> > committing” option.
>
> my suggestion is slightly different- simply let the user specify to
> ignore a given suprepo using subpaths, as opposed to a command-line
> option to "just this one time, ignore the subrepo".

Or perhaps have a config option in the .hg/hgrc:

[subrepos]
ignore.1 = subpath # what you put as first item into .hgsub

On a clone with nonaccessible subrepos, hg could ask if they should be
ignored, and if yes add them to the .hg/hgrc as ignored. Also the options
could be specified on the commandline.

Important for me would be that this is not versioned. I don’t want it to
propagate.

Best wishes,
Arne
--
Ein Mann wird auf der Straße mit einem Messer bedroht.
Zwei Polizisten sind sofort da und halten ein Transparent davor.

"Illegale Szene. Niemand darf das sehen."

Der Mann wird ausgeraubt, erstochen und verblutet,
denn die Polizisten haben beide Hände voll zu tun.

Willkommen in Deutschland. Zensur ist schön.
( http://draketo.de/stichwort/zensur )

[Franklin M. Siler]

On Fri, Oct 14, 2011 at 01:11, Arne Babenhauserheide <arne...@web.de> wrote:
> Am Dienstag, 11. Oktober 2011, 23:58:38 schrieb Franklin M. Siler:
>>
>> my suggestion is slightly different- simply let the user specify to
>> ignore a given suprepo using subpaths, as opposed to a command-line
>> option to "just this one time, ignore the subrepo".
>
> Or perhaps have a config option in the .hg/hgrc:
>
>        [subrepos]
>        ignore.1 = subpath # what you put as first item into .hgsub
>

Might I suggest instead rolling this into subpaths, with the syntax

[subpaths]
subpath = !

This is consistent with the config syntax to disable extensions, and
can be put in .hg/hgrc on a per-clone basis.