bup main rejecting local remotes

[Greg Troxel]

For background:

I see it as a bug that where index files and index-cache contents go
is logically the same place as where backup data is placed. It's
easy to give args for this, so it's not a big deal.

I never store data in ~/.bup.

I have a script do-bup-PATH that does index and save, given args.
The index files are based on the args and are in ~/.bup, and the save
destination is one of the args. The essence is
bup save ${INDEX} -n $name -r $repo $path

I call this with aliases/scripts to back up specific directories, not
for "disk crashed" backups, but "I'm going to rototill some of my
data and I want a snapshot."

An example invocation (real names projected to foo):
do-bup-PATH /b0/bup-PATH /home/gdt/foo `hostname`-foo


This used to work in 0.33, but now I get

error: remote /b0/bup-PATH has no colon


Is this a feature? bug?

[Rob Browning]

Greg Troxel <g...@lexort.com> writes:

> For background:
>
> I see it as a bug that where index files and index-cache contents go
> is logically the same place as where backup data is placed. It's
> easy to give args for this, so it's not a big deal.

The next release may help. From the pending note.md:

- bup now prefers the XDG cache location (typically
~/.cache/bup/remote/) for the client index cache, but existing
$BUP_DIR/index-cache directories will take precedence. See
bup-config(5) for additional information.

> This used to work in 0.33, but now I get
>
> error: remote /b0/bup-PATH has no colon

What's the eventual bup command, and is this main or 0.33.x? Given the
message, I suspect main, and if so, then some of the handling of
repositorys on the command line has changed. For example, from the
pending note.md:

- The --remote (-r) argument interpretation has changed. The value is
treated as a URL if it begins with a syntactically valid URL scheme
prefix that contains an "authority" (meaning that it begins with
SCHEME:// as ssh://... does) and anything else is interpreted as a
[user@]host:[path] where the host is no longer optional). file: URLs
are no longer allowed; the semantics were potentially surprising
(e.g. file://p would ssh to host p). Use ssh: URLs instead. The URL
support, though long standing, was previously undocumented. See
REMOTE OPTIONS in bup(1) for further information.

--
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

[Greg Troxel]

Rob Browning <r...@defaultvalue.org> writes:

> Greg Troxel <g...@lexort.com> writes:
>
>> For background:
>>
>> I see it as a bug that where index files and index-cache contents go
>> is logically the same place as where backup data is placed. It's
>> easy to give args for this, so it's not a big deal.
>
> The next release may help. From the pending note.md:
>
> - bup now prefers the XDG cache location (typically
> ~/.cache/bup/remote/) for the client index cache, but existing
> $BUP_DIR/index-cache directories will take precedence. See
> bup-config(5) for additional information.

I guess then I can set BUP_DIR to where I want the bits.

>> This used to work in 0.33, but now I get
>>
>> error: remote /b0/bup-PATH has no colon
>
> What's the eventual bup command, and is this main or 0.33.x?

This is main (from a week or so ago), packaged in wip.

The bup command, with identifiers redacted to host and FOO:

bup save --indexfile=/tmp/bup-PATH-host-FOO -n host-FOO -r /b0/bup-PATH /home/gdt/path/to/FOO

> Given the
> message, I suspect main, and if so, then some of the handling of
> repositorys on the command line has changed. For example, from the
> pending note.md:
>
> - The --remote (-r) argument interpretation has changed. The value is
> treated as a URL if it begins with a syntactically valid URL scheme
> prefix that contains an "authority" (meaning that it begins with
> SCHEME:// as ssh://... does) and anything else is interpreted as a
> [user@]host:[path] where the host is no longer optional). file: URLs
> are no longer allowed; the semantics were potentially surprising
> (e.g. file://p would ssh to host p). Use ssh: URLs instead. The URL
> support, though long standing, was previously undocumented. See
> REMOTE OPTIONS in bup(1) for further information.

file with two // is malformed, and sounds like an interpretation bug.

would file:///b0/bup-PATH be ok? If so, that's easy.

Did this used to ssh? It worked fine in 0.33.x, just using that repo as
if it were BUP_DIR, for the saving part, vs the other parts.

I think it should be possible to specify a directory in the fileystem
(existing bup repo) to save to. Are you saying that's a wrong thing to
want to do?

[Greg Troxel]

trying ssh, I have problems too

the arg to bup save is
-r 'ssh://gdt@localhost/b0/bup-PATH'



Backing up to repo ssh://gdt@localhost/b0/bup-PATH path /home/gdt/path/to/FOO as name host-FOO
Indexing: 1439, done (24803 paths/s).
Traceback (most recent call last):
File "/usr/pkg/lib/bup/bup/main.py", line 221, in main
rc = run_subcmd(cmd_module, subcmd, opt['profile'])
File "/usr/pkg/lib/bup/bup/main.py", line 178, in run_subcmd
return module.main(args)
~~~~~~~~~~~^^^^^^
File "/usr/pkg/lib/bup/bup/cmd/save.py", line 456, in main
dest = repo_for_location(opt.repo, compression_level=opt.compress)
File "/usr/pkg/lib/bup/bup/repo/__init__.py", line 65, in repo_for_location
return RemoteRepo(location, **kwargs)
File "/usr/pkg/lib/bup/bup/repo/remote.py", line 18, in __init__
self.client = client.Client(location, create=create)
~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/pkg/lib/bup/bup/client.py", line 327, in __init__
self._transport = Client.ViaSsh(url)
~~~~~~~~~~~~~^^^^^
File "/usr/pkg/lib/bup/bup/client.py", line 211, in __init__
assert not url.host, url
^^^^^^^^^^^^
AssertionError: URL(scheme=b'ssh', host=b'localhost', port=None, user=b'gdt', path=b'/b0/bup-PATH')


From the assert, it looks parsed correctly, scheme, host, user, path,
and no port so 22.

With a repo of
gdt@localhost:/b0/bup-PATH

I get the same assert, again indicating correct parsing.

AssertionError: URL(scheme=b'ssh', host=b'localhost', port=None, user=b'gdt', path=b'/b0/bup-PATH')


And yes ssh is fine

$ ssh localhost uptime
3:18PM up 22 hrs, 24 users, load averages: 2.24, 2.20, 1.58


I reverted to 0.33.10 and the backup ran fine. back to main, same
error.


I will update bup-git to the tip of main and retest.


I can try to look more, not demanding you do anything, especially any
particular time, but this seems not right.

[Rob Browning]

Greg Troxel <g...@lexort.com> writes:

> file with two // is malformed, and sounds like an interpretation bug.

Hmm, I thought I'd checked fairly carefully, and I currently believe
that's syntactically valid.

> would file:///b0/bup-PATH be ok? If so, that's easy.

I think so, wherever file: URLs are supported (they aren't by --remote
in main since file: isn't remote).

If you haven't read them yet, please see the new sections in bup(1) and
let me know if they're insufficient, "REMOTE OPTIONS" and "REPOSITORY
URLS":

https://codeberg.org/bup/bup/src/branch/main/Documentation/bup.1.md#remote-options
https://codeberg.org/bup/bup/src/branch/main/Documentation/bup.1.md#repository-urls

(It took some time to figure all this out, and implement it, and to
learn more than I'd intended to know about URLs...)

> Did this used to ssh? It worked fine in 0.33.x, just using that repo as
> if it were BUP_DIR, for the saving part, vs the other parts.

I think it probably used the "now testing only" pretend ssh, i.e. when
there was no host, or the host was "-", we'd just run bup as a
subprocess instead of via ssh. The changes in main were intended to
make remote *remote* and to fix all the ambiguities with the previous,
ad-hoc -r syntax.

In main an empty host is no longer allowed (as mentioned in bup(1)) and
the "-" hack is only allowed when BUP_TEST_LEVEL is set in the
environment.

> I think it should be possible to specify a directory in the fileystem
> (existing bup repo) to save to. Are you saying that's a wrong thing to
> want to do?

I'm not sure I follow. Are you wanting --remote to work for non-remote
cases? And if the concern is your existing save command, could that
just be:

bup -d /b0/bup-PATH save --indexfile=/tmp/bup-PATH-host-FOO -n host-FOO \
/home/gdt/path/to/FOO

Hope this helps

[Greg Troxel]

Rob Browning <r...@defaultvalue.org> writes:

> Greg Troxel <g...@lexort.com> writes:
>
>> file with two // is malformed, and sounds like an interpretation bug.
>
> Hmm, I thought I'd checked fairly carefully, and I currently believe
> that's syntactically valid.

I just in firefox went to

file://home/gdt/.bash_aliases

and it ends up at

file:///gdt/.bash_aliases

which is not found. I believe URLs have a host-type semantics field
after the // before the next /, and that for file: that must be empty
(or is ignored) and thus three /.

> I think so, wherever file: URLs are supported (they aren't by --remote
> in main since file: isn't remote).
>
> If you haven't read them yet, please see the new sections in bup(1) and
> let me know if they're insufficient, "REMOTE OPTIONS" and "REPOSITORY
> URLS":
>
> https://codeberg.org/bup/bup/src/branch/main/Documentation/bup.1.md#remote-options
> https://codeberg.org/bup/bup/src/branch/main/Documentation/bup.1.md#repository-urls
>
> (It took some time to figure all this out, and implement it, and to
> learn more than I'd intended to know about URLs...)

I just read that, and it says that remote uses a URL or user@host:path,
with a restriction to ssh and and bup. It then says that file: is a
legit URL, but it's unclear when you'd use it, and to me -- why it's not
ok to use with remote. That's because to me, -r means "the repo to save
to is {here}, not where it usually is" and the remote part is a detail.

BTW the bup: part is confusing because "existing bup server" presumes
that it's on some TCP port, probably, and nobody should do that. I
don't see how to start it, how to point to it, really anything other
than "don't do this anyway, so don't worry".


The file: URL section is confusing, because it blurs the 3 / that are
part of how you create a URL, and / that are part of pathnames. So

file:///b0/bup

is the bath "b0/bup" from the root of the current machine; the first two
// are part of the :// between scheme and authority, and the third /
separates the (empty) authority and the path. It feels like the man
page gives rules that amount to this, but that those rules are a
backconstruction rather then a definition.

>> I think it should be possible to specify a directory in the fileystem
>> (existing bup repo) to save to. Are you saying that's a wrong thing to
>> want to do?
>
> I'm not sure I follow. Are you wanting --remote to work for non-remote
> cases? And if the concern is your existing save command, could that
> just be:
>
> bup -d /b0/bup-PATH save --indexfile=/tmp/bup-PATH-host-FOO -n host-FOO \
> /home/gdt/path/to/FOO

The problem is that this is saying "use that place as BUP_DIR" and then
attempting to cause every known use of putting things in BUP_DIR to put
them someplace else. I really want the place-to-save-to to behave
semantically like a remote, where the config is read for hashsplit,
packs are written, refs updated, and really nothing else. No
index-cache, no index files, and no the other thing I don't know about,
now, or that gets added later.

I think what I'm asking for is for the sub-part of -d that selects the
saving-to repo to be activated, and not any other parts. -r used to do
that, even if it did bup w/o ssh.

I'm not understanding the semantics/purpose for -r being remote as part
of the essence of its being. Stepping back from syntax and how it's
organized, bup can save to a repo in the filesystems, or it can talk to
bup server. When it's talking to a bup server it fetches idx/mix, and
when it's saving directly it doesn't.

I'd be mostly fine with --local-repo /path/to repo, but for scripting,
it's awkward to switch commandline switches because the repo arg is
different. I'd be totally fine with file:///path/to as a remote, too,
if it meant "use it as if it were BUP_DIR, but for saving, not for the
other N things bupdir is for", and especially if that avoided copies of
idx files.



Probably, we're almost to the point of seprating things, and perhaps
that would bring cleanliness.

With index-cache, I could see erroring out if it's in BUP_DIR, and
make people reorg/clean. That's the rip-the-bandaid off approach, but
I tend to prefer that.

With the index, I'd also want that someplace else. Even if we use
~/.bup-local by default, and there's a different BUP_DIR_LOCAL for
that vs BUP_DIR

Maybe the thing is to have two BUP_DIRs conceptually, the working
BUP_DIR and the writing BUP_DIR. But this feels like it ends up with
--remote and file URLS for local repos, that are turned into a
"BUP_DIR" is set.


Maybe I'm the only one not happy about this and of course you shouldn't
feel pressured to do things on my account, if I haven't convinced you
that things aren't right.

[Greg Troxel]

Earlier I tried to use main's -r with an ssh: URL and couldn't get it to
work. Now it's working, so just saying that if you find in prior emails
a "ssh://user@host/path/to/repo doesn't work" vibe, I suspect I had
something wrong.

That should work with both 0.33.x and 0.34, and it seems it's doing what
0.33.x did for a bare path, so that's enough to be no-regression and
for my backups to run :-)

[Rob Browning]

Greg Troxel <g...@lexort.com> writes:

> I just in firefox went to
>
> file://home/gdt/.bash_aliases

Because it has two slashes, then everything after those and up to the
next slash is the "authority":

https://datatracker.ietf.org/doc/html/rfc3986#section-3

But I think file:foo and file:/foo* are valid, though firefox will
rewrite them to file:///foo.

> I just read that, and it says that remote uses a URL or user@host:path,
> with a restriction to ssh and and bup.

Right, --remote currently only allows those two schemes:

...the scheme must be either `ssh` or `bup`; others will be rejected.

> It then says that file: is a legit URL, but it's unclear when you'd
> use it

See the lower priority "General" release notes:

* `bup get` has added a `-S, --source-url URL` option which can
specify a remote source repository. This should usually be
preferred to `bup on HOST get` since it requires less trust in the
remote, and avoids the need for a remote index-cache. See
`bup-get(1)` for additional information.

Though now that I re-read that, I realize the remote in "remote source
repository" there might be confused with the --remote option, so I'll
plan to clarify. The get --source option accepts any URL.

It's also the case that Johannes and I have discussed and are likely to
extend more general URL use elsewhere, later (and we may also want to
make some related adjustments to "VFS references").

> and to me -- why it's not ok to use with remote. That's because to
> me, -r means "the repo to save to is {here}, not where it usually is"
> and the remote part is a detail.

Ahh, well architecturally, --remote is and always has been special,
i.e. it really means "remote" because it always creates a client and
server, and it is accordingly more complex/expensive than the direct
access provided by BUP_DIR/-d.

> BTW the bup: part is confusing because "existing bup server" presumes
> that it's on some TCP port, probably, and nobody should do that. I
> don't see how to start it, how to point to it, really anything other
> than "don't do this anyway, so don't worry".

While we're not removing it right now, I wouldn't expect anyone to use
the bup: scheme directly these days, except perhaps in unusual
circumstances, with someone who is sure they know what they were doing
(otherwise secure transport, etc.). Although bup-mux(1) does still
mention running the server via inetd...

> The file: URL section is confusing, because it blurs the 3 / that are
> part of how you create a URL, and / that are part of pathnames. So
>
> file:///b0/bup
>
> is the bath "b0/bup" from the root of the current machine; the first two
> // are part of the :// between scheme and authority, and the third /
> separates the (empty) authority and the path. It feels like the man
> page gives rules that amount to this, but that those rules are a
> backconstruction rather then a definition.

It's just trying to specify what's possible, though the simplest thing
is to just use an absolute path with file://PATH (i.e. three slashes, as
you mentioned before). Then you don't have to care what's in PATH ---
unless you need to add our "dot encoding" for a relative path.

The REPOSITORY URLS section does attempt to describe that, but perhaps
it needs adjustment.

> The problem is that this is saying "use that place as BUP_DIR" and then
> attempting to cause every known use of putting things in BUP_DIR to put
> them someplace else. I really want the place-to-save-to to behave
> semantically like a remote, where the config is read for hashsplit,
> packs are written, refs updated, and really nothing else. No
> index-cache, no index files, and no the other thing I don't know about,
> now, or that gets added later.
>
> I think what I'm asking for is for the sub-part of -d that selects the
> saving-to repo to be activated, and not any other parts. -r used to do
> that, even if it did bup w/o ssh.

As mentioned Johannes and I have discussed simialar changes at some
length, and they're fairly likely to involve "new options", so that we
don't have to break backward compatiblity.

The current changes were just a first step there, trying to actually
document --remote's semantics, avoid ambiguous values, make the parsing
more rigorous, add tests, etc. The intention is to eventually add
options/settings that are more general, like like bup get's new
--source, and then you may never use --remote again.

We've also started trying to separate out some of the entangled bits
like the index-cache and the filesystem index, hence those initial
XDG_CACHE_HOME changes.

> Maybe I'm the only one not happy about this and of course you
> shouldn't feel pressured to do things on my account, if I haven't
> convinced you that things aren't right.

Oh, I suspect we're convinced, just not there yet.

Thanks for the help

[Greg Troxel]

Rob Browning <r...@defaultvalue.org> writes:

> Greg Troxel <g...@lexort.com> writes:
>
>> I just in firefox went to
>>
>> file://home/gdt/.bash_aliases
>
> Because it has two slashes, then everything after those and up to the
> next slash is the "authority":
>
> https://datatracker.ietf.org/doc/html/rfc3986#section-3
>
> But I think file:foo and file:/foo* are valid, though firefox will
> rewrite them to file:///foo.

wow. I guess good for you if handled, but I view the file:/path form as
irregular, or at least not Third Normal Form (What would Date do?).

> See the lower priority "General" release notes:
>
> * `bup get` has added a `-S, --source-url URL` option which can
> specify a remote source repository. This should usually be
> preferred to `bup on HOST get` since it requires less trust in the
> remote, and avoids the need for a remote index-cache. See
> `bup-get(1)` for additional information.
>
> Though now that I re-read that, I realize the remote in "remote source
> repository" there might be confused with the --remote option, so I'll
> plan to clarify. The get --source option accepts any URL.
>
> It's also the case that Johannes and I have discussed and are likely to
> extend more general URL use elsewhere, later (and we may also want to
> make some related adjustments to "VFS references").

That is confusing, but I have never wanted to use bup on enough to get
my head around it. Surely not any URL, though, maybe any URL handled by
"<python library x>". There are a large number of odd schemes, I
thought.

>> and to me -- why it's not ok to use with remote. That's because to
>> me, -r means "the repo to save to is {here}, not where it usually is"
>> and the remote part is a detail.
>
> Ahh, well architecturally, --remote is and always has been special,
> i.e. it really means "remote" because it always creates a client and
> server, and it is accordingly more complex/expensive than the direct
> access provided by BUP_DIR/-d.

I can see where you are coming from but as a user I want to be able to
say "put the bits here" with a consistent syntax (that doesn't change
args based on where, just a string that could be passed in).

>> BTW the bup: part is confusing because "existing bup server" presumes
>> that it's on some TCP port, probably, and nobody should do that. I
>> don't see how to start it, how to point to it, really anything other
>> than "don't do this anyway, so don't worry".
>
> While we're not removing it right now, I wouldn't expect anyone to use
> the bup: scheme directly these days, except perhaps in unusual
> circumstances, with someone who is sure they know what they were doing
> (otherwise secure transport, etc.). Although bup-mux(1) does still
> mention running the server via inetd...

Even if I was ok with firewall/etc. config, it still was not
understandable. Maybe stick in a comment that it's not in general
recommended, and drop the inetd stuff - that sounds... unwise.

> It's just trying to specify what's possible, though the simplest thing
> is to just use an absolute path with file://PATH (i.e. three slashes, as
> you mentioned before). Then you don't have to care what's in PATH ---
> unless you need to add our "dot encoding" for a relative path.
>
> The REPOSITORY URLS section does attempt to describe that, but perhaps
> it needs adjustment.

I would be inclined to document file:///home/user/bupdir and not suggest
other forms. It's ok to parse them correctly, but I see no benefit to
users trying to play the "URLs are complicated" game.

> As mentioned Johannes and I have discussed simialar changes at some
> length, and they're fairly likely to involve "new options", so that we
> don't have to break backward compatiblity.
>
> The current changes were just a first step there, trying to actually
> document --remote's semantics, avoid ambiguous values, make the parsing
> more rigorous, add tests, etc. The intention is to eventually add
> options/settings that are more general, like like bup get's new
> --source, and then you may never use --remote again.
>
> We've also started trying to separate out some of the entangled bits
> like the index-cache and the filesystem index, hence those initial
> XDG_CACHE_HOME changes.

Thanks - an eventual path to orthogonal logical controls sounds good.

[Rob Browning]

Greg Troxel <g...@lexort.com> writes:

> That is confusing, but I have never wanted to use bup on enough to get
> my head around it. Surely not any URL, though, maybe any URL handled by
> "<python library x>". There are a large number of odd schemes, I
> thought.

Did you mean "bup get" perhaps? And I just meant "any URL" as compared
to -r's "just bup: and ssh:", i.e. it'll also support file://.

> I can see where you are coming from but as a user I want to be able to
> say "put the bits here" with a consistent syntax (that doesn't change
> args based on where, just a string that could be passed in).

Sure. I was just referring to the precedent, and "what we have", not
what we may want.

[Rob Browning]

Rob Browning <r...@defaultvalue.org> writes:

> Did you mean "bup get" perhaps? And I just meant "any URL" as compared
> to -r's "just bup: and ssh:", i.e. it'll also support file://.

Oh, now I see, you were presumably referring to the release note's
recommendation to prefer "bup get --source-url ..." to "bup on get ...".

[Greg Troxel]

Rob Browning <r...@defaultvalue.org> writes:

> Rob Browning <r...@defaultvalue.org> writes:
>
>> Did you mean "bup get" perhaps? And I just meant "any URL" as compared
>> to -r's "just bup: and ssh:", i.e. it'll also support file://.
>
> Oh, now I see, you were presumably referring to the release note's
> recommendation to prefer "bup get --source-url ..." to "bup on get ...".

yes.