Bug#1010397: git-annex sync fails in rsync remotes with rsync 3.2.4-1
Daniel
Version: 8.20211123-1
Severity: important
Dear Maintainer,
I have a couple of git-annex repositories with rsync+gpg remotes. Since rsync
3.2.4-1, download from those remotes fails with errors similar to the following
(partly anonymized):
git -c annex.debug=true annex get P1010115.JPG --from host
…
[2022-04-30 13:00:19.357884401] (Utility.Process) process [1635091] read: ssh ["-o","BatchMode=true","-S","../.git/annex/ssh/user@host-key","-o","ControlMaster=auto","-o","ControlPersist=yes","-l","user","-n","-T","user@host-key","true"]
[2022-04-30 13:00:20.397305289] (Utility.Process) process [1635091] done ExitSuccess
[2022-04-30 13:00:20.39881781] (Utility.Process) process [1635096] read: rsync ["-e","'ssh' '-S' '../.git/annex/ssh/user@host-key' '-o' 'ControlMaster=auto' '-o' 'ControlPersist=yes' '-l' 'user' '-T'","--progress","--inplace","user@host-key:/backup/dspace/repositories/repository.git/ff7/84a/'GPGHMACSHA1--f5a2f4c1b3562feff26d0d1d6db70298087fd851/GPGHMACSHA1--f5a2f4c1b3562feff26d0d1d6db70298087fd851'","../.git/annex/tmp/GPGHMACSHA1--f5a2f4c1b3562feff26d0d1d6db70298087fd851"]
rsync: [sender] change_dir "/backup/dspace/repositories/repository.git/ff7/84a/'GPGHMACSHA1--f5a2f4c1b3562feff26d0d1d6db70298087fd851" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1835) [Receiver=3.2.4]
rsync: [Receiver] write error: Broken pipe (32)
[2022-04-30 13:00:20.86166093] (Utility.Process) process [1635096] done ExitFailure 23
rsync exited 23
…
get: 1 failed
The requested file is present on the remote.
I guess rsync isn't happy with the quote, or the fact that the closing quote
went missing somehow:
rsync: [sender] change_dir "/backup/dspace/repositories/repository.git/ff7/84a/'GPGHMACSHA1--f5a2f4c1b3562feff26d0d1d6db70298087fd851" failed: No such file or directory (2)
^
Reverting to rsync 3.2.3-8 fixes this error.
I am not sure whether this is a regression in rsync or an error in git-annex.
Please feel free to point to rsync and I'd be happy to open a bug report there.
Thanks
Daniel
-- System Information:
Debian Release: bookworm/sid
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'stable-security'), (500, 'unstable'), (500, 'testing'), (500, 'stable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i368, i386
Kernel: Linux 5.17.0-1-amd64 (SMP w/4 CPU threads; PREEMPT)
Kernel taint flags: TAINT_USER
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US:en
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Versions of packages git-annex depends on:
ii curl 7.82.0-2
ii git 1:2.36.0-1
ii libc6 2.33-7
ii libffi8 3.4.2-4
ii libgmp10 2:6.2.1+dfsg-3
ii libmagic1 1:5.41-3
ii libsqlite3-0 3.38.2-1
ii netbase 6.3
ii openssh-client 1:9.0p1-1
hi rsync 3.2.3-8
ii zlib1g 1:1.2.11.dfsg-4
Versions of packages git-annex recommends:
ii aria2 1.36.0-1
ii bind9-host 1:9.18.1-1
ii git-remote-gcrypt 1.4-1
ii gnupg 2.2.34-1
ii lsof 4.94.0+dfsg-1
ii nocache 1.1-1+b1
Versions of packages git-annex suggests:
pn adb <none>
pn bup <none>
ii libnss-mdns 0.15.1-1
pn magic-wormhole <none>
pn tahoe-lafs <none>
ii tor 0.4.6.10-1
pn uftp <none>
pn xdot <none>
ii youtube-dl 2021.12.17-1
-- no debconf information
Joey Hess
> "user@host-key:/backup/dspace/repositories/repository.git/ff7/84a/'GPGHMACSHA1--f5a2f4c1b3562feff26d0d1d6db70298087fd851/GPGHMACSHA1--f5a2f4c1b3562feff26d0d1d6db70298087fd851'"
> ,"../.git/annex/tmp/GPGHMACSHA1--f5a2f4c1b3562feff26d0d1d6db70298087fd851"]
> rsync: [sender] change_dir "/backup/dspace/repositories/repository.git/ff7/84a/'GPGHMACSHA1--f5a2f4c1b3562feff26d0d1d6db70298087fd851" failed: No such file or directory (2)
seen in the first excerpt above. But rsync then complains about the path
with that close quote removed.
I'm having a hard time not seeing this as a bug in rsync.
--
see shy jo
Joey Hess
> The rsync command that git-annex runs has a trailing close quote, as
> seen in the first excerpt above. But rsync then complains about the path
> with that close quote removed.
>
> I'm having a hard time not seeing this as a bug in rsync.
## Changes in this version:
### BEHAVIOR CHANGES:
- A new form of arg protection was added that works similarly to the older
[`--protect-args`](rsync.1#opt) (`-s`) option but in a way that avoids
breaking things like rrsync (the restricted rsync script): rsync now uses
backslash escaping for sending "shell-active" characters to the remote
shell. This includes spaces, so fetching a remote file via a simple quoted
filename value now works by default without any extra quoting
This change must be the cause of the problem.
git-annex can be configured to not do its own shell escaping when accessing
a rsync special remote. If your remote is named "foo", you can configure it
that way as follows:
git-annex enableremote foo shellescape=no
--
see shy jo
Sean Whitton
people using newer rsync should use going forward?
--
Sean Whitton
Joey Hess
> > git-annex can be configured to not do its own shell escaping when accessing
> > a rsync special remote. If your remote is named "foo", you can configure it
> > that way as follows:
> >
> > git-annex enableremote foo shellescape=no
>
> Thanks for looking into it. Do you see this as a workaround or what
> people using newer rsync should use going forward?
rsync version and do the same thing automatically. Depends on it
only the local rsync version matters, not the remote one.
--
see shy jo