Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
Bug#1010024: git-buildpackage: fails to import pristine-tar: File name too long
50 views
Skip to first unread message
Drew Parsons
Apr 22, 2022, 9:30:03 AM4/22/22
to
Package: git-buildpackage
Version: 0.9.25
Severity: normal
Control: affects -1 src:rclone
Control: block 1001261 by -1
I'm trying to update the rclone package, using gbp import-orig in the
"usual" way, with uscan, pristine-tar and importing into the
experimental branch.
The import is failing, apparently due to using a git commit in a
filename during the pristine-tar step.
$ git clone g...@salsa.debian.org:go-team/packages/rclone.git
$ cd rclone
$ git checkout upstream; git checkout pristine-tar; git checkout experimental
$ git branch
* experimental
master
pristine-tar
upstream
$ uscan --report
uscan: Newest version of rclone on remote site is 1.58.0, local version is 1.53.3
uscan: => Newer package available from:
=> https://github.com/rclone/rclone/archive/refs/tags/v1.58.0.tar.gz
$ gbp import-orig --verbose --uscan --pristine-tar --debian-branch=experimental
gbp:debug: ['git', 'rev-parse', '--show-cdup']
gbp:debug: ['git', 'rev-parse', '--is-bare-repository']
gbp:debug: ['git', 'rev-parse', '--git-dir']
gbp:debug: ['git', 'for-each-ref', '--format=%(refname:short)', 'refs/heads/']
gbp:debug: ['git', 'show-ref', '--verify', 'refs/heads/upstream']
gbp:debug: ['git', 'status', '--porcelain']
gbp:info: Launching uscan...
gbp:info: Using uscan downloaded tarball ../rclone_1.58.0.orig.tar.gz
What is the upstream version? [1.58.0]
gbp:debug: ['git', 'tag', '-l', 'upstream/1.58.0']
gbp:debug: tar ['-C', '../tmpkiz0ywn3', '-a', '-xf', '../rclone_1.58.0.orig.tar.gz'] []
gbp:debug: Unpacked '../rclone_1.58.0.orig.tar.gz' to '../tmpkiz0ywn3/rclone-1.58.0'
gbp:info: <DebianUpstreamSource path='../rclone_1.58.0.orig.tar.gz' signaturefile=None>
gbp:info: Importing '../rclone_1.58.0.orig.tar.gz' to branch 'upstream'...
gbp:info: Source package is rclone
gbp:info: Upstream version is 1.58.0
gbp:debug: ['git', 'show-ref', '--verify', 'refs/heads/upstream']
gbp:debug: ['git', 'rev-parse', '--quiet', '--verify', 'upstream']
gbp:debug: ['git', 'add', '-f', '.']
gbp:debug: ['git', 'write-tree']
gbp:debug: ['git', 'rev-parse', '--quiet', '--verify', 'upstream']
gbp:debug: ['git', 'commit-tree', '260bd813325c0b30022ab31e09fc3d9ab570e072', '-p', '4d715ab7de2dd24d802fe89bd79048ba674a25ea']
gbp:debug: ['git', 'update-ref', '-m', 'gbp: New upstream version 1.58.0', 'refs/heads/upstream', '70bb7aad80b8c50a88d59be119fbdc5c0f72ad3c', '4d715ab7de2dd24d802fe89bd79048ba674a25ea']
gbp:debug: ['git', 'show-ref', '--verify', 'refs/heads/pristine-tar']
gbp:debug: ['git', 'rev-parse', '--quiet', '--verify', 'pristine-tar']
gbp:debug: ['git', 'ls-tree', '-z', 'upstream', '--']
gbp:debug: ['git', 'mktree', '-z']
gbp:debug: pristine-tar [] ['commit', '../rclone_1.58.0.orig.tar.gz', '260bd813325c0b30022ab31e09fc3d9ab570e072']
gbp:error: Import of ../rclone_1.58.0.orig.tar.gz failed: Couldn't commit to 'pristine-tar' with upstream '260bd813325c0b30022ab31e09fc3d9ab570e072': mkdir /tmp/pristine-tar.iXqSpNTndf/workdir/rclone-1.58.0/cmd/bisync/testdata/test_extended_char_paths/golden/normal-sync._testdir_path1_\346\270\254\350\251\246_\320\240\321\203\321\201\321\201\320\272\320\270\320\271_____\304\233_\303\241\303\261.._testdir_path2_\346\270\254\350\251\246_\320\240\321\203\321\201\321\201\320\272\320\270\320\271_____\304\233_\303\241\303\261.copy1to2.que: File name too long at /usr/bin/pristine-tar line 451.
gbp:error: Error detected, Will roll back changes.
gbp:info: Rolling back branch upstream by resetting it to 4d715ab7de2dd24d802fe89bd79048ba674a25ea
gbp:debug: ['git', 'update-ref', '-m', 'gbp import-orig: failure rollback of upstream', 'refs/heads/upstream', '4d715ab7de2dd24d802fe89bd79048ba674a25ea']
gbp:info: Rolling back branch pristine-tar by resetting it to f92427a73c1a897f36b08dbf40a402c144d75b73
gbp:debug: ['git', 'update-ref', '-m', 'gbp import-orig: failure rollback of pristine-tar', 'refs/heads/pristine-tar', 'f92427a73c1a897f36b08dbf40a402c144d75b73']
gbp:error: Rolled back changes after import error.
gbp:debug: rm ['-rf', '../tmpkiz0ywn3'] []
-- System Information:
Debian Release: bookworm/sid
APT prefers unstable
APT policy: (500, 'unstable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386
Kernel: Linux 5.16.0-6-amd64 (SMP w/8 CPU threads; PREEMPT)
Locale: LANG=en_AU.UTF-8, LC_CTYPE=en_AU.UTF-8 (charmap=UTF-8), LANGUAGE=en_AU:en
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Versions of packages git-buildpackage depends on:
ii devscripts 2.22.1
ii git 1:2.35.2-1
ii man-db 2.10.2-1
ii python3 3.10.4-1
ii python3-dateutil 2.8.1-6
ii python3-pkg-resources 59.6.0-1.2
ii sensible-utils 0.0.17
Versions of packages git-buildpackage recommends:
ii cowbuilder 0.89
ii pbuilder 0.231
ii pristine-tar 1.49
ii python3-requests 2.27.1+dfsg-1
Versions of packages git-buildpackage suggests:
pn python3-notify2 <none>
ii sudo 1.9.10-3
ii unzip 6.0-26
-- no debconf information
Version: 0.9.25
Severity: normal
Control: affects -1 src:rclone
Control: block 1001261 by -1
I'm trying to update the rclone package, using gbp import-orig in the
"usual" way, with uscan, pristine-tar and importing into the
experimental branch.
The import is failing, apparently due to using a git commit in a
filename during the pristine-tar step.
$ git clone g...@salsa.debian.org:go-team/packages/rclone.git
$ cd rclone
$ git checkout upstream; git checkout pristine-tar; git checkout experimental
$ git branch
* experimental
master
pristine-tar
upstream
$ uscan --report
uscan: Newest version of rclone on remote site is 1.58.0, local version is 1.53.3
uscan: => Newer package available from:
=> https://github.com/rclone/rclone/archive/refs/tags/v1.58.0.tar.gz
$ gbp import-orig --verbose --uscan --pristine-tar --debian-branch=experimental
gbp:debug: ['git', 'rev-parse', '--show-cdup']
gbp:debug: ['git', 'rev-parse', '--is-bare-repository']
gbp:debug: ['git', 'rev-parse', '--git-dir']
gbp:debug: ['git', 'for-each-ref', '--format=%(refname:short)', 'refs/heads/']
gbp:debug: ['git', 'show-ref', '--verify', 'refs/heads/upstream']
gbp:debug: ['git', 'status', '--porcelain']
gbp:info: Launching uscan...
gbp:info: Using uscan downloaded tarball ../rclone_1.58.0.orig.tar.gz
What is the upstream version? [1.58.0]
gbp:debug: ['git', 'tag', '-l', 'upstream/1.58.0']
gbp:debug: tar ['-C', '../tmpkiz0ywn3', '-a', '-xf', '../rclone_1.58.0.orig.tar.gz'] []
gbp:debug: Unpacked '../rclone_1.58.0.orig.tar.gz' to '../tmpkiz0ywn3/rclone-1.58.0'
gbp:info: <DebianUpstreamSource path='../rclone_1.58.0.orig.tar.gz' signaturefile=None>
gbp:info: Importing '../rclone_1.58.0.orig.tar.gz' to branch 'upstream'...
gbp:info: Source package is rclone
gbp:info: Upstream version is 1.58.0
gbp:debug: ['git', 'show-ref', '--verify', 'refs/heads/upstream']
gbp:debug: ['git', 'rev-parse', '--quiet', '--verify', 'upstream']
gbp:debug: ['git', 'add', '-f', '.']
gbp:debug: ['git', 'write-tree']
gbp:debug: ['git', 'rev-parse', '--quiet', '--verify', 'upstream']
gbp:debug: ['git', 'commit-tree', '260bd813325c0b30022ab31e09fc3d9ab570e072', '-p', '4d715ab7de2dd24d802fe89bd79048ba674a25ea']
gbp:debug: ['git', 'update-ref', '-m', 'gbp: New upstream version 1.58.0', 'refs/heads/upstream', '70bb7aad80b8c50a88d59be119fbdc5c0f72ad3c', '4d715ab7de2dd24d802fe89bd79048ba674a25ea']
gbp:debug: ['git', 'show-ref', '--verify', 'refs/heads/pristine-tar']
gbp:debug: ['git', 'rev-parse', '--quiet', '--verify', 'pristine-tar']
gbp:debug: ['git', 'ls-tree', '-z', 'upstream', '--']
gbp:debug: ['git', 'mktree', '-z']
gbp:debug: pristine-tar [] ['commit', '../rclone_1.58.0.orig.tar.gz', '260bd813325c0b30022ab31e09fc3d9ab570e072']
gbp:error: Import of ../rclone_1.58.0.orig.tar.gz failed: Couldn't commit to 'pristine-tar' with upstream '260bd813325c0b30022ab31e09fc3d9ab570e072': mkdir /tmp/pristine-tar.iXqSpNTndf/workdir/rclone-1.58.0/cmd/bisync/testdata/test_extended_char_paths/golden/normal-sync._testdir_path1_\346\270\254\350\251\246_\320\240\321\203\321\201\321\201\320\272\320\270\320\271_____\304\233_\303\241\303\261.._testdir_path2_\346\270\254\350\251\246_\320\240\321\203\321\201\321\201\320\272\320\270\320\271_____\304\233_\303\241\303\261.copy1to2.que: File name too long at /usr/bin/pristine-tar line 451.
gbp:error: Error detected, Will roll back changes.
gbp:info: Rolling back branch upstream by resetting it to 4d715ab7de2dd24d802fe89bd79048ba674a25ea
gbp:debug: ['git', 'update-ref', '-m', 'gbp import-orig: failure rollback of upstream', 'refs/heads/upstream', '4d715ab7de2dd24d802fe89bd79048ba674a25ea']
gbp:info: Rolling back branch pristine-tar by resetting it to f92427a73c1a897f36b08dbf40a402c144d75b73
gbp:debug: ['git', 'update-ref', '-m', 'gbp import-orig: failure rollback of pristine-tar', 'refs/heads/pristine-tar', 'f92427a73c1a897f36b08dbf40a402c144d75b73']
gbp:error: Rolled back changes after import error.
gbp:debug: rm ['-rf', '../tmpkiz0ywn3'] []
-- System Information:
Debian Release: bookworm/sid
APT prefers unstable
APT policy: (500, 'unstable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386
Kernel: Linux 5.16.0-6-amd64 (SMP w/8 CPU threads; PREEMPT)
Locale: LANG=en_AU.UTF-8, LC_CTYPE=en_AU.UTF-8 (charmap=UTF-8), LANGUAGE=en_AU:en
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Versions of packages git-buildpackage depends on:
ii devscripts 2.22.1
ii git 1:2.35.2-1
ii man-db 2.10.2-1
ii python3 3.10.4-1
ii python3-dateutil 2.8.1-6
ii python3-pkg-resources 59.6.0-1.2
ii sensible-utils 0.0.17
Versions of packages git-buildpackage recommends:
ii cowbuilder 0.89
ii pbuilder 0.231
ii pristine-tar 1.49
ii python3-requests 2.27.1+dfsg-1
Versions of packages git-buildpackage suggests:
pn python3-notify2 <none>
ii sudo 1.9.10-3
ii unzip 6.0-26
-- no debconf information
Sergio Durigan Junior
Jul 14, 2022, 3:10:04 PM7/14/22
to
because it's the former that's complaining about the long filename.
On a side note, I also believe that it's worth proceeding with rclone's
update and just resort to Files-Excluded meanwhile in order to remove
these tests from the orig tarball.
A quick attempt with the following excerpt on d/copyright succeeded
here:
--8<---------------cut here---------------start------------->8---
Files-Excluded:
cmd/bisync/testdata/test_extended_char_paths*
--8<---------------cut here---------------end--------------->8---
Cheers,
--
Sergio
GPG key ID: 237A 54B1 0287 28BF 00EF 31F4 D0EB 7628 65FC 5E36
Please send encrypted e-mail if possible
https://sergiodj.net/
Matthew Vernon
Oct 31, 2022, 7:50:03 AM10/31/22
to
Hi,
I've reassigned this bug from git-buildpackage to pristine-tar; as
Sergio says, it's pristine-tar that's failing.
I've pushed this to important, since it is preventing rclone packaging
from proceeding (without bodges at least); might be RC? I think the
problem relates to pristine-tar's handling of non-ascii filenames.
I used gbp import-orig --verbose --uscan --pristine-tar to tell me what
args it was passing to pristine-tar, and then re-ran that command adding
-v -d to get more debug output (at the bottom of this mail)
The complained-of file is in the upstream tarball:
cmd/bisync/testdata/test_extended_char_paths/golden/normal-sync._testdir_path1_測試_Русский_____ě_áñ.._testdir_path2_測試_Русский_____ě_áñ.copy1to2.que
...and in upstream git:
https://github.com/rclone/rclone/blob/v1.60.0/cmd/bisync/testdata/test_extended_char_paths/golden/normal-sync._testdir_path1_%E6%B8%AC%E8%A9%A6_%D0%A0%D1%83%D1%81%D1%81%D0%BA%D0%B8%D0%B9_____%C4%9B_%C3%A1%C3%B1.._testdir_path2_%E6%B8%AC%E8%A9%A6_%D0%A0%D1%83%D1%81%D1%81%D0%BA%D0%B8%D0%B9_____%C4%9B_%C3%A1%C3%B1.copy1to2.que
So I think the problem may be both in pristine-tar's decision that that
file is missing in the manifest, and then in whatever it's doing to make
a filename too long - which is a bit confusing, since if I do the
"mkdir" that it says is failing, it works just fine...
matthew@tsk:~/packages/rclone$ pristine-tar -v -d commit
../rclone_1.60.0.orig.tar.gz 28901851aa753bee063df1fff99ee90f097fb2e9
pristine-tar: git archive --format=tar
28901851aa753bee063df1fff99ee90f097fb2e9 | (cd
'/tmp/pristine-tar.QSprqRxiCj' && tar x)
pristine-tar: set subdir to rclone-1.60.0
pristine-tar: subdir is rclone-1.60.0
pristine-tar: mkdir /tmp/pristine-tar.nK854kS9zf/workdir
pristine-tar: mv /tmp/pristine-tar.QSprqRxiCj
/tmp/pristine-tar.nK854kS9zf/workdir/rclone-1.60.0
pristine-tar:
rclone-1.60.0/cmd/bisync/testdata/test_extended_char_paths/golden/normal-sync._testdir_path1_\346\270\254\350\251\246_\320\240\321\203\321\201\321\201\320\272\320\270\320\271_____\304\233_\303\241\303\261.._testdir_path2_\346\270\254\350\251\246_\320\240\321\203\321\201\321\201\320\272\320\270\320\271_____\304\233_\303\241\303\261.copy1to2.que
is listed in the manifest but may not be present in the source directory
pristine-tar: creating missing
rclone-1.60.0/cmd/bisync/testdata/test_extended_char_paths/golden/normal-sync._testdir_path1_\346\270\254\350\251\246_\320\240\321\203\321\201\321\201\320\272\320\270\320\271_____\304\233_\303\241\303\261.._testdir_path2_\346\270\254\350\251\246_\320\240\321\203\321\201\321\201\320\272\320\270\320\271_____\304\233_\303\241\303\261.copy1to2.que
mkdir
/tmp/pristine-tar.nK854kS9zf/workdir/rclone-1.60.0/cmd/bisync/testdata/test_extended_char_paths/golden/normal-sync._testdir_path1_\346\270\254\350\251\246_\320\240\321\203\321\201\321\201\320\272\320\270\320\271_____\304\233_\303\241\303\261.._testdir_path2_\346\270\254\350\251\246_\320\240\321\203\321\201\321\201\320\272\320\270\320\271_____\304\233_\303\241\303\261.copy1to2.que:
Matthew
I've reassigned this bug from git-buildpackage to pristine-tar; as
Sergio says, it's pristine-tar that's failing.
I've pushed this to important, since it is preventing rclone packaging
from proceeding (without bodges at least); might be RC? I think the
problem relates to pristine-tar's handling of non-ascii filenames.
I used gbp import-orig --verbose --uscan --pristine-tar to tell me what
args it was passing to pristine-tar, and then re-ran that command adding
-v -d to get more debug output (at the bottom of this mail)
The complained-of file is in the upstream tarball:
cmd/bisync/testdata/test_extended_char_paths/golden/normal-sync._testdir_path1_測試_Русский_____ě_áñ.._testdir_path2_測試_Русский_____ě_áñ.copy1to2.que
...and in upstream git:
https://github.com/rclone/rclone/blob/v1.60.0/cmd/bisync/testdata/test_extended_char_paths/golden/normal-sync._testdir_path1_%E6%B8%AC%E8%A9%A6_%D0%A0%D1%83%D1%81%D1%81%D0%BA%D0%B8%D0%B9_____%C4%9B_%C3%A1%C3%B1.._testdir_path2_%E6%B8%AC%E8%A9%A6_%D0%A0%D1%83%D1%81%D1%81%D0%BA%D0%B8%D0%B9_____%C4%9B_%C3%A1%C3%B1.copy1to2.que
So I think the problem may be both in pristine-tar's decision that that
file is missing in the manifest, and then in whatever it's doing to make
a filename too long - which is a bit confusing, since if I do the
"mkdir" that it says is failing, it works just fine...
matthew@tsk:~/packages/rclone$ pristine-tar -v -d commit
../rclone_1.60.0.orig.tar.gz 28901851aa753bee063df1fff99ee90f097fb2e9
pristine-tar: git archive --format=tar
28901851aa753bee063df1fff99ee90f097fb2e9 | (cd
'/tmp/pristine-tar.QSprqRxiCj' && tar x)
pristine-tar: set subdir to rclone-1.60.0
pristine-tar: subdir is rclone-1.60.0
pristine-tar: mkdir /tmp/pristine-tar.nK854kS9zf/workdir
pristine-tar: mv /tmp/pristine-tar.QSprqRxiCj
/tmp/pristine-tar.nK854kS9zf/workdir/rclone-1.60.0
pristine-tar:
rclone-1.60.0/cmd/bisync/testdata/test_extended_char_paths/golden/normal-sync._testdir_path1_\346\270\254\350\251\246_\320\240\321\203\321\201\321\201\320\272\320\270\320\271_____\304\233_\303\241\303\261.._testdir_path2_\346\270\254\350\251\246_\320\240\321\203\321\201\321\201\320\272\320\270\320\271_____\304\233_\303\241\303\261.copy1to2.que
is listed in the manifest but may not be present in the source directory
pristine-tar: creating missing
rclone-1.60.0/cmd/bisync/testdata/test_extended_char_paths/golden/normal-sync._testdir_path1_\346\270\254\350\251\246_\320\240\321\203\321\201\321\201\320\272\320\270\320\271_____\304\233_\303\241\303\261.._testdir_path2_\346\270\254\350\251\246_\320\240\321\203\321\201\321\201\320\272\320\270\320\271_____\304\233_\303\241\303\261.copy1to2.que
mkdir
/tmp/pristine-tar.nK854kS9zf/workdir/rclone-1.60.0/cmd/bisync/testdata/test_extended_char_paths/golden/normal-sync._testdir_path1_\346\270\254\350\251\246_\320\240\321\203\321\201\321\201\320\272\320\270\320\271_____\304\233_\303\241\303\261.._testdir_path2_\346\270\254\350\251\246_\320\240\321\203\321\201\321\201\320\272\320\270\320\271_____\304\233_\303\241\303\261.copy1to2.que:
File name too long at /usr/bin/pristine-tar line 451.
Thanks,
Matthew
Matthew Vernon
Oct 31, 2022, 11:40:04 AM10/31/22
to
Hi,
Looking at this further.
The offending file is
cmd/bisync/testdata/test_extended_char_paths/golden/normal-sync._testdir_path1_測試_Русский_____ě_áñ.._testdir_path2_測試_Русский_____ě_áñ.copy1to2.que
In generating the manifest, pristine-tar calls:
LANG='C' tar --quoting-style=escape -tf rclone-1.60.0.tar.gz
which renders it thus:
rclone-1.60.0/cmd/bisync/testdata/test_extended_char_paths/golden/normal-sync._testdir_path1_\346\270\254\350\251\246_\320\240\321\203\321\201\321\201\320\272\320\270\320\271_____\304\233_\303\241\303\261.._testdir_path2_\346\270\254\350\251\246_\320\240\321\203\321\201\321\201\320\272\320\270\320\271_____\304\233_\303\241\303\261.copy1to2.que
That appears in manifest as:
rclone-1.60.0/cmd/bisync/testdata/test_extended_char_paths/golden/normal-sync._testdir_path1_\346\270\254\350\251\246_\320\240\321\203\321\201\321\201\320\272\320\270\320\271_____\304\233_\303\241\303\261.._testdir_path2_\346\270\254\350\251\246_\320\240\321\203\321\201\321\201\320\272\320\270\320\271_____\304\233_\303\241\303\261.copy1to2.que
pristine-tar then prints that out in its warning message:
pristine-tar:
rclone-1.60.0/cmd/bisync/testdata/test_extended_char_paths/golden/normal-sync._testdir_path1_\346\270\254\350\251\246_\320\240\321\203\321\201\321\201\320\272\320\270\320\271_____\304\233_\303\241\303\261.._testdir_path2_\346\270\254\350\251\246_\320\240\321\203\321\201\321\201\320\272\320\270\320\271_____\304\233_\303\241\303\261.copy1to2.que
is listed in the manifest but may not be present in the source directory
...but if I strace it, it's looking at something different - like it's
interpreting the \s in the manifest as literal \s rather than escape
characters...
stat("/tmp/pristine-tar.lndiNRBXMu/workdir/rclone-1.60.0/cmd/bisync/testdata/test_extended_char_paths/golden/normal-sync._testdir_path1_\\346\\270\\254\\350\\251\\246_\\320\\240\\321\\203\\321\\201\\321\\201\\320\\272\\320\\270\\320\\271_____\\304\\233_\\303\\241\\303\\261.._testdir_path2_\\346\\270\\254\\350\\251\\246_\\320\\240\\321\\203\\321\\201\\321\\201\\320\\272\\320\\270\\320\\271_____\\304\\233_\\303\\241\\303\\261.copy1to2.que",
0x55a1149ee4b8) = -1 ENAMETOOLONG (File name too long)
and indeed it tries to create something similar:
mkdir("/tmp/pristine-tar.lndiNRBXMu/workdir/rclone-1.60.0/cmd/bisync/testdata/test_extended_char_paths/golden/normal-sync._testdir_path1_\\346\\270\\254\\350\\251\\246_\\320\\240\\321\\203\\321\\201\\321\\201\\320\\272\\320\\270\\320\\271_____\\304\\233_\\303\\241\\303\\261.._testdir_path2_\\346\\270\\254\\350\\251\\246_\\320\\240\\321\\203\\321\\201\\321\\201\\320\\272\\320\\270\\320\\271_____\\304\\233_\\303\\241\\303\\261.copy1to2.que",
0777) = -1 ENAMETOOLONG (File name too long)
HTH,
Matthew
Looking at this further.
The offending file is
cmd/bisync/testdata/test_extended_char_paths/golden/normal-sync._testdir_path1_測試_Русский_____ě_áñ.._testdir_path2_測試_Русский_____ě_áñ.copy1to2.que
In generating the manifest, pristine-tar calls:
LANG='C' tar --quoting-style=escape -tf rclone-1.60.0.tar.gz
which renders it thus:
rclone-1.60.0/cmd/bisync/testdata/test_extended_char_paths/golden/normal-sync._testdir_path1_\346\270\254\350\251\246_\320\240\321\203\321\201\321\201\320\272\320\270\320\271_____\304\233_\303\241\303\261.._testdir_path2_\346\270\254\350\251\246_\320\240\321\203\321\201\321\201\320\272\320\270\320\271_____\304\233_\303\241\303\261.copy1to2.que
That appears in manifest as:
rclone-1.60.0/cmd/bisync/testdata/test_extended_char_paths/golden/normal-sync._testdir_path1_\346\270\254\350\251\246_\320\240\321\203\321\201\321\201\320\272\320\270\320\271_____\304\233_\303\241\303\261.._testdir_path2_\346\270\254\350\251\246_\320\240\321\203\321\201\321\201\320\272\320\270\320\271_____\304\233_\303\241\303\261.copy1to2.que
pristine-tar then prints that out in its warning message:
pristine-tar:
rclone-1.60.0/cmd/bisync/testdata/test_extended_char_paths/golden/normal-sync._testdir_path1_\346\270\254\350\251\246_\320\240\321\203\321\201\321\201\320\272\320\270\320\271_____\304\233_\303\241\303\261.._testdir_path2_\346\270\254\350\251\246_\320\240\321\203\321\201\321\201\320\272\320\270\320\271_____\304\233_\303\241\303\261.copy1to2.que
is listed in the manifest but may not be present in the source directory
interpreting the \s in the manifest as literal \s rather than escape
characters...
stat("/tmp/pristine-tar.lndiNRBXMu/workdir/rclone-1.60.0/cmd/bisync/testdata/test_extended_char_paths/golden/normal-sync._testdir_path1_\\346\\270\\254\\350\\251\\246_\\320\\240\\321\\203\\321\\201\\321\\201\\320\\272\\320\\270\\320\\271_____\\304\\233_\\303\\241\\303\\261.._testdir_path2_\\346\\270\\254\\350\\251\\246_\\320\\240\\321\\203\\321\\201\\321\\201\\320\\272\\320\\270\\320\\271_____\\304\\233_\\303\\241\\303\\261.copy1to2.que",
0x55a1149ee4b8) = -1 ENAMETOOLONG (File name too long)
and indeed it tries to create something similar:
mkdir("/tmp/pristine-tar.lndiNRBXMu/workdir/rclone-1.60.0/cmd/bisync/testdata/test_extended_char_paths/golden/normal-sync._testdir_path1_\\346\\270\\254\\350\\251\\246_\\320\\240\\321\\203\\321\\201\\321\\201\\320\\272\\320\\270\\320\\271_____\\304\\233_\\303\\241\\303\\261.._testdir_path2_\\346\\270\\254\\350\\251\\246_\\320\\240\\321\\203\\321\\201\\321\\201\\320\\272\\320\\270\\320\\271_____\\304\\233_\\303\\241\\303\\261.copy1to2.que",
0777) = -1 ENAMETOOLONG (File name too long)
HTH,
Matthew
Matthew Vernon
Nov 3, 2022, 6:40:03 AM11/3/22
to
Hi,
Turns out that change in tar is probably where this all started to go wrong.
[in pre-stretch, I think manifests were quoted and there were no files
starting -]
In stretch (pristine-tar 1.38; tar 1.29), manifests were quoted, and
extraction was done with tar --verbatim-files-from (that change was a
fix for #851286). This worked, because --verbatim-files-from unquoted
for you, and so in stretch there was a tar -t output that could be fed
back to tar reliably (via tar --verbatim-files-from)
Then tar made this change between 1.29 and 1.30:
"
2017-11-09 Sergey Poznyakoff <gr...@gnu.org.ua>
Fix --verbatim-files-from
* src/names.c (read_next_name): Don't unquote name read from the
file, if --verbatim-files-from option is in effect.
(names_options): improve description of --verbatim-files-from
* tests/T-null2.at: Test the change.
"
Unfortunately, this meant there was no longer a round-trip possible from
any tar -t output to tar -T (as I noted previously in this bug report).
The tar authors presumably didn't notice this, but it was a big breaking
change.
This fallout was reported as #902115 against pristine-tar 1.44 - the
problem was in fact that tar --verbatim-files-from was no longer
unquoting the manifest; alas we missed this fact at the time and
pristine-tar 1.46 (buster) introduced the erroneous fix and started to
store unquoted manifests (and also unquote the contents of those
manifests a second time in some cases, which is definitely wrong).
Separately, the reporting and fixing of 933031 again missed the
underlying problem, and instead essentially went "if extraction with
--verbatim-files-from fails, try without that option as well"; while git
bisect picked up the fix to #851286 as the cause of the bug, this was
really incorrect - it was the underlying change in tar's behaviour that
was at fault. That went into pristine-tar 1.49 (bullseye)
[I'm not sure what sort of backwards-compatibility guarantees
pristine-tar makes?]
Regards,
Matthew
Turns out that change in tar is probably where this all started to go wrong.
[in pre-stretch, I think manifests were quoted and there were no files
starting -]
In stretch (pristine-tar 1.38; tar 1.29), manifests were quoted, and
extraction was done with tar --verbatim-files-from (that change was a
fix for #851286). This worked, because --verbatim-files-from unquoted
for you, and so in stretch there was a tar -t output that could be fed
back to tar reliably (via tar --verbatim-files-from)
Then tar made this change between 1.29 and 1.30:
"
2017-11-09 Sergey Poznyakoff <gr...@gnu.org.ua>
Fix --verbatim-files-from
* src/names.c (read_next_name): Don't unquote name read from the
file, if --verbatim-files-from option is in effect.
(names_options): improve description of --verbatim-files-from
* tests/T-null2.at: Test the change.
"
Unfortunately, this meant there was no longer a round-trip possible from
any tar -t output to tar -T (as I noted previously in this bug report).
The tar authors presumably didn't notice this, but it was a big breaking
change.
This fallout was reported as #902115 against pristine-tar 1.44 - the
problem was in fact that tar --verbatim-files-from was no longer
unquoting the manifest; alas we missed this fact at the time and
pristine-tar 1.46 (buster) introduced the erroneous fix and started to
store unquoted manifests (and also unquote the contents of those
manifests a second time in some cases, which is definitely wrong).
Separately, the reporting and fixing of 933031 again missed the
underlying problem, and instead essentially went "if extraction with
--verbatim-files-from fails, try without that option as well"; while git
bisect picked up the fix to #851286 as the cause of the bug, this was
really incorrect - it was the underlying change in tar's behaviour that
was at fault. That went into pristine-tar 1.49 (bullseye)
[I'm not sure what sort of backwards-compatibility guarantees
pristine-tar makes?]
Regards,
Matthew
Matthew Vernon
Nov 4, 2022, 9:00:04 AM11/4/22
to
tags 1010024 +patch
quit
Hi,
I've made an MR (
https://salsa.debian.org/debian/pristine-tar/-/merge_requests/9 ) that
fixes this issue, and I think puts us in a stable place for manifest
handling, specifically:
* manifests (go back to) contain quoted paths, newline-separated. This
is backwards-compatible (we've basically always handled manifests as
containing quoted paths even when they didn't), and also unambiguous.
* we make a null-separated manifest to pass to tar --null (this is the
only way of unambiguously providing a file list to tar)
* if unquote-and-null-separate doesn't work on a manifest, we fall back
to null-separate-only, which will catch that subset of unquoted
manifests where a further unquote isn't a no-op
* incorporate Kevin Locke's fixes to make unquoting work reliably
This passes the regression testing in salsa CI. Are you happy for my
Debian hat (mat...@debian.org) to upload this?
[I'll do a delayed upload next week if I don't hear back from you]
Thanks,
Matthew
quit
Hi,
I've made an MR (
https://salsa.debian.org/debian/pristine-tar/-/merge_requests/9 ) that
fixes this issue, and I think puts us in a stable place for manifest
handling, specifically:
* manifests (go back to) contain quoted paths, newline-separated. This
is backwards-compatible (we've basically always handled manifests as
containing quoted paths even when they didn't), and also unambiguous.
* we make a null-separated manifest to pass to tar --null (this is the
only way of unambiguously providing a file list to tar)
* if unquote-and-null-separate doesn't work on a manifest, we fall back
to null-separate-only, which will catch that subset of unquoted
manifests where a further unquote isn't a no-op
* incorporate Kevin Locke's fixes to make unquoting work reliably
This passes the regression testing in salsa CI. Are you happy for my
Debian hat (mat...@debian.org) to upload this?
[I'll do a delayed upload next week if I don't hear back from you]
Thanks,
Matthew
0 new messages