tar/libarchive gzip problem [was: portsnap corrupted]
Simon L. Nielsen
I think one of the recent changes to tar or libarchive broke gzip
handling in some cases. See more below.
[portsnap extract fails with gzip error]
I'm not sure why I didn't run into it in my tests, but I think the
problem is in tar / libarchive's handling of gzip files. Taking one
random "broken" file [1] it fails with tar's build in decompression,
but works using external zcat.
[1] http://portsnap1.freebsd.org/f/19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d.gz
http://people.freebsd.org/~simon/tmp/19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d.gz
Broken system:
[simon@eddie:/tmp] tar --version
bsdtar 2.5.903a - libarchive 2.5.903a
[simon@eddie:/tmp] uname -a
FreeBSD eddie.nitro.dk 8.0-CURRENT FreeBSD 8.0-CURRENT #1: Tue Dec 30 22:28:33 CET 2008 si...@eddie.nitro.dk:/FreeBSD/obj/FreeBSD/system-CURRENT/sys/EDDIE i386
[simon@eddie:/tmp] tar tvf 19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d.gz > /dev/null
tar: Error opening archive: Invalid GZip header (saw 99 at offset 1)
[simon@eddie:/tmp] zcat 19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d.gz | tar tf - > /dev/null
[simon@eddie:/tmp] zcat 19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d.gz | sha256
19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d
OK system:
[simon@benji:/tmp] tar --version
bsdtar 2.5.5 - libarchive 2.5.5
[simon@benji:/tmp] uname -a
FreeBSD benji.s 7.1-RC2 FreeBSD 7.1-RC2 #0: Tue Dec 23 15:18:30 UTC 2008 ro...@logan.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC i386
[simon@benji:/tmp] tar tvf 19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d.gz > /dev/null
[simon@benji:/tmp] zcat 19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d.gz | sha256
19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d
Another OK system:
[simon@ref8-amd64:files] tar --version
bsdtar 2.5.5 - libarchive 2.5.5
[simon@ref8-amd64:files] uname -a
FreeBSD ref8-amd64.freebsd.org 8.0-CURRENT FreeBSD 8.0-CURRENT #2 r184542:185402: Fri Nov 28 19:14:40 UTC 2008 pe...@ref8-amd64.freebsd.org:/scratch/src/sys/amd64/compile/REF8-AMD64 amd64
[simon@ref8-amd64:files] tar tvf 19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d.gz > /dev/null
[simon@ref8-amd64:files] zcat 19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d.gz | sha256
19faa3b8bd15bb8f4cd9f665a7623887729f3bd834d780e8b069df979f228e8d
--
Simon L. Nielsen
_______________________________________________
freebsd...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-curre...@freebsd.org"
Tim Kientzle
I have in fact been refactoring the decompression
support recently. I've been testing pretty carefully
as I go, but it's certainly possible that I broke
something. I'll take a careful look over the weekend.
And, of course, test even more before I commit the
next round of refactoring, which is almost done. ;-/
Tim
Tim Kientzle
--------------010107030804000001000003
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit
I think I know what this is. I tried recently to add
support for concatenated gzip files. This involves looking
ahead for another GZip header after the end of the compressed
data. The current version screws this up and ends up trying
to match the file CRC as a Gzip header. About 1 in 256 files
will match the first byte, which triggers the subsequent meltdown.
This also explains why neither of us saw it in testing.
(I knew the code didn't actually work for concatenated gzip
files but didn't realize it would break decode of some
regular non-concatenated files.)
The attached patch simply disables this additional
header check, which should fix the immediate problem.
Please try it and let me know.
Apologies,
Tim
--------------010107030804000001000003
Content-Type: text/x-patch;
name="gzip_decompression.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="gzip_decompression.patch"
Index: archive_read_support_compression_gzip.c
===================================================================
--- archive_read_support_compression_gzip.c (revision 185679)
+++ archive_read_support_compression_gzip.c (working copy)
@@ -428,8 +428,7 @@
"Failed to clean up gzip decompressor");
return (ARCHIVE_FATAL);
}
- /* Restart header parser with the next block. */
- state->header_state = state->header_done = 0;
+ state->eof = 1;
/* FALL THROUGH */
case Z_OK: /* Decompressor made some progress. */
/* If we filled our buffer, update stats and return. */
--------------010107030804000001000003
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
freebsd...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-curre...@freebsd.org"
--------------010107030804000001000003--
Tim Kientzle
me know if you see any other problems.
Tim
> ------------------------------------------------------------------------
>
> Index: archive_read_support_compression_gzip.c
> ===================================================================
> --- archive_read_support_compression_gzip.c (revision 185679)
> +++ archive_read_support_compression_gzip.c (working copy)
> @@ -428,8 +428,7 @@
> "Failed to clean up gzip decompressor");
> return (ARCHIVE_FATAL);
> }
> - /* Restart header parser with the next block. */
> - state->header_state = state->header_done = 0;
> + state->eof = 1;
> /* FALL THROUGH */
> case Z_OK: /* Decompressor made some progress. */
> /* If we filled our buffer, update stats and return. */
Simon L. Nielsen
> I just committed a fix for this (r186670). Please let
> me know if you see any other problems.
This seems to fix it, thanks!
Max, could you see if you have portsnap problems with the latest
libarchive?
--
Simon L. Nielsen
Max Laier
> On 2008.12.31 18:31:17 -0800, Tim Kientzle wrote:
> > I just committed a fix for this (r186670). Please let
> > me know if you see any other problems.
>
> This seems to fix it, thanks!
>
> Max, could you see if you have portsnap problems with the latest
> libarchive?
Non at all! Thank you for the quick fix! Happy New Year!
--
/"\ Best regards, | mla...@freebsd.org
\ / Max Laier | ICQ #67774661
X http://pf4freebsd.love2party.net/ | mlaier@EFnet
/ \ ASCII Ribbon Campaign | Against HTML Mail and News