[go] archive/zip: fix Zip64 edge cases

0 views

Skip to first unread message

Gopher Robot (Gerrit)

unread,

2:33 PM (3 hours ago) 2:33 PM

to Filippo Valsorda, goph...@pubsubhelper.golang.org, golang-...@googlegroups.com, Russ Cox, golang...@luci-project-accounts.iam.gserviceaccount.com, Dmitri Shuralyov, Dmitri Shuralyov, SLSA Policy Verification Service, K. York, Joseph Tsai, golang-co...@googlegroups.com

Gopher Robot submitted the change

Change information

Commit message:

archive/zip: fix writer-side Zip64 edge cases

I ran into this because the broken Writer caused mysterious and very
hard to debug failures uploading archive/zip-generated files to the
Internet Archive. (Only zip files bigger than 4GiB *and* smaller than
around 6.5GiB failed. I still don't have an explanation for the latter
part, maybe the parser has different logic for when the count of records
crosses 65535 and the Zip64 EOCD is used.)

Reproducing testdata/zip64/*.zsparse:

Inputs (sparse zero files via `truncate -s N NAME`, sizes in bytes):
  big5g.bin     5<<30                   big4g.bin     4<<30
  big4g-1.bin   (4<<30) - 1             big4g-2.bin   (4<<30) - 2
  under4g.bin   (4<<30) - 59            first         (4<<30) - 36
  small.bin     42  (use `dd` for the non-sparse 42-byte file)

Cases (case → entries, M=0 Store, M=9 Deflate):
  store-5g             big5g.bin/0
  deflate-zeros-5g     big5g.bin/9
  store-4g-minus-1     big4g-1.bin/0
  store-4g-minus-2     big4g-2.bin/0
  store-just-under-4g  under4g.bin/0
  store-exact-4g       big4g.bin/0
  offset-past-4g       big5g.bin/0, small.bin/0
  offset-eq-4g         first/0,     small.bin/0

Producers:
  infozip-*    Info-ZIP 3.0:
                 zip -q -X -M OUT.zip <entries>
  libarchive-* bsdtar (libarchive):
                 bsdtar -cf OUT.zip --format zip \
                   --options zip:compression={store|deflate} <entries>
  go126-*      archive/zip from Go 1.26. Build with GOTOOLCHAIN=go1.26.0
                 from a tempdir whose go.mod declares `go 1.26.0`.
                 For each entry:
                 zip.FileHeader{Name, Method: zip.Store|zip.Deflate},
                 CreateHeader, io.CopyN(fw, zeros, size), w.Close().

Convert each OUT.zip to ${producer}-${case}.zsparse using the format
defined in archive/zip/zip64_sparse_test.go (scanSparse / readSparse):
walk the zip in 4 KiB chunks, drop chunks that are entirely zero,
coalesce adjacent non-zero chunks into spans, and serialize the result
as gzip of:

  uint64 LE  totalSize
  uint32 LE  numSpans
  numSpans times:
    uint64 LE  offset
    uint32 LE  dataLen
    dataLen bytes

Updates #22520
Fixes #23572
Fixes #33116
Fixes #69415

Change-Id: I6e24e7170094346af494da153c63e6b56a6a6964

Reviewed-on: https://go-review.googlesource.com/c/go/+/725161

Auto-Submit: Filippo Valsorda <fil...@golang.org>

LUCI-TryBot-Result: golang...@luci-project-accounts.iam.gserviceaccount.com <golang...@luci-project-accounts.iam.gserviceaccount.com>

Reviewed-by: Dmitri Shuralyov <dmit...@google.com>

SLSA-Policy-Verified: SLSA Policy Verification Service <devtools-gerritco...@google.com>

Reviewed-by: Russ Cox <r...@golang.org>

Files:

M src/archive/zip/struct.go
A src/archive/zip/testdata/zip64/go126-deflate-zeros-5g.zsparse
A src/archive/zip/testdata/zip64/go126-offset-eq-4g.zsparse
A src/archive/zip/testdata/zip64/go126-offset-past-4g.zsparse
A src/archive/zip/testdata/zip64/go126-store-4g-minus-1.zsparse
A src/archive/zip/testdata/zip64/go126-store-4g-minus-2.zsparse
A src/archive/zip/testdata/zip64/go126-store-5g.zsparse
A src/archive/zip/testdata/zip64/go126-store-exact-4g.zsparse
A src/archive/zip/testdata/zip64/go126-store-just-under-4g.zsparse
A src/archive/zip/testdata/zip64/infozip-deflate-zeros-5g.zsparse
A src/archive/zip/testdata/zip64/infozip-offset-eq-4g.zsparse
A src/archive/zip/testdata/zip64/infozip-offset-past-4g.zsparse
A src/archive/zip/testdata/zip64/infozip-store-4g-minus-1.zsparse
A src/archive/zip/testdata/zip64/infozip-store-4g-minus-2.zsparse
A src/archive/zip/testdata/zip64/infozip-store-5g.zsparse
A src/archive/zip/testdata/zip64/infozip-store-exact-4g.zsparse
A src/archive/zip/testdata/zip64/infozip-store-just-under-4g.zsparse
A src/archive/zip/testdata/zip64/libarchive-deflate-zeros-5g.zsparse
A src/archive/zip/testdata/zip64/libarchive-offset-eq-4g.zsparse
A src/archive/zip/testdata/zip64/libarchive-offset-past-4g.zsparse
A src/archive/zip/testdata/zip64/libarchive-store-4g-minus-1.zsparse
A src/archive/zip/testdata/zip64/libarchive-store-4g-minus-2.zsparse
A src/archive/zip/testdata/zip64/libarchive-store-5g.zsparse
A src/archive/zip/testdata/zip64/libarchive-store-exact-4g.zsparse
A src/archive/zip/testdata/zip64/libarchive-store-just-under-4g.zsparse
M src/archive/zip/writer.go
A src/archive/zip/zip64_sparse_test.go
A src/archive/zip/zip64_test.go

Change size: XL

Delta: 28 files changed, 975 insertions(+), 66 deletions(-)

Branch: refs/heads/master

Submit Requirements:

Code-Review: +2 by Russ Cox, +1 by Dmitri Shuralyov
TryBots-Pass: LUCI-TryBot-Result+1 by golang...@luci-project-accounts.iam.gserviceaccount.com

Open in Gerrit

Inspect html for hidden footers to help with email filtering. To unsubscribe visit settings.

Gerrit

open

diffy

satisfied_requirement

Reply all

Reply to author

Forward

0 new messages