zip files created with archive/zip aren't recognised as zip files by java.util.zip

[JohnGB]

I've written some code that creates a .zip file from a number of other files (.png in case it matters).  The zip file seems to be fine, and I can open it on both a mac and windows computer without a problem.  However when an Android device (using the java.util.zip package) tries to open the file, it throws an error that the file is not a .zip file.  Given how long java.util.zip has been in use, I doubt that the problem is there, and assume the issue is with the method that I've used to create the .zip file in the first place.  

Below (or in play) is the function that I'm using to zip the files:

func createZip(zipDir, zipName string, filesToZip []string) error {

// creat the file to zip the contents into

newfile, err := os.Create(zipDir + zipName)

if err != nil {

return err

}

defer newfile.Close()

zipit := zip.NewWriter(newfile)

defer zipit.Close()

for _, filename := range filesToZip {

zipFile, err := os.Open(zipDir + filename)

if err != nil {

return err

}

defer zipFile.Close()

// get the file information

info, err := zipFile.Stat()

if err != nil {

return err

}

// create a zip header from the os.FileInfo

header, err := zip.FileInfoHeader(info)

if err != nil {

return err

}

// write the header to the zip file

writer, err := zipit.CreateHeader(header)

if err != nil {

return err

}

// copy the file to the zip file

_, err = io.Copy(writer, zipFile)

if err != nil {

return err

}

}

return nil

}

I would appreciate feedback on my code, or suggestions as to where I can look for the cause.

[Tamás Gulácsi]

Please check the returned error when closing a file opened for writing.

What about ZIP64? My blind shot is that you create a 64-bit archive, and java can't cope with it.

[JohnGB]

Thanks for the suggestion Tamás,

I'm checking for returned errors now when closing, and I'm not getting any.

It's very likely that your thought that it's related to being a 64 bit archive is correct, but I can't seem to find a way of creating a 32 bit archive in Go.  Any ideas how this can be done?

~ John

[Tamás Gulácsi]

The source suggests that it creates zip64 off it can't fit in zip32 (compressed or uncompressed size or count is greater than 4GiB).
You can check it if you read the zip: "For files requiring the ZIP64 format the 32 bit fields will be 0xffffffff and the 64 bit fields must be used instead." as the doc says.

[JohnGB]

Okay, then it isn't likely a ZIP64 issue as the zip file in question is less than 1MB.  I read that in the documentation, but it only specifies what is done for files requiring ZIP64, but nothing about 32.  I didn't realise that the files are 32 bit unless they are too big.

I've created some other files using compress/gzip that seem to work, so I'll look into using a tar.gz format instead.

[Jakob Borg]

You're not flushing the writer returned by CreateHeader. See if that makes a difference?

//jb

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tong Sun]

Hi John, it's been a while. Have you solved the problem yet? I'm interested in creating zip files as well. Could you post your final working version please? Thanks!

[Alex Skinner]

I took a look at this tonight mainly to learn about the zip format spec since I've never bothered.  I used your Go code to create a zip, then tried to extract with java on linux as I don't have time to fiddle with testing on Android at the moment.  Here is what I found if it's relevant - and it's very late, so expect errors below - 

(I used this page as spec, if that matters - https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT) .  

Using a ZipInputStream, java failed to parse the file with error 'only DEFLATED entries can have EXT descriptor'.   Further checking shows that Go seems to consistently set bitflag 3(08) in byte 7.  The spec says "Note: PKZIP version 2.04g for DOS only recognizes this bit for method 8 compression, newer versions of PKZIP recognize this bit for any compression method." This seems to imply to me that Java is behaving as described in the above, because the Go version is in fact doing what bitflag 3 implies, which is putting the data descriptor after the data, but not using compression method 8. I honestly don't know what benefit this serves, if any, though.

The only way I could get java to successfully expand the Go zip was to -

1 - Change byte 7 to '00' for each file

2 - Move data in data descriptor into their fields in the file header

3 - Remove the descriptor signature and descriptor

4 - Change offset at end of file

Or, take your Go created zip file(test.zip) and run 'zip -F test.zip --out fixed.zip' which does all this in my tests. If that resulting file then works, that may be the reason. If that is in fact the problem, it should be straightforward to make a local copy of archive/zip and change this behavior I think.

Curious to know what you find,

Alex

[minux]

On Sat, May 16, 2015 at 3:06 AM, Alex Skinner <al...@lx.lc> wrote:

I took a look at this tonight mainly to learn about the zip format spec since I've never bothered.  I used your Go code to create a zip, then tried to extract with java on linux as I don't have time to fiddle with testing on Android at the moment.  Here is what I found if it's relevant - and it's very late, so expect errors below - 

(I used this page as spec, if that matters - https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT) .  

Using a ZipInputStream, java failed to parse the file with error 'only DEFLATED entries can have EXT descriptor'.   Further checking shows that Go seems to consistently set bitflag 3(08) in byte 7.  The spec says "Note: PKZIP version 2.04g for DOS only recognizes this bit for method 8 compression, newer versions of PKZIP recognize this bit for any compression method." This seems to imply to me that Java is behaving as described in the above, because the Go version is in fact doing what bitflag 3 implies, which is putting the data descriptor after the data, but not using compression method 8. I honestly don't know what benefit this serves, if any, though.

Go archive/zip has to set that flag because it's a streaming interface.

Before actually compression each file, Go can't know even the uncompressed length of the file,

let alone CRC, so if it want to fill in correct value in the local header, Go has to buffer the whole

uncompressed file in memory, and then start compressing it.

[JohnGB]

Sorry for the late reply here.

I didn't manage to get the zip file working with java.utils.zip, but it seemed to work with every other zip application or library that I tried.  So I just used a different Java package to handle the zip files, and that solved the problem.

[ndi...@apigee.com]

Hi could you share how you actually managed to do all of this? I am dealing with this same issue and really need help moving the data descriptor stuff to the file header. Thank you.

[John Beckett]

The code that I wrote to handle this is:

func writeZip(data []byte, filename string) error {

var fileGZ bytes.Buffer

zipper := gzip.NewWriter(&fileGZ)

_, err := zipper.Write(data)

if err != nil {

return err

}

// call the close explicitly

if err = zipper.Close(); err != nil {

return err

}

err = ioutil.WriteFile(filename, fileGZ.Bytes(), 0644)

if err != nil {

return err

}

return nil

}

--
You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/0iae5Ng-I-0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-nuts+unsubscribe@googlegroups.com.

[thebroke...@gmail.com]

I didn't look into this deeply, but if there's something strange that Go's implementation of zip is doing, feel free to file an issue. It's too late for anything to happen in Go 1.8, but I can investigate it deeper in Go 1.9.

JT

To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.

[ndi...@apigee.com]

My coworker made a wrapper for writing zip files that are readable by java.util.ZipInputStream, according to the reply given by Alex Skinner. The pkg is on GitHub here.