[groovy-user] Download zip file and process

[rakesh mailgroups]

Hi All,

struggling to get all the pieces to work together:

I would like to download a zip file over http, and process it in memory, NOT via a file.

More details, I am trying to download a geoIP database made available as a zipped csv file: http://geolite.maxmind.com/download/geoip/database/GeoIPCountryCSV.zip

I then need to process each line and insert into my db.

I'm finding it quite difficult with all the ZipFile/ZipInputStream options as well as not wanting to store it on the file system first.

Any help appreciated.

Rakesh

[Tim Yates]

This grabs a zip file, and processes each entry without saving the zip file to disk first.

Obviously in this example, it just unpacks the zip to /tmp.  You would want to do something else with the stream zs once you have got the first entry.

Hope it helps!

import java.util.zip.*

"http://some.url/to/afile.zip".toURL().withInputStream { s ->

  new ZipInputStream( s ).with { zs ->

    while( zent = zs.nextEntry ) {

      println "Processing $entry.name"

      

      def local = new File( "/tmp/$entry.name" )

      // Do stuff with the entry

      if( zent.isDirectory() ) {

        local.mkdir()

      }

      else {

        local << zs

      }

      

      zs.closeEntry()

    }

  }

}

[Andre Steingress]

Hi,

The first part is to fetch the file content with URL#bytes (assuming you are accessing a non-protected HTTP url). 

If you need an example on how to unzip a file in Groovy, you can have a look at Tim Yates Groovy Common Extensions project on Github:

https://github.com/timyates/groovy-common-extensions/blob/master/src/main/groovy/com/bloidonia/groovy/extensions/FileExtensionMethods.groovy

The code there creates java.io.File instances, but this could be easily replaced with a map/a single instance of ByteArrayOutputStream.

Cheers, André

[Andre Steingress]

I should have known that Tim Yates himself would answer this question ;-) Sorry for dbl posting the answer...

Cheers, André

[rakesh mailgroups]

Hi,

thanks but the issue seems to be the code that does NOT use a file. Any example code for that?

Thanks

Rakesh

[Tim Yates]

Not sure what you mean...  My example streams from a URLConnection

[rakesh mailgroups]

"The code there creates java.io.File instances, but this could be easily replaced with a map/a single instance of ByteArrayOutputStream."

That bit!

[Tim Yates]

So what do you want it to do?

[Andre Steingress]

You should end up with something like this (assuming the ZIP file has only a single entry):

final bytes = // ... the zip file's byte[] 

final zipInput = new ZipInputStream(new ByteArrayInputStream(bytes))

final output = new ByteArrayOutputStream()

     

zipInput.withStream {

    def entry = zipInput.nextEntry

    

    int len = 0;

    byte[] buffer = new byte[4096]

    while ((len = zipInput.read(buffer)) > 0){

        output.write(buffer, 0, len);

    } 

}

I am currently underway and don't have access to my dev environment, but this example should give you a glimpse on how to solve this issue.

Cheers, André

[Tim Yates]

Ok, for this example I am going to grab the zip for v0.3 of my Common Groovy Extension source on Github.

I will then go through each entry looking for 'ObjectExtensionMethods.groovy'[1], and when I find this I will put it into a StringWriter which I will convert to a String at the end of the processing.

Hope it helps.

import java.util.zip.*

String zipURL = "https://github.com/timyates/groovy-common-extensions/archive/v0.3.zip"

String fileContent = zipURL.toURL().withInputStream { s ->

  new ZipInputStream( s ).with { zs ->

    new StringWriter().with { sw ->

      while( zent = zs.nextEntry ) {

        if( entry.name.endsWith( 'ObjectExtensionMethods.groovy' ) ) {

          sw << zs

          break

        }

        zs.closeEntry()

      }

      sw.toString()

    }

  }

}

println fileContent

Tim

1. https://github.com/timyates/groovy-common-extensions/blob/master/src/main/groovy/com/bloidonia/groovy/extensions/ObjectExtensionMethods.groovy

[Tim Yates]

Or, as Andre says, you can copy it from the stream to a byte buffer

    zipInput.eachByte( 4096 ) { buffer, len ->

      output.write( buffer, 0, len )

    } 

Is a shortcut to wrapping your own buffering input stream reader :-)

Tim