[1.2.5] Stream data from ZipOutputStream in response

[Erin Drummond]

Hello,

I am trying to on-the-fly generate and stream a zip file. I have the following code in my controller:

    public static void downloadFileZip(String hash, String name) {
        if (StringUtils.isEmpty(hash)) {
            notFound();
        }
        File path = new File(Config.getTorrentsCompletePath(), hash);
        if (!path.exists()) {
            notFound();
        }
        if (name == null) {
            name = hash;
        }
        try {
            response.setContentTypeIfNotSet("application/zip");
            response.setHeader("Content-Disposition", "attachment; filename=\"" + Util.URLDecode(name) + ".zip\"");
            ZipOutputStream zip = new ZipOutputStream(response.out);
            zip.setLevel(0); //no compression, we just want it as fast as possible                       
            addDirectory(zip, path, "/");
            zip.close();
        } catch (IOException ex) {
            error("Unable to create zip file!");           
        }       
    }
   
    private static void addDirectory(ZipOutputStream zip, File directory, String prefix) throws IOException {
        for (File f : directory.listFiles()) {
            if (f.isDirectory()) {               
                addDirectory(zip, f, prefix + f.getName() + "/");               
                continue;
            }           
            zip.putNextEntry(new ZipEntry(prefix + f.getName()));
            StreamUtils.copy(new FileInputStream(f), zip);
            zip.closeEntry();
        }           
    }


This works fine for small files, but I tried it with a larger file (200mb) and it failed with an OutOfMemoryError: Jave Heap Space. I am aware of the CSV example in the documentation but I dont know how to apply it to an OutputStream instead of just some text.

I have tried calling 'zip.flush()' after every 'zip.closeEntry' and I also tried using a PipedInputStream/PipedOutputStream to pipe the ZipOutputStream directly into an InputStream, then calling renderBinary() on it but it doesnt work, it just hangs.

How can I stream the contents of the ZipOutputStream? Setting response.chunked=true just results in a NPE inside Play's response object

Cheers,
Erin

[Ivan San José García]

As you said, I think you need to use writeChunk(). Otherwise, Play! store all the response in memory before send it, and this is why you got an OOM exception.

2012/12/26 Erin Drummond <erin...@gmail.com>

--
 
 

[Erin Drummond]

Hi Ivan,

Thanks for your reply. What I am having trouble with, is what argument
do I pass writeChunk and where do I call it from in my code?

Cheers,
Erin

> --
>
>

[fehmica...@gmail.com]

Zip is not suitable for streaming. You may use gzip instead.

 

http://stackoverflow.com/questions/4797315/rails-on-the-fly-streaming-of-output-in-zip-format

http://stackoverflow.com/questions/6225488/java-stream-contents-of-zipfile-via-http

http://en.wikipedia.org/wiki/ZIP_(file_format)#Structure

 

 

Erin Drummond wrote:

--

 

 

 

[Erin Drummond]

Im not trying to unzip it on the fly, the user can wait until the
whole zip has downloaded before trying to open it, so "zip is not
suitable for streaming" doesnt apply here.

I am "streaming" it because the contents of the zip cannot fit in
memory, and so it must be streamed to the client. Its the same as any
other file, except I want it to be generated on the fly instead of
writing it to a file first and then streaming that file.

If the transfer is interrupted then obviously it has to start again
from the beginning, this is fine

> --
>
>

[Tom Carchrae]

On Wed, Dec 26, 2012 at 4:18 PM, Erin Drummond <erin...@gmail.com> wrote:
> Im not trying to unzip it on the fly, the user can wait until the
> whole zip has downloaded before trying to open it, so "zip is not
> suitable for streaming" doesnt apply here.
>

As mentioned, I'm not sure you can stream the zip format because of
how it writes the file. You can write to a temp file then send that -
this is what I have done with large zips.

Tom

[Erin Drummond]

Writing a temp file will take up too much space and introduce more
maintenance (ie, cleaning up temp files) which is why I am dynamically
generating the zip.

However, I have managed to get it working by subclassing
ByteArrayOutputStream and passing its internal buffer to
response.writeChunk() every 4kb. This works because it appears
writeChunk() was patched in 1.2.3 to write out a byte[] if supplied
instead of calling toString() on the object passed in. eg:

protected static class MyOutputStream extends ByteArrayOutputStream {

private Response r;

public MyOutputStream(Response r) {
super(4096);
this.r = r;
}

@Override
public synchronized void write(int b) {
super.write(b);
if (count >= 4096) {
flushBuffer();
}
}

@Override
public void close() throws IOException {
flushBuffer(); //get the last few bytes
}

private void flushBuffer() {
r.writeChunk(this.toByteArray());
reset();
}

}


and then using it like:

ZipOutputStream zip = new ZipOutputStream(new
MyOutputStream(response));

zip.setLevel(0); //no compression, we just want it as fast
as possible
addDirectory(zip, path, "/");
zip.close();

Thanks for your help, and I hope this helps someone.

> --
>
>

[Ivan San José García]

But with this solution you will eat up the thread pool for HTTP requests with few active connections. Did you increase the thread pool for HTTP connections? Or do you think put this streaming in a Job would be more appropiate?

2012/12/27 Erin Drummond <erin...@gmail.com>

--

[Erin Drummond]

You're right, I have wrapped it in a job and passed the controller
Response object into the job so I can call writeChunk() on it.
Hopefully this wont cause any issues, I guess ill find out when some
load gets put on the system

> --
>
>

[Julien L.]

Can you post your solution? It will interest a lot of people!

[Erin Drummond]

Sure, heres the abridged version. Basically, I subclass Job, override
doJob() to create the zip and call writeChunk(), and then instantiate
and await() the job in the action method.

I dont know if this has any side effects because I dont know the
internals of Play well enough - normally the request/response objects
arent available inside a job, so by passing in a reference to the
response object I dont know if it will tie up the response or cause
other problems with other requests (perhaps someone who knows about
the internals of Play could enlighten me)


public class Download extends Controller {

public static void downloadFileZip(String hash, String name) {
if (StringUtils.isEmpty(hash)) {
notFound();
}
File path = new File(Config.getTorrentsCompletePath(), hash);
if (!path.exists()) {
notFound();
}
if (name == null) {
name = hash;
}

await(new StreamZipJob(path, name, response).now());
}

protected static class StreamZipJob extends Job {

private File baseDirectory;
private String zipFileName;
private Response response;

public StreamZipJob(File baseDirectory, String zipFileName,
Response response) {
this.baseDirectory = baseDirectory;
this.zipFileName = zipFileName;
this.response = response;
}

@Override
public void doJob() {

try {
response.setContentTypeIfNotSet("application/zip");
response.setHeader("Content-Disposition", "attachment;

filename=\"" + Util.URLDecode(zipFileName) + ".zip\"");

ZipOutputStream zip = new ZipOutputStream(new
MyOutputStream(response));
zip.setLevel(0); //no compression, we just want it as
fast as possible

addDirectory(zip, baseDirectory, "/");

zip.close();
} catch (IOException ex) {
error("Unable to create zip file!");
}
}

private static void addDirectory(ZipOutputStream zip, File
directory, String prefix) throws IOException {
for (File f : directory.listFiles()) {
if (f.isDirectory()) {
addDirectory(zip, f, prefix + f.getName() + "/");
continue;
}
zip.putNextEntry(new ZipEntry(prefix + f.getName()));

StreamUtils.copy(new BufferedInputStream(new
FileInputStream(f)), zip);
zip.closeEntry();

> --
>
>

[Gaëtan Renaudeau]

2012/12/27 <fehmica...@gmail.com>

Zip is not suitable for streaming. You may use gzip instead.

 

http://stackoverflow.com/questions/4797315/rails-on-the-fly-streaming-of-output-in-zip-format

http://stackoverflow.com/questions/6225488/java-stream-contents-of-zipfile-via-http

http://en.wikipedia.org/wiki/ZIP_(file_format)#Structure

Well, sorry to disagree, but we've made Zip Streaming working fine on Play 2,

http://blog.greweb.fr/2012/11/play-framework-enumerator-outputstream/

BTW, you can create a zip with a OutputStream so what is the reason you can't use it for streaming?

I don't know about Play 1.x, but it is probably possible.

--
 
 

--
Gaëtan Renaudeau, greweb.fr

[Erin Drummond]

Hi Gaetan,

I see by your blog that you're a Play developer. Is my solution for
Play 1.x going to cause any unexpected issues?

> --
>
>

[Fehmi Can Saglam]

28 Ara 2012 16:58 tarihinde "Gaëtan Renaudeau" <renaudea...@gmail.com> yazdı:

>
>
>
>
> 2012/12/27 <fehmica...@gmail.com>
>>
>> Zip is not suitable for streaming. You may use gzip instead.
>>
>>  
>>
>> http://stackoverflow.com/questions/4797315/rails-on-the-fly-streaming-of-output-in-zip-format
>>
>> http://stackoverflow.com/questions/6225488/java-stream-contents-of-zipfile-via-http http://en.wikipedia.org/wiki/ZIP_(file_format)#Structure
>
>
>
> Well, sorry to disagree, but we've made Zip Streaming working fine on Play 2, http://blog.greweb.fr/2012/11/play-framework-enumerator-outputstream/
> BTW, you can create a zip with a OutputStream so what is the reason you can't use it for streaming?

As explained in the links I have given, whole zip file must be created and stored in the memory before streaming -when using OutputStream-. Yes you can stream but you cannot create and stream large zip files practically.

> --
>  
>  

[Fehmi Can Saglam]

If Erin's method is working there should be smth wrong with the docs. Does anybody have a clear idea about zip file format and its variations?

29 Ara 2012 00:14 tarihinde "Fehmi Can Saglam" <fehmica...@gmail.com> yazdı:

[Erin Drummond]

Exactly, "zip cannot be streamed" is simply not true because I just
did it. I know that it wasnt being buffered completely in memory
because I had a problem with OOM errors until I fixed my
implementation, and the resulting zip files worked fine.

I think its implementation-specific whether or not zip can be
streamed. I noticed one of the links was talking about Rails - the zip
implementation they were using had to jump between the middle and the
end of the file everytime it wrote a zip entry because the metadata
was stored at the end of the file. This meant that it couldnt be
streamed.

However, from what I can tell, I dont think that's possible using a
OutputStream in java because its a constant stream of bits and you
cant jump around in it. So I believe the Java zip implementation
either calculates and stores the metadata at the start of the file
before writing the zip entries (the zip specification allows this
apparently) or calculates everything it needs as it goes so it can
write the metadata at the end without looking at other parts of the
file.

I could be completely wrong of course as I dont have intimate
knowledge of the Java streams system/zip implementation.

Also, I should probably note that I am using zip simply as a container
for many files - no compression is being used. Some of the links
suggested that may affect its streamability.

> --
>
>