serving a zip file

[Matt Broadstone]

Hi,
In our project, we are trying to provide the ability to download a
dataset in a zipped format. We've been transitioning old code to use
web2py, and while I've been able to serve files using web2py elsewhere
(images, for instance), I can't quite figure out how to serve this
particular file. Below is the code I've come up with. When I take out
the code to zip the file, it serves perfectly. Does anyone have
experience with this?

filename = "%s-backup" % (time.strftime("%Y%m%d-%H%M"))
raw_data = <a bunch of data>
stream = cStringIO.StringIO()
zip_file = zipfile.ZipFile(stream, "w", zipfile.ZIP_DEFLATED, False)
zip_file.writestr(filename, raw_data)
response.headers['Content-Type'] = "application/octet-stream"
response.headers['Content-Disposition'] = "attachment;
filename=%s%s.bin" % (time.strftime("%Y%m%d-%H%M"), filename)
return response.stream(stream, request=request)

Thanks,
Matt

[pbreit]

My first suggestion would be to save the files to disk and serve statically if possible.

[Matt Broadstone]

On Thu, Oct 13, 2011 at 11:34 AM, pbreit <pbreit...@gmail.com> wrote:
> My first suggestion would be to save the files to disk and serve statically
> if possible.

I would prefer not to touch the disk if possible, these are not huge
files just logs. I think I'm very close, when I serve this without the
zipfile code (just wrapping raw_data in a StringIO), a dialog pops up
asking me to save the file. However, with the zipfile code, the
request seems to take much longer (as if its downloading something)
and then an error is displayed in the dev console:

"Failed to load resource"
"Unsafe JavaScript attempt to access frame with URL
chrome://chromewebdata/ from frame with URL http://10.0.15.97:2345/.
Domains, protocols and ports must match."

Perhaps the headers need to be modified to serve the zip file? Any
suggestions are welcome here!
Matt

[peter]

As I have reported previously in this forum, I think that
response.stream does not quite stream zip files correctly.

So if I were you I would first stream a static zip file, when this is
working then try and get your code above working.

What I suggest you change in the code above is

response.headers['Content-Type'] = c.contenttype(filename)

where filename is the path to the zip file.

On Oct 13, 5:36 pm, Matt Broadstone <mbroa...@gmail.com> wrote:

> On Thu, Oct 13, 2011 at 11:34 AM, pbreit <pbreitenb...@gmail.com> wrote:
> > My first suggestion would be to save the files to disk and serve statically
> > if possible.
>
> I would prefer not to touch the disk if possible, these are not huge
> files just logs. I think I'm very close, when I serve this without the
> zipfile code (just wrapping raw_data in a StringIO), a dialog pops up
> asking me to save the file. However, with the zipfile code, the
> request seems to take much longer (as if its downloading something)
> and then an error is displayed in the dev console:
>
> "Failed to load resource"
> "Unsafe JavaScript attempt to access frame with URL

> chrome://chromewebdata/ from frame with URLhttp://10.0.15.97:2345/.

[Massimo Di Pierro]

On Oct 13, 12:46 pm, peter <peterchutchin...@gmail.com> wrote:
> As I have reported previously in this forum, I think that
> response.stream does not quite stream zip files correctly.

I have no bug report about this. Can you tell us more so we can fix it?

[Brian M]

I wrote a controller a few weeks ago that does this.  Takes in big dataset and splits it into multiple CSV files bundled into a zip, all using cStringIO

Here's part of it.
--------------------------
if record_count > 1000:
        #split into chunks of 500 each and bundle up in a zip file
        import zipfile, cStringIO
        #setup our filelike object to hold the zip file
        exported_chunks_zip = cStringIO.StringIO()
        zipf = zipfile.ZipFile(exported_chunks_zip, "w", compression=zipfile.ZIP_DEFLATED )

        #define chunk size and figure out how many we'll have
        chunk_size = 500
        chunk_count = record_count/chunk_size #technically this will often be 1 short due to partial chunks (less than 500 records, but we'll deal with that later)
        #start generating the chunks and adding them to zipf
        for c in range(0, chunk_count):
            start = c*chunk_size
            end = start+chunk_size
            chunk_records =records[start:end]
            #buld the csv file object
            csv_stream = csv_export(chunk_records, column_names, fields, mode="dict") #also uses cStringIO to hold csv "file"
            #add this chunk of csv to the zip file object.
            chunk_filename = "export chunk %s.csv" % (c)
            zipf.writestr(chunk_filename, csv_stream.getvalue())
        #ok now add in the records for any final partial chunk
        if chunk_count * chunk_size < record_count:
            chunk_records =records[end:]
            csv_stream = csv_export(chunk_records, column_names, fields, mode="dict")
            #add this chunk of csv to the zip file object.
            chunk_filename = "export chunk %s.csv" % (c+1)
            zipf.writestr(chunk_filename, csv_stream.getvalue())
        #close the zipf so necessary archive data & info gets included
        zipf.close()
        #return to browser
        response.headers['Content-Type']='multipart/x-zip'
        response.headers['Content-Disposition']='attachment; filename=export_%s.zip' % date.today()
        return exported_chunks_zip.getvalue() #causes browser to prompt for download of zip file with name specified above.

[peter]

I sent from my wifes email
http://groups.google.com/group/web2py/browse_thread/thread/fe85dca9e48c79ad

However with hindsight I think I did not give sufficient information
in my forum entry. I guess I was seeing if other people had had
problems with downloading zip files.

Today, I just tried the following

def downloady():

import os
import contenttype as c
path="somepath/album.zip"
response.headers['Content-Type'] = c.contenttype(path)
response.headers['Content-Disposition'] = 'attachment;
filename=album.zip'# to force download as attachment

return response.stream(open(path,'rb'),chunk_size=4096)

and this did work correctly. So I do not know why I was having
problems last month, when I repeatedly had problems with downloaded
zip files not unzipping.

So I aplogise for suggesting there was a bug here. I will see if I can
recreate the problems I was having and get to the root cause.

Peter



On Oct 14, 12:22 am, Massimo Di Pierro <massimo.dipie...@gmail.com>
wrote:

[peter]

If I now do exactly what I did one month ago, there is now no error
with zip streaming. So maybe you have changed things in
response.stream since then.

Peter

On Oct 14, 1:24 pm, peter <peterchutchin...@gmail.com> wrote:
> I sent from my wifes emailhttp://groups.google.com/group/web2py/browse_thread/thread/fe85dca9e4...

> > I have no bug report about this. Can you tell us more so we can fix it?- Hide quoted text -
>
> - Show quoted text -

[Matt Broadstone]

On Fri, Oct 14, 2011 at 9:35 AM, peter <peterchu...@gmail.com> wrote:
> If I now do exactly what I did one month ago, there is now no error
> with zip streaming. So maybe you have changed things in
> response.stream since then.
>
> Peter
>
> On Oct 14, 1:24 pm, peter <peterchutchin...@gmail.com> wrote:
>> I sent from my wifes emailhttp://groups.google.com/group/web2py/browse_thread/thread/fe85dca9e4...
>>
>> However with hindsight I think I did not give sufficient information
>> in my forum entry. I guess I was seeing if other people had had
>> problems with downloading zip files.
>>
>> Today, I just tried the following
>>
>> def downloady():
>>
>>     import os
>>     import contenttype as c
>>     path="somepath/album.zip"
>>     response.headers['Content-Type'] = c.contenttype(path)
>>     response.headers['Content-Disposition'] = 'attachment;
>> filename=album.zip'# to force download as attachment
>>
>>     return response.stream(open(path,'rb'),chunk_size=4096)
>>

Does this actually work for you? When I use this code, I get a
download, but it saves out a zero byte file with the proper name.

[Massimo Di Pierro]

What browser?


On Oct 14, 10:30 am, Matt Broadstone <mbroa...@gmail.com> wrote:

[Matt Broadstone]

On Fri, Oct 14, 2011 at 12:48 PM, Massimo Di Pierro
<massimo....@gmail.com> wrote:
> What browser?
That was chrome. The previously fix suggested by Brian works for me (thanks!).

Matt

[Massimo Di Pierro]

I understand. The problem is that response.stream does not form for a
StringIO which is not a regular file. StringIO is in memory therefore
you have no reason for streaming.

filename = "%s-backup" % (time.strftime("%Y%m%d-%H%M"))
raw_data = <a bunch of data>
stream = cStringIO.StringIO()
zip_file = zipfile.ZipFile(stream, "w", zipfile.ZIP_DEFLATED,
False)
zip_file.writestr(filename, raw_data)
response.headers['Content-Type'] = "application/octet-stream"
response.headers['Content-Disposition'] = "attachment;
filename=%s%s.bin" % (time.strftime("%Y%m%d-%H%M"), filename)

return stream.getvalue() ### << I only changed this

On Oct 14, 1:13 pm, Matt Broadstone <mbroa...@gmail.com> wrote:

[peter]

Okay this is where I am now.

My example 'downloady' above works correctly in chrome but incorrectly
in IE8. In IE8, the file appears to download correctly but will not
unzip.

Peter

On Oct 14, 7:13 pm, Matt Broadstone <mbroa...@gmail.com> wrote:

> >> >> - Show quoted text -- Hide quoted text -

[Massimo Di Pierro]

You mean the downloaded file is corrupted? Can you check the size?

[peter]

Yes, the downloaded file is corrupted with IE8 but not with chrome.
When I compare the files with a hex editor, they both begin and end
the same, however, the last byte of the original file is at 6EA11,
whereas the last byte of the downloaded file is at 6D7311.

So it appears that it is losing a few bytes but not at either end.

Windows reports the size of the two files as 6.91MB on server and
6.83MB after downloading

Peter

On Oct 14, 11:40 pm, Massimo Di Pierro <massimo.dipie...@gmail.com>

[Massimo Di Pierro]

Why
response.headers['Content-Type'] = "application/octet-stream"
shouldn't it be
response.headers['Content-Type'] = "application/zip"

I am not sure this causes the problem but it may be, if IE thinks the
data is ascii and not binary.

[peter]

This is the code I used, as above

def downloady():


import os
import contenttype as c
path="somepath/album.zip"
response.headers['Content-Type'] = c.contenttype(path)
response.headers['Content-Disposition'] = 'attachment;
filename=album.zip'# to force download as attachment


return response.stream(open(path,'rb'),chunk_size=4096)

Peter

On Oct 15, 3:53 pm, Massimo Di Pierro <massimo.dipie...@gmail.com>

[peter]

I have now tried the downloady example remotely, on a linux server
with niginx and uswgi. This streams the zip file correctly with both
chrome and IE8.

So the problem is there only when I use the rocket server on windows.

I just tried the rocket server remotely, and streaming is correct.

So I only get the problem with the rocket server on windows and
streaming on windows using IE8.

I hope that this helps. As I only need it to work remotely on my linux
server I am not left with a problem, but it does appear there is a
minor bug there.

Peter

[Yebach]

Using parts of your code I have a problem with zip file. It appends text to new file in my zip

So first worker has in his ics file his data but second one has his and from previous

This is my code

 cal = Calendar()

    import zipfile, cStringIO

    exported_chunks_zip = cStringIO.StringIO()

    zipf = zipfile.ZipFile(exported_chunks_zip, "w", compression=zipfile.ZIP_DEFLATED )

    

    for i, rec in enumerate(grouped):

        worker =  rec['rw_worker_nick'].encode('cp1250')

        for rr in rec["allData"]:

            startDate = rr['rw_date']

            startTime = rr['rw_time_start']

            endTime = rr['rw_time_end']

            evtstart = datetime.datetime.combine(startDate,startTime)

            evtend = datetime.datetime.combine(startDate,endTime)

            event = Event()

            

            event.add('summary', rec['rw_worker_nick'])

            event.add('dtstart', evtstart)

            event.add('dtend', evtend)

            cal.add_component(event)

            text = cal.to_ical()

        zipf.writestr(worker +'.ics', text)

        text = ''