Chunked downloads and corrupt files with Internet Explorer 8 (IE8)

[Stefan Scholl]

The classic download function:

def download():
return response.download(request, db)


I'm developing on localhost (127.0.0.1, no SSL) and one strange thing
happened: Downloads in IE8 (Windows XP) were all corrupt/broken if
they weren't below 64KiB in size. Very easy to see with large images.

Using a higher value for the argument 'chunk_size' solves this
problem, up to this new maximum.

web2py 1.91.6

[Massimo Di Pierro]

Can you try 1.95.1

[Stefan Scholl]

Sorry, no time for updates at the moment.

Updated this project 2 times and every time it broke something.

Haven't seen anything related to downloads, streaming etc. in the
changelog, though.


On May 6, 5:51 pm, Massimo Di Pierro <massimo.dipie...@gmail.com>
wrote:

[Stefan Scholl]

Is nobody else experiencing this problem? Is nobody using Internet
Explorer to download more than 64KiB from a web2py app?

On 6 Mai, 13:03, Stefan Scholl <stefan.sch...@gmail.com> wrote:
> The classicdownloadfunction:
>
> defdownload():

[Stefan Scholl]

The higher value for chunk_size didn't work with a 33 MiB file. Even
in Firefox 4.
So I tried 1.96.4 (Rocket 1.2.2) on Windows XP.

Made a new and simple app (dtest). The download there uses
"response.download(request,db)" as well.

1 simple table: db.define_table('stuff', Field('file', 'upload'))

Upload of the 33 MiB file via db admin, content listed on
http://127.0.0.1:8001/dtest/default/data/select/stuff (default
function "data" with "return dict(form=crud())". Download with
Internet Explorer 8 (after removing the tag that switches to "Chrome
Frame", to have a realistic test like "normal" users).

Download was broken. A few KiB were missing. This was on localhost.
Remote tests have even worse results.

[Stefan Scholl]

(This answer was moved to another group with another signature. Don't
know what's wrong here in Google Groups.)

The higher value for chunk_size didn't work with a 33 MiB file. Even
in Firefox 4.
So I tried 1.96.4 (Rocket 1.2.2) on Windows XP.

Made a new and simple app (dtest). The download there uses
"response.download(request,db)" as well.

1 simple table: db.define_table('stuff', Field('file', 'upload'))

Upload of the 33 MiB file via db admin, content listed on
http://127.0.0.1:8001/dtest/default/data/select/stuff (default

function "data" with "return dict(form=3Dcrud())". Download with

Internet Explorer 8 (after removing the tag that switches to "Chrome
Frame", to have a realistic test like "normal" users).

Download was broken. A few KiB were missing. This was on localhost.
Remote tests have even worse results.

On May 6, 5:51 pm, Massimo Di Pierro <massimo.dipie...@gmail.com>
wrote:

[Stefan Scholl]

OK, it was Rocket.

Tested it with the old web2py and Tornado 1.2.1 via anyserver.py
and the download is OK.

[Stefan Scholl]

Now I can't access the admin interface, because the password
isn't set. (It isn't reading the stored password.)

[Massimo Di Pierro]

Try this:

python
>>> from gluon.main import save_password
>>> save_password(raw_input('admin password: '),XXX)

This will create a parameters_XXX.py file. It must be in the main
web2py folder. Caveats, the admin interface is disabled if you are not
form localhost and you are not using https.
Hope this helps. Hope to have you back on the mailing list.

On Jun 17, 8:43 am, Stefan Scholl <ste...@no-spoon.de> wrote:
> Now I can't access the admin interface, because the password
> isn't set. (It isn't reading the stored password.)
>
>
>
>
>
>
>
> Stefan Scholl <ste...@no-spoon.de> wrote:
> > OK, it was Rocket.
>
> > Tested it with the old web2py and Tornado 1.2.1 via anyserver.py
> > and the download is OK.
>

[Stefan Scholl]

I have a parameters_XXX.py file from the normal web2py (with
rocket) and used the same IP and port with anyserver.py+Tornado
(and the other one stopped, of course).

Tested with web2py 1.91.6. Were there any changes regarding this?

(I'm still very reluctant to upgrade this project.)

--
Web (en): http://www.no-spoon.de/ -*- Web (de): http://www.frell.de/
<!--[if IE 6]><script>for(x in document.open);</script><![endif]-->

[Stefan Scholl]

To Massimo and the list/group:

You asked on Reddit if the only constant is the browser. No it
isn't. But it was the browser which had the problem first, with
smaller files.

For smaller files it was enough to raise the chunk_size. IE8 is
slow, maybe this is the reason?

Firefox 4 failed when I tried to download a 33 MiB file remotely.
IE8 failed for anything above 64 KiB on localhost.

It's almost as if Rocket is so fast because it sends without
regard for any receiver. Direct (localhost) or behind a proxy
(Apache 2.2 on the remote Linux server).
Don't know how this could happen. HTTP isn't ZModem. ;-)

[Massimo Di Pierro]

Can you make a list of combinations

browser name, version, server os, web server
FF, 4, Windows 7, Rocket
...

for which you experienced the problem?

can you also try the following:
1) in the rocket.py code replace

'HTTP/1.1 ' with 'HTTP/1.0 '

and replace

environ['SERVER_PROTOCOL'] = request['protocol']

with

environ['SERVER_PROTOCOL'] = "1.0"

Looks like acts as if the protocol of response is the same as the
request but always declare the protocol of the response to be 1.1 even
if the request is 1.0. This may result in keep-alive connections
ignored by the browser. Perhaps this is part of the problem?

On Jun 17, 1:35 pm, Stefan Scholl <ste...@no-spoon.de> wrote:
> To Massimo and the list/group:
>
> You asked on Reddit if the only constant is the browser. No it
> isn't. But it was the browser which had the problem first, with
> smaller files.
>
> For smaller files it was enough to raise the chunk_size. IE8 is
> slow, maybe this is the reason?
>
> Firefox 4 failed when I tried to download a 33 MiB file remotely.
> IE8 failed for anything above 64 KiB on localhost.
>
> It's almost as if Rocket is so fast because it sends without
> regard for any receiver. Direct (localhost) or behind a proxy
> (Apache 2.2 on the remote Linux server).
> Don't know how this could happen. HTTP isn't ZModem. ;-)
>
>
>
>
>
>
>
>
>
> Stefan Scholl <ste...@no-spoon.de> wrote:
> > I have a parameters_XXX.py file from the normal web2py (with
> > rocket) and used the same IP and port with anyserver.py+Tornado
> > (and the other one stopped, of course).
>
> > Tested with web2py 1.91.6. Were there any changes regarding this?
>
> > (I'm still very reluctant to upgrade this project.)
>

> Web (en):http://www.no-spoon.de/-*- Web (de):http://www.frell.de/

[Stefan Scholl]

Sever OS: Linux (remote, behind Apache 2.2), Windows XP (local,
direct)
Client OS: Windows XP
Browser: Firefox 4, Internet Explorer 8
Rocket version: 1.2.2

All combinations break the download for big files (33 MiB),
regardless of chunk_size or server.

Only Internet Explorer 8 (all servers) had problems with small
files (160 KiB), before increasing chunk_size for the streamed
download.

Made the changes to rocket.py (1.2.2), restarted web2py, and the
download was still broken.

By the way: One of the new examples (Dog and owner registration,
with picture upload/download) doesn't use streamed download. It
reads the whole file and sends it. This method doesn't work for
big files, too.

Massimo Di Pierro <massimo....@gmail.com> wrote:

--
Web (en): http://www.no-spoon.de/ -*- Web (de): http://www.frell.de/

[Massimo Di Pierro]

I just tried with chrome and osx and I cannot reproduce the problem (I
tried with a 166MB file).

TIm, who made rocket, also claims he tried this extensively on windows
and cannot reproduce the problem.

I do not doubt you experience this issue. In order to try isolate
better what may be causing it... is there anybody else having this
problem with large files download?

Massimo

On Jun 20, 2:53 am, Stefan Scholl <ste...@no-spoon.de> wrote:
> Sever OS: Linux (remote, behind Apache 2.2), Windows XP (local,
> direct)
> Client OS: Windows XP
> Browser: Firefox 4, Internet Explorer 8
> Rocket version: 1.2.2
>
> All combinations break the download for big files (33 MiB),
> regardless of chunk_size or server.
>
> Only Internet Explorer 8 (all servers) had problems with small
> files (160 KiB), before increasing chunk_size for the streamed
> download.
>
> Made the changes to rocket.py (1.2.2), restarted web2py, and the
> download was still broken.
>
> By the way: One of the new examples (Dog and owner registration,
> with picture upload/download) doesn't use streamed download. It
> reads the whole file and sends it. This method doesn't work for
> big files, too.
>

> >> Web (en):http://www.no-spoon.de/-*-Web (de):http://www.frell.de/

[Anthony]

I've tried on Windows 7 in IE 7, 8, and 9 with a 50MB file and cannot reproduce the problem either.

 

Anthony

[Stefan Scholl]

I had to do something perverted to save the project: I make a
local redirect to a PHP script, which uses readfile
<http://de.php.net/manual/en/function.readfile.php>. No problem
there. (Retrieving the original file name and content-type in
web2py.)

It's only rocket. And it was reported by a co-worker and a
customer.

One silly idea: What if Rocket has problems with some
proxies/firewalls? The customer has a broken proxy which caused
different problems for other projects.

I don't know about any proxy at work, but it could be that they
reroute everything on port 80 through a proxy without me knowing.
(Ignore the non-technical implications of this assumption.)

Massimo Di Pierro <massimo....@gmail.com> wrote:

--
Web (en): http://www.no-spoon.de/ -*- Web (de): http://www.frell.de/

[Stefan Scholl]

Forget it. (If this would be a real newsgroup I would cancel the
article.)

Forgot about the problems on localhost.

[David Tse]

I'm experiencing this problem as well, and signs on my side are also pointing to Rocket.  Although my case is a little different because I'm not currently experiencing this problem with web2py, but I'm using Rocket by itself to serve a light-weight service using a smaller micro-framework.  However, I have experience this in the past with web2py, and I forgot what I did at the time but the problem went away somehow.

Back to the issue at hand.  I'm current using Rocket to serve a tiny WSGI app and using bottle.py as the framework, and I've seen large file downloads (100MB+) missing anywhere between 8 to 32 KB, usually 8K.  I've tested with serving directly with Rocket and also proxy-ing Rocket behind Nginx.  I'm also using SSL when serving directly with Rocket, and Nginx doing the SSL when running behind it.  The issue is somewhat intermittent, and for some reason it seems to happen less behind Nginx but still does.  I primarily use curl for my testing, and when it does happen curl would get stuck near the very end and times out after a while.

I'm running with Rocket 1.2.4, on Ubuntu 10.04.  The problem goes away if I change to cherrypy or gevent, which is what leads me to think it might be Rocket.  I found this thread when googling to see if anyone else was experiencing this in the web2py community.  I'd be happy to help track this down.

[Timothy Farrell]

David,

Thanks for your offer to help with this. The best way to help right
now would be to provide me a smallish pcap file that records it
happening so I can see which parts of the files are missing.

Thanks,
Timothy Farrell

[Timothy Farrell]

https://github.com/explorigin/Rocket/issues/1

[nick name]

I posted https://github.com/explorigin/Rocket/issues/1#issuecomment-3648126 - I suspect it is an interplay between timeouts and sendall(), though I can't really prove it (and I can't reliably reproduce this either right now). Also some characterization about when this happens to me (slow links, dependency on chunk size, etc).

[Phyo Arkar]

I am using 1.99.1 and it happens to me frequently. If download is big ( for my case > 100,200 MB) it usually end up with corrupted file. Tested on Local wifi network with 300 Mbps.

[Massimo Di Pierro]

which python version? Did you try different python versions?

On Jan 26, 2:52 am, Phyo Arkar <phyo.arkarl...@gmail.com> wrote:
> I am using 1.99.1 and it happens to me frequently. If download is big ( for
> my case > 100,200 MB) it usually end up with corrupted file. Tested on
> Local wifi network with 300 Mbps.
>

[nick name]

I posted https://github.com/explorigin/Rocket/issues/1#issuecomment-3648126 - I suspect it is an interplay between timeouts and sendall(), though I can't really prove it (and I can't reliably reproduce this either right now). Also some characterization about when this happens to me (slow links, dependency on chunk size, etc).

TL;DR: I think it is impossible to use sendall/write reliably with Python 2.6's socket wrapper if there's a chance for a timeout (Not sure about earlier versions). Unfortunately, Rocket seems to use sendall/write a lot.

I'll be able to test this more next week.

[Massimo Di Pierro]

Thank you. This is really an important issue and should get priority.

On Jan 27, 4:04 pm, nick name <i.like.privacy....@gmail.com> wrote:
> I postedhttps://github.com/explorigin/Rocket/issues/1#issuecomment-3648126-I suspect it is an interplay between timeouts and sendall(), though I

[Phyo Arkar]

its 2.7 as all my servers are (distro default)

[nick name]

Ok, the culprit is definitely ignoring exceptions raised in sendall. In my humble opinion this is serious enough to be on the 2.0 blocker list.

How to reproduce: you have to have a wsgi worker, that produces output in parts (that is, returns a list or yields part as a generator). e.g: use web2py's "static" file server (which uses wsgi and does not use the FileSystermWorker).

1. Make sure that there's a large payload produced, and that it is made of a lot of small parts. e.g. put a 10MB file in web2py/applications/welcome/static/file10mb.data (web2py will use 64K parts by default)
2. Consume file slowly, e.g. wget --limit=100k http://localhost:8000/welcome/static/file10mb.data ; this would take 100 seconds to download the whole file even on localhost.
3. Let file download for 10 seconds, then pause wget (e.g. suspend it by using Ctrl-Z on linux/osx)
4. Wait 20 seconds
5. Let it continue (e.g. type 'fg' if you suspended it with ctrl-z)
6. Notice that when it reaches the end, wget will complain about missing bytes, reconnect and download the rest of the file (and will be happy with it). However, the file will be corrupt: A block (or many blocks) will be missing from the middle, and the last few blocks will be repeated (by the 2nd wget connection; if you disallow wget from resuming, the file will just be shorter).

A better idea where the problem is can be seen from the following ugly patch (applied against web2py's "one file" rocket.py)

@@ -1929,6 +1929,9 @@ class WSGIWorker(Worker):
                 self.conn.sendall(b('%x\r\n%s\r\n' % (len(data), data)))
             else:
                 self.conn.sendall(data)
+        except socket.timeout:
+            self.closeConnection = True
+            print 'Exception lost'
         except socket.error:
             # But some clients will close the connection before that
             # resulting in a socket error.
 

Running the same experiment with the patched rocket.py will show that files get corrupted if 'exception lost' is printed to the web2py's terminal.

Discussion: The only way to use sendall() reliably is to immediately terminate the connection upon any error (including timeout), as there is no way to know how many bytes were sent. (That there is no way to know how many bytes were sent is clearly stated in the documentation; the implication that it is impossible to reliably recover from this is not). However, there are sendall() calls all over rocket.py, and some will result in additional sendalls() following a failed sendall(). The worst offender seems to be WSGIWorker.write(), but I'm not sure the other sendalls are safe either.

Temporary workaround: increase SOCKET_TIMEOUT significantly (default is 1 second; bump to e.g. 10), and not swallow socket.timeout in WSGIWorker.write().

Increasing the chunk size is NOT a helpful, because it only changes the number of bytes before the first loss (at a given bandwidth), but from that point, the problem is the same.

cross reference: https://github.com/explorigin/Rocket/issues/1#issuecomment-3734231

[nick name]

On Saturday, January 28, 2012 10:22:58 AM UTC-5, Phyo Arkar wrote:

its 2.7 as all my servers are (distro default)

Sorry for the confusion. This is true for every version down to at least 2.3 and up to 2.7; At the time I posted, I wrote 2.6 because that was the only one I verified and did not want to make a possibly incorrect generalization. See my post above in this thread that discusses the cause.

[Massimo Di Pierro]

This is very valuable.

In trunk socket timeout is 60 and this resulted in another problem.
Ctrl-C waits for 60 seconds before joining the worker processes.
Perhaps we should increate socket-timeout, catch Ctrl+C and then kill
the process instead of joining the workers.

On Jan 31, 12:16 am, nick name <i.like.privacy....@gmail.com> wrote:
> Ok, the culprit is definitely ignoring exceptions raised in sendall. In my
> humble opinion this is serious enough to be on the 2.0 blocker list.
>
> How to reproduce: you have to have a wsgi worker, that produces output in
> parts (that is, returns a list or yields part as a generator). e.g: use
> web2py's "static" file server (which uses wsgi and does not use the
> FileSystermWorker).
>

>    1. Make sure that there's a large payload produced, and that it is made

>    of a lot of small parts. e.g. put a 10MB file in
>    web2py/applications/welcome/static/file10mb.data (web2py will use 64K parts
>    by default)

>    2. Consume file slowly, e.g. wget --limit=100k

>    http://localhost:8000/welcome/static/file10mb.data; this would take 100
>    seconds to download the whole file even on localhost.

>    3. Let file download for 10 seconds, then pause wget (e.g. suspend it by
>    using Ctrl-Z on linux/osx)
>    4. Wait 20 seconds
>    5. Let it continue (e.g. type 'fg' if you suspended it with ctrl-z)
>    6. Notice that when it reaches the end, wget will complain about missing

[nick name]

On Tuesday, January 31, 2012 9:37:54 AM UTC-5, Massimo Di Pierro wrote:

In trunk socket timeout is 60 and this resulted in another problem.
Ctrl-C waits for 60 seconds before joining the worker processes.
Perhaps we should increate socket-timeout, catch Ctrl+C and then kill
the process instead of joining the workers.

Note that rocket.py has two "socket timeout" settings:
1. a constant of 1 second (in web2py trunk and in rocket trunk), which is the socket timeout passed down to the python socket library. calling send() will raise an exception if not a single byte can be sent within that time. calling sendall() will  raise an exception if this time has passed without being able to send a single byte (but: if it succeeds in sending 1 byte every second, sendall() will never actually time out - a 100K file will take more than 24hours!).

2. The timeout passed to the rocket initialization. This is the "monitor" timeout, which rocket will use to determine if a connection is stale (or otherwise working too slow).

So the 60 seconds that affect Ctrl-C is almost unrelated to the SOCKET_TIMEOUT constant in rocket which can be used to see if sendall() is indeed the cause of the corruption problem

[Massimo Di Pierro]

I agree with the fact that there are two timeouts but the one I am
talking about is SOCKET_TIMEOUT.
It is set in --socket-timeout in widget.py and passed to rocket in
line 777 of main.py

rocket.SOCKET_TIMEOUT = socket_timeout

which does change the constant value SOCKET_TIMEOUT=1 to 60.

I am pretty sure decreasing this value fixes this problem on my
machine.

[Phyo Arkar]

have you guys test on upload too?

i have corrupted files on upload too.

mostly on slow connection (well my internet connection only have 10 - 20K/s upload speed.