file upload fails with slow network bandwith in django

[Pablo Conesa]

Hi, I posted this in stackoverflow. Now I'm trying here: http://stackoverflow.com/questions/35413649/file-upload-fails-with-slow-network-bandwith-in-django

I've got a code working fine that uploads a file to a DJANGO server.

It works fine on a fine connection.

Now if, using fiddler to simulate a slow connection....page freezes (Chrome pop up offering to kill the page).

I've tested in different ways and the only factor that causes the error is a slow upload speed. e.g. If I choose a wi-fi network it fails but over the ethernet it works.

File is 100MB!.

python: 2.7.6 

DJANGO: 1.5.5 or 1.9 (I updated to 1.9 and is not working either)

Fiddler show -1 in the response, and says: "No response body".

Any ideas?

[Ezequiel Bertti]

You are using de runserver to develop and getting this error or something like gunicorn and nginx?

Because the nginx have some time to proccess each request.

http://stackoverflow.com/questions/6816215/gunicorn-nginx-timeout-problem

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/3625feed-bc42-4d80-a8cb-d6a8c94a48cf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Ezequiel Bertti
E-Mail: ebe...@gmail.com
Cel: (21) 99188-4860

[James Schneider]

How long does the file upload work before receiving the error? 

My guess is that Fiddler is attempting to use an HTTP chunked transfer to send the file 'slowly' (by rate-limiting the chunks it sends rather than traffic shaping the data stream itself), which is how modern JS libraries normally handle large file uploads. However, this takes advantage of features built into the HTTP 1.1 standard, and I'm not entirely sure whether or not the Django dev server fully implements HTTP 1.1. That would explain why you are getting a response body error (since the Django dev server probably doesn't understand chunked requests and doesn't know to expect more data). See https://code.djangoproject.com/ticket/25619 That's purely a guess though, as I'm not familiar with how Fiddler works, and Django's internal support for HTTP 1.1 is somewhat ambiguous. 

This type of testing should be done against the production server (which likely fully implements HTTP 1.1), not the Django dev server, since performance of the dev server cannot be correlated with a production server that implements threading and/or multiple worker processes.

On a side note, though. If you plan to have users regularly upload large files (>2MB), you should seriously look in to implementing chunked uploads via a JS library. With a standard upload via a regular form, the worker process in your server handling the upload are dedicated to that upload until it finishes. Usually this is fine for small uploads, but for larger uploads that can take minutes or hours, it becomes a large problem even on a site with a small user base. If 5 users are uploading a large file, and you only have 5 worker processes running, then literally nobody can access your site until at least one of those uploads finishes. With chunked transfers, the file is broken up into pieces and sent as many small requests rather than a single large one. This allows a server worker process to process the small upload, then step away to serve other users, and then process the next small upload. The net result on a busy server is usually a slower response, but at least there is a response.

-James

[Pablo Conesa]

Thanks Ezequiel, answer below.

El lunes, 15 de febrero de 2016, 21:22:23 (UTC+1), Ezequiel Bertti escribió:

You are using de runserver to develop and getting this error or something like gunicorn and nginx?

In production  we are using gunicorn, locally I don't use guinicorn, and in both cases it happens the same.

Unfortunately there is no message, no time out, nothing from the server.

[Pablo Conesa]

Thanks James. answers in line:

El lunes, 15 de febrero de 2016, 23:04:48 (UTC+1), James Schneider escribió:

On Mon, Feb 15, 2016 at 10:20 AM, Pablo Conesa <p.cones...@gmail.com> wrote:

Hi, I posted this in stackoverflow. Now I'm trying here: http://stackoverflow.com/questions/35413649/file-upload-fails-with-slow-network-bandwith-in-django

I've got a code working fine that uploads a file to a DJANGO server.

It works fine on a fine connection.

Now if, using fiddler to simulate a slow connection....page freezes (Chrome pop up offering to kill the page).

I've tested in different ways and the only factor that causes the error is a slow upload speed. e.g. If I choose a wi-fi network it fails but over the ethernet it works.

File is 100MB!.

python: 2.7.6 

DJANGO: 1.5.5 or 1.9 (I updated to 1.9 and is not working either)

Fiddler show -1 in the response, and says: "No response body".

Any ideas?

How long does the file upload work before receiving the error? 

There is no response from the server (fiddler returns -1, whatever this means) 

My guess is that Fiddler is attempting to use an HTTP chunked transfer to send the file 'slowly' (by rate-limiting the chunks it sends rather than traffic shaping the data stream itself), which is how modern JS libraries normally handle large file uploads. However, this takes advantage of features built into the HTTP 1.1 standard, and I'm not entirely sure whether or not the Django dev server fully implements HTTP 1.1. That would explain why you are getting a response body error (since the Django dev server probably doesn't understand chunked requests and doesn't know to expect more data). See https://code.djangoproject.com/ticket/25619 That's purely a guess though, as I'm not familiar with how Fiddler works, and Django's internal support for HTTP 1.1 is somewhat ambiguous. 

Don't know either details about FIddler..but what you are saying sounds reasonable and in the right direction...

This type of testing should be done against the production server (which likely fully implements HTTP 1.1), not the Django dev server, since performance of the dev server cannot be correlated with a production server that implements threading and/or multiple worker processes.

I've done this. Actually the bug was reported in production (apache forwarding to guinicorn)

 

On a side note, though. If you plan to have users regularly upload large files (>2MB), you should seriously look in to implementing chunked uploads via a JS library. With a standard upload via a regular form, the worker process in your server handling the upload are dedicated to that upload until it finishes. Usually this is fine for small uploads, but for larger uploads that can take minutes or hours, it becomes a large problem even on a site with a small user base. If 5 users are uploading a large file, and you only have 5 worker processes running, then literally nobody can access your site until at least one of those uploads finishes. With chunked transfers, the file is broken up into pieces and sent as many small requests rather than a single large one. This allows a server worker process to process the small upload, then step away to serve other users, and then process the next small upload. The net result on a busy server is usually a slower response, but at least there is a response.

Nice advice...I'll look into this. Do you know any nice example for that JS library chunk upload.

-James

[James Schneider]

On Mon, Feb 15, 2016 at 10:41 PM, Pablo Conesa <p.cones...@gmail.com> wrote:

Thanks James. answers in line:

El lunes, 15 de febrero de 2016, 23:04:48 (UTC+1), James Schneider escribió:

On Mon, Feb 15, 2016 at 10:20 AM, Pablo Conesa <p.cones...@gmail.com> wrote:

Hi, I posted this in stackoverflow. Now I'm trying here: http://stackoverflow.com/questions/35413649/file-upload-fails-with-slow-network-bandwith-in-django

I've got a code working fine that uploads a file to a DJANGO server.

It works fine on a fine connection.

Now if, using fiddler to simulate a slow connection....page freezes (Chrome pop up offering to kill the page).

I've tested in different ways and the only factor that causes the error is a slow upload speed. e.g. If I choose a wi-fi network it fails but over the ethernet it works.

File is 100MB!.

python: 2.7.6 

DJANGO: 1.5.5 or 1.9 (I updated to 1.9 and is not working either)

Fiddler show -1 in the response, and says: "No response body".

Any ideas?

How long does the file upload work before receiving the error? 

There is no response from the server (fiddler returns -1, whatever this means) 

Well, no response from the server doesn't necessarily mean that there were no logs. Can you check the error and access logs of the server? 

Does Fiddler return an immediate response of -1, or does it seem like there is a timeout period?

After a quick glance at Fiddler itself, are you using any customizations on the rules 

My guess is that Fiddler is attempting to use an HTTP chunked transfer to send the file 'slowly' (by rate-limiting the chunks it sends rather than traffic shaping the data stream itself), which is how modern JS libraries normally handle large file uploads. However, this takes advantage of features built into the HTTP 1.1 standard, and I'm not entirely sure whether or not the Django dev server fully implements HTTP 1.1. That would explain why you are getting a response body error (since the Django dev server probably doesn't understand chunked requests and doesn't know to expect more data). See https://code.djangoproject.com/ticket/25619 That's purely a guess though, as I'm not familiar with how Fiddler works, and Django's internal support for HTTP 1.1 is somewhat ambiguous. 

Don't know either details about FIddler..but what you are saying sounds reasonable and in the right direction...

This type of testing should be done against the production server (which likely fully implements HTTP 1.1), not the Django dev server, since performance of the dev server cannot be correlated with a production server that implements threading and/or multiple worker processes.

I've done this. Actually the bug was reported in production (apache forwarding to guinicorn)

Hmm, Apache should handle that fine. It's had 1.1 support forever, and I don't believe it requires any special configuration. Might be related to gunicorn, though, if it is responsible for handling the chunked response. Found this to be similar to your issue: https://github.com/benoitc/gunicorn/issues/1125, although sadly no resolution. Might point you in the right direction, though.

 

 

On a side note, though. If you plan to have users regularly upload large files (>2MB), you should seriously look in to implementing chunked uploads via a JS library. With a standard upload via a regular form, the worker process in your server handling the upload are dedicated to that upload until it finishes. Usually this is fine for small uploads, but for larger uploads that can take minutes or hours, it becomes a large problem even on a site with a small user base. If 5 users are uploading a large file, and you only have 5 worker processes running, then literally nobody can access your site until at least one of those uploads finishes. With chunked transfers, the file is broken up into pieces and sent as many small requests rather than a single large one. This allows a server worker process to process the small upload, then step away to serve other users, and then process the next small upload. The net result on a busy server is usually a slower response, but at least there is a response.

Nice advice...I'll look into this. Do you know any nice example for that JS library chunk upload.

Probably best to stick with your existing JS framework of choice if you are using one. All of the major ones have some version of a chunked uploader (JQuery and Angular), and I come across resumable.js fairly frequently. I've never personally had a need for one, so I can't really comment on them in any sort of detail or make a recommendation. 

-James

[Pablo Conesa]

Thanks once more James for your help and time. Answers below..

El martes, 16 de febrero de 2016, 8:33:08 (UTC+1), James Schneider escribió:

On Mon, Feb 15, 2016 at 10:41 PM, Pablo Conesa <p.cones...@gmail.com> wrote:

Thanks James. answers in line:

El lunes, 15 de febrero de 2016, 23:04:48 (UTC+1), James Schneider escribió:

On Mon, Feb 15, 2016 at 10:20 AM, Pablo Conesa <p.cones...@gmail.com> wrote:

Hi, I posted this in stackoverflow. Now I'm trying here: http://stackoverflow.com/questions/35413649/file-upload-fails-with-slow-network-bandwith-in-django

I've got a code working fine that uploads a file to a DJANGO server.

It works fine on a fine connection.

Now if, using fiddler to simulate a slow connection....page freezes (Chrome pop up offering to kill the page).

I've tested in different ways and the only factor that causes the error is a slow upload speed. e.g. If I choose a wi-fi network it fails but over the ethernet it works.

File is 100MB!.

python: 2.7.6 

DJANGO: 1.5.5 or 1.9 (I updated to 1.9 and is not working either)

Fiddler show -1 in the response, and says: "No response body".

Any ideas?

How long does the file upload work before receiving the error? 

There is no response from the server (fiddler returns -1, whatever this means) 

Well, no response from the server doesn't necessarily mean that there were no logs. Can you check the error and access logs of the server? 

 

Does Fiddler return an immediate response of -1, or does it seem like there is a timeout period?

immediate response, no time out period.

After a quick glance at Fiddler itself, are you using any customizations on the rules 

no customization. Default "Simulate Modem speeds". I must say I use fiddle to be able to reproduce the error locally. If I hit the server (production) using wifi (slower than ethernet) I got the same thing, no fiddler in the middle.

My guess is that Fiddler is attempting to use an HTTP chunked transfer to send the file 'slowly' (by rate-limiting the chunks it sends rather than traffic shaping the data stream itself), which is how modern JS libraries normally handle large file uploads. However, this takes advantage of features built into the HTTP 1.1 standard, and I'm not entirely sure whether or not the Django dev server fully implements HTTP 1.1. That would explain why you are getting a response body error (since the Django dev server probably doesn't understand chunked requests and doesn't know to expect more data). See https://code.djangoproject.com/ticket/25619 That's purely a guess though, as I'm not familiar with how Fiddler works, and Django's internal support for HTTP 1.1 is somewhat ambiguous. 

Don't know either details about FIddler..but what you are saying sounds reasonable and in the right direction...

This type of testing should be done against the production server (which likely fully implements HTTP 1.1), not the Django dev server, since performance of the dev server cannot be correlated with a production server that implements threading and/or multiple worker processes.

I've done this. Actually the bug was reported in production (apache forwarding to guinicorn)

Hmm, Apache should handle that fine. It's had 1.1 support forever, and I don't believe it requires any special configuration. Might be related to gunicorn, though, if it is responsible for handling the chunked response. Found this to be similar to your issue: https://github.com/benoitc/gunicorn/issues/1125, although sadly no resolution. Might point you in the right direction, though.

Looks like the same problem ... 

 

 

On a side note, though. If you plan to have users regularly upload large files (>2MB), you should seriously look in to implementing chunked uploads via a JS library. With a standard upload via a regular form, the worker process in your server handling the upload are dedicated to that upload until it finishes. Usually this is fine for small uploads, but for larger uploads that can take minutes or hours, it becomes a large problem even on a site with a small user base. If 5 users are uploading a large file, and you only have 5 worker processes running, then literally nobody can access your site until at least one of those uploads finishes. With chunked transfers, the file is broken up into pieces and sent as many small requests rather than a single large one. This allows a server worker process to process the small upload, then step away to serve other users, and then process the next small upload. The net result on a busy server is usually a slower response, but at least there is a response.

Nice advice...I'll look into this. Do you know any nice example for that JS library chunk upload.

Probably best to stick with your existing JS framework of choice if you are using one. All of the major ones have some version of a chunked uploader (JQuery and Angular), and I come across resumable.js fairly frequently. I've never personally had a need for one, so I can't really comment on them in any sort of detail or make a recommendation. 

Testing resumable.js....let's see how it goes. 

So, far there is some response, but after a while got 404 response (testing locally against DAJNGO)....will it make a difference against gunicorn?

-James

Many thanks James. 

[James Schneider]

How long does the file upload work before receiving the error? 

There is no response from the server (fiddler returns -1, whatever this means) 

Well, no response from the server doesn't necessarily mean that there were no logs. Can you check the error and access logs of the server? 

Have you checked the server logs?

 

 

Does Fiddler return an immediate response of -1, or does it seem like there is a timeout period?

immediate response, no time out period.

Without looking at the server logs, I can only guess that Gnuicorn is unhappy with the headers that are being sent, and might be closing the connection. A long timeout would probably indicate that Gnuicorn is waiting for more data but something in the upload process is not triggering that it has reached the end of the data stream.

 

After a quick glance at Fiddler itself, are you using any customizations on the rules 

no customization. Default "Simulate Modem speeds". I must say I use fiddle to be able to reproduce the error locally. If I hit the server (production) using wifi (slower than ethernet) I got the same thing, no fiddler in the middle.

But it works using a wired Ethernet connection? Honestly, the speed difference between Wi-Fi and Ethernet for this purpose is probably not relevant to the problem. The server logs would tell you more about what is happening.

 

Nice advice...I'll look into this. Do you know any nice example for that JS library chunk upload.

Probably best to stick with your existing JS framework of choice if you are using one. All of the major ones have some version of a chunked uploader (JQuery and Angular), and I come across resumable.js fairly frequently. I've never personally had a need for one, so I can't really comment on them in any sort of detail or make a recommendation. 

Testing resumable.js....let's see how it goes. 

So, far there is some response, but after a while got 404 response (testing locally against DAJNGO)....will it make a difference against gunicorn?

This smells like either a memory exhaustion/ceiling issue or some kind of internal connection/process timeout, neither of which are likely easy to adjust using the Django dev server. Gunicorn probably has 10,000 knobs to adjust those timers and limits, though. 

Look at the server error/access logs.

-James

[Pablo Conesa]

El martes, 16 de febrero de 2016, 21:12:32 (UTC+1), James Schneider escribió:

How long does the file upload work before receiving the error? 

There is no response from the server (fiddler returns -1, whatever this means) 

Well, no response from the server doesn't necessarily mean that there were no logs. Can you check the error and access logs of the server? 

Have you checked the server logs?

Yes....there are no logs!! I know....just joined the project and we are using the email handler....sad! 

 

 

Does Fiddler return an immediate response of -1, or does it seem like there is a timeout period?

immediate response, no time out period.

Without looking at the server logs, I can only guess that Gnuicorn is unhappy with the headers that are being sent, and might be closing the connection. A long timeout would probably indicate that Gnuicorn is waiting for more data but something in the upload process is not triggering that it has reached the end of the data stream.

I've done some more test and seems that the -1 is normal while data it's been uploaded. Nothing wrong with that. So I guess we need to config the logging and start looking at it. 

 

After a quick glance at Fiddler itself, are you using any customizations on the rules 

no customization. Default "Simulate Modem speeds". I must say I use fiddle to be able to reproduce the error locally. If I hit the server (production) using wifi (slower than ethernet) I got the same thing, no fiddler in the middle.

But it works using a wired Ethernet connection? Honestly, the speed difference between Wi-Fi and Ethernet for this purpose is probably not relevant to the problem. The server logs would tell you more about what is happening.

 

logs, logs, logs....I'm on it. :-( 

Nice advice...I'll look into this. Do you know any nice example for that JS library chunk upload.

Probably best to stick with your existing JS framework of choice if you are using one. All of the major ones have some version of a chunked uploader (JQuery and Angular), and I come across resumable.js fairly frequently. I've never personally had a need for one, so I can't really comment on them in any sort of detail or make a recommendation. 

Testing resumable.js....let's see how it goes. 

So, far there is some response, but after a while got 404 response (testing locally against DAJNGO)....will it make a difference against gunicorn?

This smells like either a memory exhaustion/ceiling issue or some kind of internal connection/process timeout, neither of which are likely easy to adjust using the Django dev server. Gunicorn probably has 10,000 knobs to adjust those timers and limits, though. 

Look at the server error/access logs.

again....logs, logs, logs....I'll setup gunicorn locally and enable logging.

For someone wanting to move to a file chunk upload, I've tried resumable.js and a django component for it. it was failing. Now I'm happier with this other: https://github.com/juliomalegria/django-chunked-upload-demo (test over wifi ave worked fine). But again.....maybe the problem is in our setup and not in those tools.

Thanks, James, you've been vary helpful.

-James