web2py, nginx and big files

[Carlos Cesar Caballero Díaz]

Hi, I have a web2py + nginx setup
(https://github.com/arisobel/web2py_scripts/blob/master/setup-web2py-nginx-p3-uwsgi-ubuntu.sh)
and I need to upload big files to my application, the app is working ok
with the built in server, but with nginx the form submission fails with
no response when there are big files.

Any tip for solving this?


Greetings.

[Richard Vézina]

How big?

https://www.bookstackapp.com/docs/admin/upload-limits/

https://www.nginx.com/resources/wiki/modules/upload/#upload-max-file-size

Quote from the above link: 

For “hard” limit client_max_body_size directive must be used. The value of zero for this directive specifies that no restrictions on file size should be applied.


Have a look at your nginx config upload_max_file_size and client_max_body_size

I can't find the default nginx limit, but I think upload are around 5 mg, I recall having issue long time ago about upload limit and the attachement was a little over 5 meg, so...

In the link you provide there not seems to have limit overriding default, so my guess is that your file is bigger than the client_max_body_size

Good luck

Richard

--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- You received this message because you are subscribed to the Google Groups "web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Carlos Cesar Caballero Díaz]

Thanks Richard, It seems to be working now. The files can be several Gigabytes.

Greetings.

El 17/04/18 a las 13:10, Richard Vézina escribió:

To unsubscribe from this group and stop receiving emails from it, send an email to web2py+un...@googlegroups.com.

[Richard Vézina]

That big... Hope it could work for you...

Are you trying to make El packet transiting the island faster or what...

:D

Richard

On Wed, Apr 18, 2018 at 1:45 PM, Carlos Cesar Caballero Díaz <carlos.c...@cfg.jovenclub.cu> wrote:

Thanks Richard, It seems to be working now. The files can be several Gigabytes.

Greetings.

El 17/04/18 a las 13:10, Richard Vézina escribió:

How big?

https://www.bookstackapp.com/docs/admin/upload-limits/

https://www.nginx.com/resources/wiki/modules/upload/#upload-max-file-size

Quote from the above link: 

For “hard” limit client_max_body_size directive must be used. The value of zero for this directive specifies that no restrictions on file size should be applied.


Have a look at your nginx config upload_max_file_size and client_max_body_size

I can't find the default nginx limit, but I think upload are around 5 mg, I recall having issue long time ago about upload limit and the attachement was a little over 5 meg, so...

In the link you provide there not seems to have limit overriding default, so my guess is that your file is bigger than the client_max_body_size

Good luck

Richard

[Carlos Cesar Caballero Díaz]

El 18/04/18 a las 13:54, Richard Vézina escribió:

That big... Hope it could work for you...

Well, right now is working with your suggestion, but not perfect, is really slow because nginx uploads the file to a tmp file, next seems to copy the file to another place (I assume to uwsgi) and next is copied to web2py uploads, really annoying. I think that there should be a better way to do that.

Are you trying to make El packet transiting the island faster or what...

:D

Not exactly, but I like how you think :D

I am working with Joven Club, a "company" that provides technical and learning services throughout the country, and I am building a tool for helping with the software installation service, it's really simple, basically a software installers repository where people can search and download (and certain people upload the files, of course).

To unsubscribe from this group and stop receiving emails from it, send an email to web2py+un...@googlegroups.com.

[Richard Vézina]

Ah ok, make sens then. Are you storing the file in database or on file system at web2py level??

There is surely a way to make the copy only once at the right place, but I am pretty sure it mean writting some customization at uwsgi level and most probably some conf in nginx to bypass it usual mechanism.

I try to make some research and get back to you if I can fin anything that could help.

Good luck

Richard

On Wed, Apr 18, 2018 at 10:37 PM, Carlos Cesar Caballero Díaz <carlos.c...@cfg.jovenclub.cu> wrote:

El 18/04/18 a las 13:54, Richard Vézina escribió:

That big... Hope it could work for you...

Well, right now is working with your suggestion, but not perfect, is really slow because nginx uploads the file to a tmp file, next seems to copy the file to another place (I assume to uwsgi) and next is copied to web2py uploads, really annoying. I think that there should be a better way to do that.

Are you trying to make El packet transiting the island faster or what...

:D

Not exactly, but I like how you think :D

I am working with Joven Club, a "company" that provides technical and learning services throughout the country, and I am building a tool for helping with the software installation service, it's really simple, basically a software installers repository where people can search and download (and certain people upload the files, of course).

Richard

[Carlos Cesar Caballero Díaz]

El 19/04/18 a las 16:21, Richard Vézina escribió:

Ah ok, make sens then. Are you storing the file in database or on file system at web2py level??

In the file system, I think that storing big (or so many) files in the database can overload it.

There is surely a way to make the copy only once at the right place, but I am pretty sure it mean writting some customization at uwsgi level and most probably some conf in nginx to bypass it usual mechanism.

I have being reading and found this https://stackoverflow.com/a/44751210/1818267 its interesting, besides it carries some new issues to solve.

I try to make some research and get back to you if I can fin anything that could help.

Good luck

Thanks, I'll be very grateful (more grateful :D).

Greetings

To unsubscribe from this group and stop receiving emails from it, send an email to web2py+un...@googlegroups.com.

[Richard Vézina]

You seems to have found good information...

I don't have much time to investigate that subject and realize it could be quite complexe...

And involve security issue : https://www.nginx.com/resources/wiki/start/topics/tutorials/config_pitfalls/

Although, I thought that you might have a read that code base of apps like these listing :

https://opensource.com/life/15/12/5-open-source-web-apps-self-hosted

This kind of apps surely had to workaround large file issue that you have...

Richard

On Fri, Apr 20, 2018 at 10:39 AM, Carlos Cesar Caballero Díaz <carlos.c...@cfg.jovenclub.cu> wrote:

El 19/04/18 a las 16:21, Richard Vézina escribió:

Ah ok, make sens then. Are you storing the file in database or on file system at web2py level??

In the file system, I think that storing big (or so many) files in the database can overload it.

There is surely a way to make the copy only once at the right place, but I am pretty sure it mean writting some customization at uwsgi level and most probably some conf in nginx to bypass it usual mechanism.

I have being reading and found this https://stackoverflow.com/a/44751210/1818267 its interesting, besides it carries some new issues to solve.

I try to make some research and get back to you if I can fin anything that could help.

Good luck

Thanks, I'll be very grateful (more grateful :D).

Greetings

Richard

[Carlos Cesar Caballero Díaz]

Thanks Richard, I will take a look.

Greetings.

El 20/04/18 a las 14:56, Richard Vézina escribió:

To unsubscribe from this group and stop receiving emails from it, send an email to web2py+un...@googlegroups.com.