Re: Problem with uploading large file web2py version 2.3.2

[Phyo Arkar]

How much Ram it takes.

Wont take alot more than 400mb i believe.

On Monday, March 4, 2013 2:40:12 PM UTC+6:30, Mạnh Trần Đức wrote:

Dear everyone,

I uploaded file about <100mb successfully but it got "Exceptions.MemoryError" when I uploaded a file ~400mb.

I tried upload to ftp with pyfilesystem but got that error too

I saw  in Windows Task Manager, the Python progress  takes  too much ram and makes system be frozen. 

This is my system that I running web2py:

Windows 7 pro 64bit 4gb ram

python 2.7.3 64bit

Web2py: 2.3.2

[Mạnh Trần Đức]

When I upload a ~400Mb file:

Firstly, After Chrome show "upload 100%" , the python progress jump from 50 -. 400 MB

Then after few senconds, it jumps to max of RAM (Step by step, 1GB, 1.5GB,   ... 2.5 GB) then my PC hang (Fully of RAM)

[Massimo Di Pierro]

Can you show us your code? Web2py is transparent to the upload and does not stores the uploaded file in ram but your app may do it and that may be a problem.

[Mạnh Trần Đức]

Hi,this is my db:

########################################

db.define_table('clsb_product',

    Field('product_category', type = 'reference clsb_category', notnull = True,

          label = T('Category')),

    Field('product_type', type = 'reference clsb_product_type', notnull = True,

          label = T('Product Type')),

    Field('product_relation', type = 'reference clsb_relation', notnull = True,

          label = T('Product Relation')),

    Field('product_creator', type = 'reference clsb_dic_creator', notnull = True,

          label = T('Product Creator')),

    Field('product_publisher', type = 'reference clsb_dic_publisher', notnull = True,

          label = T('Product Publisher')),

    Field('product_title', type = 'string', notnull = True,

          label = T('Product Title')),

    Field('product_code', type = 'string', notnull = True, unique = True,

          label = T('Product Code')),

    Field('total_file', type = 'integer',

          label = T('Total File')),

    Field('product_cover', type = 'upload', requires = IS_IMAGE(extensions = 'png'), notnull = True,

          label = T('Cover Image')),

    Field('product_data', type = 'upload', requires = IS_UPLOAD_FILENAME(extension = 'zip'), notnull = True,

          label = T('Product Data')),

    auth.signature,

    format = '%(product_title)s')

this is function in controller:

########################################

def manager():

    if request.args:

        table = 'clsb_' + request.args(0)

        if not table in db.tables(): redirect(URL('error'))

        form = SQLFORM.grid(db[table], args = request.args[:1],

                            onupdate = auth.archive,

                            showbuttontext = False,

                            user_signature = False)

        return dict(form = form)

    return None

Thank everyone for repling me :-)

[Massimo Di Pierro]

I cannot reproduce the problem.

It would help if you could isolate it. Do you see the memory increase during upload? After upload? During validation? After validation? When the form is successfully uploaded or only when there is error in the form? Can you reproduce it if there is only one field?

massimo

[Ricardo Pedroso]

On Sat, Mar 9, 2013 at 4:27 AM, Massimo Di Pierro
<massimo....@gmail.com> wrote:
> I cannot reproduce the problem.

I was able to reproduce it with web2py
Version 2.4.2-stable+timestamp.2013.03.09.00.16.49

I attach a small sample that demonstrate the issue.
Just drop the attachment in controllers in an empty
application. it does not need any view and db connection
and table definition is in it.

> It would help if you could isolate it. Do you see the memory increase during
> upload? After upload? During validation? After validation? When the form is
> successfully uploaded or only when there is error in the form? Can you
> reproduce it if there is only one field?
>

The problem is with SQLFORM.grid.

I track it down to:

- sqlhtml.py (1817) in def url
`- sqlhtml.py (1824) when calling URL(**b)
`- html.py (357) when calling urllib.urlencode(list_vars)

list_vars contains the FieldStorage. So I guess the content of the upload
is used in urlencode.

The call to url() is the culprit and it's called in two places in sqlhtml.py:

line 1835:
referrer = session.get('_web2py_grid_referrer_' + formname, url()

line 1850:
def gridbutton(buttonclass='buttonadd', buttontext=T('Add'),
buttonurl=url(args=[]), callback=None,
delete=None, trap=True)


Ricardo

> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "web2py-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to web2py+un...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Massimo Di Pierro]

Thank you Ricardo. This helps a lot. I will fix it asap.

massimo

[Massimo Di Pierro]

Your post very very helpful. Indeed the problem is problem being passed to the URL function by the grid. I believe this is now fixed. Can you help me check it?

Massimo

[Massimo Di Pierro]

This was indeed a problem with grid (not uploaded) as suggested by Roberto. Can you please check it is fixed in trunk?

[Ricardo Pedroso]

On Sun, Mar 10, 2013 at 2:21 PM, Massimo Di Pierro
<massimo....@gmail.com> wrote:
> This was indeed a problem with grid (not uploaded) as suggested by Roberto.
> Can you please check it is fixed in trunk?

I did the same test as before and now I don't see the memory and CPU
spikes, so it's fixed.

Massimo, who is Roberto?

If I'm not wrong you are referring to me and I'm Ricardo ;)

Both names have 7 letters and starts/ends with same letters,
remind me http://en.wikipedia.org/wiki/Typoglycemia


Regards,
Ricardo

[Niphlod]

Massimo, who is Roberto?

If I'm not wrong you are referring to me and I'm Ricardo ;)

Both names have 7 letters and starts/ends with same letters,
remind me http://en.wikipedia.org/wiki/Typoglycemia

 

lol. I'm niphlod, niphold, niphodl , etc ..... commits messages ala "autocowrecks" makes my day brighter :P 

[Massimo Di Pierro]

I am all red. :-(

Anyway, I got your name correct in the commit. I should definitively know better. You have an italian name.

Massimo

[Massimo Di Pierro]

I have a script to automatically group all your aliases when generating the book acknowledgements. ;-)