uploading big files (more than 100Mb)

Farsheed

unread,

Feb 10, 2009, 8:04:36 AM2/10/09

to cherrypy-users

I use this code to upload:
#========Code Start==========
class NoteServer(object):
_cp_config = { 'tools.sessions.on': True }

def index(self):
return """
<html><body>
<form action="upload" method="post" enctype="multipart/
form-data">
filename: <input type="file" name="myFile" /><br />
<input type="submit" />
</form>
<p>
Pars Studios, 2009
</p>
</body></html>
"""
#~ return compose().run()
index.exposed = True
def upload(self, myFile):
out = """<html>
<body>
myFile length: %s<br />
myFile filename: %s<br />
myFile mime-type: %s
</body>
</html>"""

# Although this just counts the file length, it demonstrates
# how to read large files in chunks instead of all at once.
# CherryPy uses Python's cgi module to read the uploaded file
# into a temporary file; myFile.file.read reads from that.
size = 0
allData=''
while True:
data = myFile.file.read(8192)
allData+=data
if not data:
break
size += len(data)
savedFile=open(myFile.filename, 'wb')
savedFile.write(allData)
savedFile.close()
return out % (size, myFile.filename, myFile.type)
#~ return allData
upload.exposed = True
#========Code End============

But I couldn't upload files bigger than 100Mb.
Why and what is workaround?

Thanks in advanced.

Eric Abrahamsen

unread,

Feb 10, 2009, 10:47:51 AM2/10/09

to cherryp...@googlegroups.com

I can't tell you exactly what's going wrong here, but you can try
setting the config value "server.max_request_body_size" to something
larger for this particular handler, and that might help. I've got a
handler that does large uploads, and I'm using a separate thread to do
the copy to disk, so that the handler itself can return a little
earlier and avoid timeouts. That might also be something to consider.
I limit uploads at around 300MB, and it works fine (though it can be
slow). Here's the threaded copier:

import threading
import shutil

class UploadCopyThread(threading.Thread):
"""
This moves upload copying out of the request thread,
reducing the wait time for very large uploads.
"""
def __init__(self, file_contents, filename):
self.cont = file_contents
self.fname = filename
threading.Thread.__init__(self)

def run(self):
outfile = open(self.fname, 'wb')
shutil.copyfileobj(self.cont, outfile)

Eric

>
>
> Thanks in advanced.
>
> >

Robert Brewer

unread,

Feb 10, 2009, 2:04:00 PM2/10/09

to cherryp...@googlegroups.com

Eric Abrahamsen wrote:
> On Feb 10, 2009, at 9:04 PM, Farsheed wrote:
> > But I couldn't upload files bigger than 100Mb.
> > Why and what is workaround?
>
> I can't tell you exactly what's going wrong here, but you can try
> setting the config value "server.max_request_body_size" to something
> larger for this particular handler, and that might help.

Yes, with a "but". The default maximum body size is indeed 100MB:

>>> import cherrypy
>>> cherrypy.server.max_request_body_size
104857600
>>> _ / (1024 ** 2)
100

But: you can't specify that per handler. It's per-server. If you wanted
different per-handler limits, you'd either have to make your own
subclass of cgi.FieldStorage and override its read_* methods, or wrap
cherrypy.request.rfile in a SizeCheckWrapper (like wsgiserver does)
yourself. If you know you're only ever going to be using wsgiserver, a
hacky way would be to just change rfile.maxlen inside your handler, and
set request.process_request_body to False. I just added an example of
that in http://www.cherrypy.org/changeset/2119.

Robert Brewer
fuma...@aminus.org

Eric Abrahamsen

unread,

Feb 10, 2009, 9:37:18 PM2/10/09

to cherryp...@googlegroups.com

On Feb 11, 2009, at 3:04 AM, Robert Brewer wrote:

>
> Eric Abrahamsen wrote:
>> On Feb 10, 2009, at 9:04 PM, Farsheed wrote:
>>> But I couldn't upload files bigger than 100Mb.
>>> Why and what is workaround?
>>
>> I can't tell you exactly what's going wrong here, but you can try
>> setting the config value "server.max_request_body_size" to something
>> larger for this particular handler, and that might help.
>
> Yes, with a "but". The default maximum body size is indeed 100MB:
>
>>>> import cherrypy
>>>> cherrypy.server.max_request_body_size
> 104857600
>>>> _ / (1024 ** 2)
> 100
>
> But: you can't specify that per handler. It's per-server.

Of course! I don't know why I wrote that.

> If you wanted different per-handler limits, you'd either have to
> make your own
> subclass of cgi.FieldStorage and override its read_* methods, or wrap
> cherrypy.request.rfile in a SizeCheckWrapper (like wsgiserver does)
> yourself. If you know you're only ever going to be using wsgiserver, a
> hacky way would be to just change rfile.maxlen inside your handler,
> and
> set request.process_request_body to False. I just added an example of
> that in http://www.cherrypy.org/changeset/2119.

This is very interesting, and useful for some other stuff I'm doing,
thanks!

>
>
> Robert Brewer
> fuma...@aminus.org
>
>
> >

Reply all

Reply to author

Forward