large file upload

[JP]

Hello all!

I am trying to implement a simple frontend for ffmpeg tests and find
problems due to socket errors which have been discussed previously on
this group. Unfortunately, I was not able to use any of the
suggestions and Ulrik's post (http://groups.google.com/group/webpy/
browse_thread/thread/507bd537be623fb7/9932b4e7e2c3c000?
lnk=gst&q=ulrik#9932b4e7e2c3c000) remained unanswered...

I based the following code on a Django fix addressing the same issue
on that platform.

class upload:
def POST(self):
asset = web.input(asset={}).asset
video = open(os.path.join('./static',
'in'+os.path.basename(os.path.splitext(asset.filename)[1])), 'wb')
len = web.intget(web.ctx.env.get('CONTENT_LENGTH'), 0)
while True:
r = len - video.tell()
if r <= 0: break
chunk = web.ctx.env['wsgi.input'].read(min(2048L, r))
if not chunk: break
video.write(chunk)
video.close()
# os.system("ffmpeg -y -i %s %s" % (video.name, 'static/out.flv'))
web.redirect(web.ctx.home+'/view')

Unfortunately, the code above still fails:
[...]
chunk = web.ctx.env['wsgi.input'].read(min(2048L, r))
File "/tmp/python.6884/usr/lib/python2.5/socket.py", line 309, in
read
data = self._sock.recv(recv_size)
timeout: timed out

Any suggestions?

TIA

- JP

[jpscaletti]

I solve the large file uploading problem using this,
It's kind of ugly, but it works:

1. To send the large file use the <input type="file"... as the only
field in the form
2. You can use the url to pass other fields

class upload:
def POST(self):
from socket import error as SckError
from os import unlink

inbox = web.utils.intget(web.ctx.env.get('CONTENT_LENGTH'), 0)
outbox = 0
f = None

try:
if inbox > MAXSIZE:
raise UploadError('maxsize') # Custom exception

filename = None
mimetype = None
parsedHeader = False

fullpath = getFullPath() # External function.
f = open(fullpath, 'wb')
try:
while inbox > 0:
chunk = 131072 if inbox > 131072 else inbox
#131072 = 1024*128 = 128 KB
data = web.ctx.env['wsgi.input'].read(chunk)

if not parsedHeader:
match = re.search("filename=\"([^\"]+)\"",
data)
if match:
filename = match.group(1)
end = match.end()+2
match = re.search("Content-Type:\s(\S+)",
data)
if match:
mimetype = match.group(1)
end = match.end()+4
parsedHeader = True
#Total length - header
data = data[end:]
inbox -= end
globals()['FS'][fid].size = inbox

l = len(data)
inbox -= l
# Removing the data tail
if inbox == 0:
data = data[:data.rfind('-'*29)-2]
l = len(data)

outbox += l
f.write(data)
f.flush()
sleep(0.01)
finally:
f.close()

result = processFile(fullpath, filename, mimetype,
outbox) # External function

except (UploadError, SckError), e:
if f:
unlink(fullpath)


Good luck.

[jpscaletti]

The "sleep(0.01)" is intended to leave time to another function to
work.
That function could be called using AJAX during the uploading and
return its progress in order to display in the page a nice progress
bar.

On 12 mar, 18:01, jpscaletti <juanpablo.scale...@gmail.com> wrote:
> ...
> f.flush()
> sleep(0.01)
> finally:
> f.close()

JPS