A simple patch to provide upload progress

[AndCycle]

after read over this "thread [web2py:23034] Upload progress"
http://www.mail-archive.com/web...@googlegroups.com/msg13111.html

I have looks around how other web framework done this,

in pylons's way, this require a WSGI middleware and URLmap,
unfortunately URLmap is impossible for web2py,
and wrap middleware like "gp.fileupload" or insert filer in cherrypy
also require lot's of hack,

in django's way, it hooks handler at input chain, which web2py doesn't
have,
but it just store the upload progress in cache, which is possible for
web2py,
so I made the patch to proof this works

I just made a proof of concept to provide upload progress,

part of code and idea taken from django snippets with some
modification
http://www.djangosnippets.org/snippets/678
http://www.djangosnippets.org/snippets/679/

here is the major part

--------------------------------------------------------------------------------

--- web2py-orig/main.py 2009-09-22 09:07:52.000000000 +0800
+++ web2py/main.py 2009-10-07 16:07:55.000000000 +0800
@@ -335,6 +335,51 @@

if request.env.content_length:
request.body = tempfile.TemporaryFile()
+ def copystream(
+ src,
+ dest,
+ size,
+ chunk_size=10 ** 5,
+ ):
+ """
+ original from fileutils,
+ this is a hacking for progress upload status,
+ """
+ import time
+ progressCache = True
+ try:
+ cache_key = 'X-Progress-ID:'+request.get_vars
['X-Progress-ID']
+ except KeyError:
+ progressCache = False
+ if progressCache:
+ from cache import Cache
+ cache = Cache(request)
+ cache.ram(cache_key+':length', lambda: size,
0)
+ cache.ram(cache_key+':uploaded', lambda: 0,
0)
+ while size > 0:
+ if size < chunk_size:
+ data = src.read(size)
+ if progressCache:
+ cache.ram.increment(cache_key
+':uploaded', size)
+ else:
+ data = src.read(chunk_size)
+ if progressCache:
+ cache.ram.increment(cache_key
+':uploaded', chunk_size)
+ length = len(data)
+ if length > size:
+ (data, length) = (data[:size], size)
+ size -= length
+ if length == 0:
+ break
+ dest.write(data)
+ if length < chunk_size:
+ break
+ dest.seek(0)
+ if progressCache:
+ cache.ram(cache_key+':length', None)
+ cache.ram(cache_key+':uploaded', None)
+ return
copystream(request.env.wsgi_input, request.body,
int(request.env.content_length))
else:

--------------------------------------------------------------------------------
maybe just wrap wsgi_input will make it pretty,

next is how to use it,

at controller, just require a function to respond json
--------------------------------------------------------------------------------
def post():
#I need a from, give me give me
return dict(form=crud.create(db.foo_table))

@service.jsonrpc
def progress_report_by_id():
cache_key = 'X-Progress-ID:'+request.get_vars['X-Progress-ID']
return dict(
length=cache.ram(cache_key+':length', lambda: 0, None),
uploaded=cache.ram(cache_key+':uploaded', lambda: 0,
None),
)
--------------------------------------------------------------------------------


at view you can do whatever you want,
this one is post.html for post function up there,

or just take javascript snippet part,

watch out "var progress_url",
you have to modify this to match your app/controller/function
somewhere
--------------------------------------------------------------------------------
{{extend 'layout.html'}}
<script type="text/javascript">
// Generate 32 char random uuid
function gen_uuid() {
var uuid = ""
for (var i=0; i < 32; i++) {
uuid += Math.floor(Math.random() * 16).toString(16);
}
return uuid
}

// Add upload progress for multipart forms.
$(function() {
$('form[enctype=multipart/form-data]').submit(function(){
// Prevent multiple submits
if ($.data(this, 'submitted')) return false;

var freq = 1000; // freqency of update in ms
var uuid = gen_uuid(); // id for this upload so we can fetch
progress info.
var progress_url = /poc/default/
progress_report_by_id.json'; // ajax view serving progress info

// Append X-Progress-ID uuid form action
this.action += (this.action.indexOf('?') == -1 ? '?' : '&') +
'X-Progress-ID=' + uuid;

var $progress = $('<div id="upload-progress" class="upload-
progress"></div>').
insertAfter($('input[type=submit]')).append('<div
class="progress-container"><span class="progress-info">uploading 0%</
span><div class="progress-bar"></div></div>');

$('input[type=submit]').remove()
// progress bar position
/*
$progress.css({
position: ($.browser.msie && $.browser.version < 7 )?
'absolute' : 'fixed',
left: '50%', marginLeft: 0-($progress.width()/2), bottom:
'20%'
}).show();
*/

$progress.find('.progress-bar').height('1em').width(0).css
("background-color", "red");
// Update progress bar
function update_progress_info() {
$progress.show();
$.getJSON(progress_url, {'X-Progress-ID': uuid}, function
(data, status){
if (data) {
var progress = parseInt(data.uploaded) / parseInt
(data.length);
var width = $progress.find('.progress-
container').width()
var progress_width = width * progress;
$progress.find('.progress-bar').width
(progress_width);
$progress.find('.progress-info').text('uploading '
+ progress*100 + '%');
}
window.setTimeout(update_progress_info, freq);
});
};
window.setTimeout(update_progress_info, freq);

$.data(this, 'submitted', true); // mark form as submitted.
});
});
</script>
{{=BEAUTIFY(response._vars)}}
--------------------------------------------------------------------------------

that's all, hope one developer can pretty this implementation and
migrate it into the core and widget :)

[AndCycle]

oops, the patch been cut by width restriction on mailing list lol

here is pastebin
http://pastebin.com/f219d603b

[mdipierro]

Check again. I added it now.

[AndCycle]

thanks :)

I will take some free time to write a example base on trunk :p

[mdipierro]

@AndCycle

I rearranged the code in trunk a bit. Could you check it still works?
Thanks again.

Massimo

[AndCycle]

@Massiom

http://pastebin.com/f69e7b008

here is the patch include a working sample,
by the way you got another indent error in copystream_progress :p

[mdipierro]

Would you post a web2pyslice on this?

Massimo

[Yarko Tymciurak]

just to complete the link (I didn't immediately find web2pyslice searching the group): 
http://www.web2pyslices.com/main/default/index

[mdipierro]

I am going to link it soon. I will take suggestions on the best way to
link it form the main page.

On Oct 11, 12:00 pm, Yarko Tymciurak <yark...@gmail.com> wrote:
> just to complete the link (I didn't immediately find web2pyslice searching
> the group):http://www.web2pyslices.com/main/default/index
>

[mr.freeze]

I may have mentioned this already, but you could create an 'Other
Resources' section under the Documentation page at web2py.com for
unofficial docs/sites/wiki posts.

[mdipierro]

Yes but I think web2pyslices and the new wiki are as official as it
gets so there should also be a link from the main page.

[Alex Fanjul]

web2pyslices its getting very cool indeed... i like the ¿new? user
interface..
Alex F

--
Alejandro Fanjul Fdez.
alex....@gmail.com
www.mhproject.org

[mr.freeze]

Thanks! I am going through a code cleanup but will be posting the
source when done for anyone interested.

> alex.fan...@gmail.comwww.mhproject.org

[Yarko Tymciurak]

so glad to hear it - it is a really nice job!   (consider have a "code review" at post, and posting to google code in mercurial to enable such).

- Yarko

[AndCycle]

later, I am trying to keep my job lol

[AndCycle]

by the way,
the validation widget also require a client side javascript to do
their job,
it's really piss user off when the system raise a error after they
uploaded hundreds of mega,

[mr.freeze]

Much obliged. I have yet to tackle mercurial but it's on my list. 9
entries for web2pyslices, 1729 for djangosnippets. I'd better get
moving!

[mdipierro]

Yes but:
1) pyslices does not yet include alterego entries and there is about
200 of them
2) Django needs 1729, do we? I hope not.

Massimo

[Richard]

hello,

I noticed copystream_progress() in trunk. Is there an example how to
use it?
Richard

[mdipierro]

I remember the author of the patch posted an example but I cannot find
it. I hope he will step in and point us to it.

Massimo

[Richard]

is this it? http://pastebin.com/f69e7b008

[mdipierro]

yes. Thank you! We should post a web2pyslice about this.

On Oct 14, 11:50 pm, Richard <richar...@gmail.com> wrote:
> is this it?http://pastebin.com/f69e7b008

[AndCycle]

would you? :p

[AndCycle]

http://www.web2pyslices.com/main/slices/take_slice/10

anyway, I have done the sample, any comment? :~

[mr.freeze]

Looks great!

[Richard]

thanks for putting that together!
I tried it out with latest trunk but found that the returned json was
always {'length':0, 'uploaded':0}
Any ideas? I didn't change anything.

In case it matters I am using the builtin web2py server with sqlite on
Ubuntu 9.04.

Also is post.json necessary? It will use generic.json by default.

Richard

[AndCycle]

sorry, I currently don't have time to help you :X

http://www.andcycle.idv.tw/~andcycle/tmp/web2py-rev1289.7z
here is my trunk with example at
web2py\applications\examples\controllers\upload_progress_examples.py

help yourself,

just tell me if this works for you

[mdipierro]

does upload work? This assumes cache.ram is working so it would not
work with CGI.

Massimo

[Richard]

No, nothing is uploaded. The progress bar shows NaN (divide by zero
error), then the page submits but the file is not uploaded.

I'll check the uploaded example - thanks for that.
Richard

[Richard]

I got the same problem with the example from AndCycle, using the
builtin web2py server.
Can anyone else get it working? If so what setup do you have?

Richard


On Oct 20, 3:40 pm, Richard <richar...@gmail.com> wrote:
> No, nothing is uploaded. Theprogressbar shows NaN (divide by zero

> error), then the page submits but the file is not uploaded.
>
> I'll check the uploaded example - thanks for that.
> Richard
>
> On Oct 20, 6:32 am, mdipierro <mdipie...@cs.depaul.edu> wrote:
>

> > doesuploadwork? This assumes cache.ram is working so it would not

[AndCycle]

have you tried print debug lol

put some print in gluon/main.py copy_steram func to verify it received
X-Progress-ID,
and see is it doing increment for cache?

next checkout controller if it received X-Progress-ID too,
if it get X-Progress-ID, what it extract from cache?

if the cache is always zero, well, that's tricky,
you may have to hack gluon/cache.py to see what's happened

my environment is Py 2.6.3, under Vista x64 sp2,
I have installed pywin32 for portalock in web2py,
and use fx 3.5 for development

[AndCycle]

oops, there must be something breaked, I can't it work too,

I will reply after fix it, sorry :(

[AndCycle]

ok, Richard, are you trying to upload a really big file that over 2GB?
try smaller one like 100MB,
I am investigating where is the limit

[AndCycle]

found a article about this,
http://www.motobit.com/help/scptutl/pa98.htm
it's the limit applied on browser lol

so if you use FireFox with HttpFox you can see the post sends nothing
if file bigger than 2GB

[Richard]

my file is just a few MB. That's interesting about the browser
limitation...

Has anyone got this to work on Linux?

As you suggest, I'll try debugging deeper into gluon to see what is
going on.

Richard


On Oct 22, 8:30 pm, AndCycle <andcycle-goo...@andcycle.idv.tw> wrote:
> found a article about this,http://www.motobit.com/help/scptutl/pa98.htm

[Richard]

From debugging gluon/main.py I found out the problem - me!
The upload was so fast that it had finished before the first progress
bar call. At the end of the upload the cache is cleared, so that the
first progress bar call returned an undefined cache.

Does the cache need to be cleared, or is it better to update the
controller/view to avoid NaN?

Richard

On Oct 22, 11:52 pm, Richard <richar...@gmail.com> wrote:
> my file is just a few MB. That's interesting about the browser
> limitation...
>
> Has anyone got this to work on Linux?
>
> As you suggest, I'll try debugging deeper into gluon to see what is
> going on.
>
> Richard
>
> On Oct 22, 8:30 pm, AndCycle <andcycle-goo...@andcycle.idv.tw> wrote:
>
> > found a article about this,http://www.motobit.com/help/scptutl/pa98.htm
> > it's the limit applied on browser lol
>
> > so if you use FireFox with HttpFox you can see the post sends nothing
> > if file bigger than 2GB
>
> > On Oct 22, 5:24 pm, AndCycle <andcycle-goo...@andcycle.idv.tw> wrote:
>

> > > ok, Richard, are you trying touploada really big file that over 2GB?

[AndCycle]

lol

update the view is better, and maybe the only solution,
current cache implement have no direct way to check if the cache do
exist or not,

so you have write some js for this :p

[stefan]

Hi all,

I've got the upload progress bar working based on the example on
web2pyslices, but it only works with web2py in standalone mode. I
can't get it work when I use web2py with apache via mod_wsgi. The
lookup in the cache always returns 0 then.

Has anyone tried this (successfully)?

Best regards,
Stefan

[AndCycle]

well, I think there are so many solutions to do this, but I don't have
a perfect solution for this,

this simplest one might be limit only one process running for web2py,
like
WSGIDaemonProcess myweb2py processes=1

I know this is a silly solution,
or you have to hack relative source code not to use cache.ram to do
the counting,
use cache.file might work, but this result in file locking issue, this
hurt performance,
do counting in sqlite? this really hurt performance too,

any idea is welcome

On Dec 22 2009, 9:58 pm, stefan <stefan.meier.k...@googlemail.com>
wrote: