Progress bars and session

[Alfonso Serra]

Im trying to make a progress bar by reading and writing a session variable:

There are 2 controllers:

Get session.progress

def getprogress():
    print "getprogress:", session.progress
    return session.progress

This one takes 5 seconds to perform and changes session.progress each time.


def progress():
    # when a form is submitted, change progress each second
    if request.post_vars:
        for i in range(5):
            session.progress = i
            print "main:", session.progress
            sleep(1)
        return "done"
    else:
        session.progress = -1
    return locals()

At the view, there are two ajax calls. One that triggers "getprogress" each second, and another one that triggers "progress" when a random form is submitted through ajax to avoid redirection.

at the browser's console, getprogress only returns -1

at the python console i get a log like this:

getprogress: -1
getprogress: -1
main: 0
getprogress: -1
main: 1
getprogress: -1
main: 2
getprogress: -1
main: 3
getprogress: -1
main: 4
getprogress: -1

getprogress should be changing since im writing the session variable.

If im changing a session variable in a controller why hasnt changed when i ask for it on another one?

Should i be using cron jobs or threads to make a progress bar? Whats the best way?

Thank you very much.

[Anthony]

During a single request, the "session" object is just a copy of the session data stored in the session file. Any updates to the session during a request are not written back out to the session file until after the request has completed. So, in the progress() function, you are not updating the session file itself every second, you are only updating that request's local copy of the session data. Repeated calls to getprogress() will therefore only see the original value from the session.

Anthony

On Tuesday, February 16, 2016 at 4:15:06 PM UTC-5, Alfonso Serra wrote:

Im trying to make a progress bar by reading and writing a session variable:

There are 2 controllers:

Get session.progress

def getprogress():
    print "getprogress:", session.progress
    return session.progress

This one takes 5 seconds to perform and changes session.progress each one.

[Anthony]

Here's a hack that might work:

def progress():
    # when a form is submitted, change progress each second
    if request.post_vars:
        for i in range(5):
            session.progress =

i
            # Write to the session file and unlock/close it.
            session._try_store_in_file(request, response)
            print "main:", session.progress
            sleep(1)
            # Re-open and re-lock the session file.
            session.connect()
        return "done"

    else:
        session.progress = -1
    return locals()

Note, this is not part of the public API and writing/closing/reconnecting to the session multiple times per request is not an officially supported feature.

Also, note that it is important to leave the session file unlocked while the processing is happening (i.e., don't re-connect to the session until you have to), as other requests (including your getprogress requests) will be prevented from proceeding while the session is locked.

Anthony

[Alfonso Serra]

Thanks Anthony, it does work.

The hack is easy to implement, but to represent the insertion of 30k records into the database, writing the session file and connecting 30k times might not be optimal.

Theres a chapter about the Scheduler that may work for this. Ill read about it, it has something to report percentages 

Does anyone knows any resource to implement a progress bar in web2py? (not just for uploading files)

Thanks
Best regards.

[Carlos Correia]

On 17-02-2016 00:17, Alfonso Serra wrote:
> Thanks Anthony, it does work.
>
> The hack is easy to implement, but to represent the insertion of 30k records
> into the database, writing the session file and connecting 30k times might not
> be optimal.
>

Perhaps you should update a field in some db table and use it instead of session.

Carlos Correia
=========================
MEMÓRIA PERSISTENTE
Tel.: 218 471 841 (NOVO)
GSM: 917 157 146
e-mail: ge...@memoriapersistente.pt
URL: http://www.memoriapersistente.pt
XMPP (Jabber): car...@memoriapersistente.pt (NOVO)
GnuPG: wwwkeys.eu.pgp.net
URL Suporte: https://t5.m16e.com/gps

[Alfonso Serra]

Perhaps you should update a field in some db table and use it instead of session.

Please note that the end of this, is to create an ajax progress bar that indicates the numbers of records being inserted on a request.

Although that might work as well, still it isnt good to read and write a record each second (progress update interval) while im performing the insertion of 30k records.

The optimal way to solve this is, as your company name indicates, is persistent memory, so the session was my first choice until Anthony said that it has to be written to disk. And either writing to disk or db isnt good.

web2py Scheduler is a candidate to solve a problem like this but im not sure yet.
Check the progress of a background job, might do the trick or caching a variable in ram (althought the variable has to be updated frequently) or even a temporary memory table in the db.

Thanks.

[Anthony]

Yes, I would say cache.ram might be a better approach. Also, keep in mind that although you might want to check every second or two, you don't have to update the progress variable after every single record insertion. For example, for a precision of 1%, just update the progress after every 300 insertions.

Anthony

On Wednesday, February 17, 2016 at 10:06:33 AM UTC-5, Alfonso Serra wrote:

Perhaps you should update a field in some db table and use it instead of session.

Please note that the end of this, is to create an ajax progress bar that indicates the numbers of records being inserted on a request.

Although that might work as well, still it isnt good to read and write a record each second (progress update interval) when im performing the insertion of 30k records.

[Alfonso Serra]

Ive adapted the code to use cache.ram, but the docs about this mechanism are quite confusing, and i can't get it to work.

There are several questions like,
how to store simple values instead callables?
what happens if i access a cache value after expiration?

This is the code:

def getprogress():
    print "progress", cache.ram.storage['message'][1]
    return cache.ram.storage['message'][1]

def progress():
    # when a form is submitted, change progress each second
    if request.post_vars:
        for i in range(5):

            message = cache.ram('message', lambda: i, time_expire=1)
            print "main:", message
            sleep(1)
        return "done"
    else:
        message = cache.ram('message', lambda: -1, time_expire=1)
    return locals()

Something weird happens at the python console
print "progress", cache.ram.storage['message'][1]
doesnt get printed on its own line, but the important thing is that im not accessing the declared message value.

And i get a log like this

progress progress progress progress main: 0
progress main: 1
progress progress progress main: 2
progress main: 3
progress main: 4
progress

Any idea on what's wrong?

Thanks


[Anthony]

    return cache.ram.storage['message'][1]

Where did you see that as the way to access cached values? That is not the proper API. Note, cache.ram.storage is not initialized until cache.ram has been called at least once, so unless you have a cache.ram() somewhere prior in the same request, cache.ram.storage will be an empty dictionary.

Instead, you should use the proper API:

    return cache.ram('message', lambda: None, None)

Using None as the third argument means the existing cached value will be retrieved, regardless of how much time has elapsed since it was stored (so the output of the lambda function is irrelevant, as it will never be called in this context).
 

def progress():
    # when a form is submitted, change progress each second
    if request.post_vars:
        for i in range(5):
            message = cache.ram('message', lambda: i, time_expire=1)

Note, here you want to make sure you update the cached value no matter what, so set time_expire to 0:

            message = cache.ram('message', lambda: i, time_expire=0)


Anthony

[Alfonso Serra]

Omg it did work!. Thanks you very much Anthony, i owe you one.

Console log:

progress -1
progress -1
main: 0
progress 0
main: 1
progress 1
main: 2
progress 2
main: 3
progress 3
main: 4
progress 4

 

[Dan]

I am working on a progress bar and I have used the suggestion on this post to use cache.ram as the mechanism to update the client (via ajax call) while the controller (on the server) is still working.

Using cache makes sense, since a global variable defined in a model such as db.py cannot be updated in the controller in a timely fashion and a session variable is not useful until the controller has finished working - which kind of beats the purpose of a progress bar...

I've noticed a strange behaviour and it took me a while to debug it:

If the controller that requires a progress bar has ANY session.variable defined, the progress bar will run the first time but not after that - unless sessions are cleared !

The solution I found was described in a different post (by Anthony as well) - the need to declare in the controller:

session.forget(response)

"session.forget(response) that will not only prevent the session from being saved, but it will also immediately unlock the session file. That could be useful if you will have multiple simultaneous requests coming from the same user/session (e.g., Ajax calls, or multiple browser tabs open) and don't want each request to have to wait for the previous one to complete. "

I will finalize the code the next days and will publish my "recipe" for a nice and easy progress bar, soon after.

Thanks

[Dan]

I've hit another obstacle using session.forget(response)...

In the same controller / function where I'm using this session.forget(response) for the progress bar (ajax) to work properly, I cannot use any session variables (since these are "forgotten" as per the command above) ?!

Is there a way to tell web2py to start "remembering" sessions again,  after I'm done with the progress bar (the lengthy process has finished, there's no more need for the progress bar) - and now I'd like to use the global nature of the session variables to pass data to other controllers.

Any ideas ?

Thanks

[Anthony]

Try calling session.connect().

Anthony

[chen...@gmail.com]

Alfonso, so if you have a working progress bar, can you please post your complete solution (with a view and java-script) here?

TIA,

[Alfonso Serra]

Ill try to explain it here but this is gonna require a video tutorial:

First there are several ways to achieve it. Although Anthony's suggestion (cache.ram) is optimal it didnt work on production, I dont know why cache.ram doesnt work on pythonanywhere.

I got the progress bar working on production using the scheduler, which i dont like cos its db intensive and more complex to setup, but i had no choice since the processes where too long to finish (10 minutes) and the http conection times out before its done.

Cache.ram
Create 2 controllers, one for your view and another one to get the progress.

def getprogress():

    return cache.ram('message', lambda: None, None)

def progress():
    # when a form is submitted, change progress each second
    if request.post_vars:

        n = 20 #simulate 20 iterations
        for i in range(n):
            msg = "Inserting records"
            percent = 100*(i+1) / n

            #update the percent each 5%
            if percent % 5 == 0:
                #store the array in ram [percent, text msg]
                message = cache.ram('message', lambda: ";".join([percent, msg]), time_expire=0)
               
            print "main:", percent, message
            sleep(1)
        #we are calling this controller through ajax so return a string when done
        return "done"
    else:
        message = cache.ram('message', lambda: -1, time_expire=0)
    return locals()

The view:

<div class="well">
        <div id="pbar">
            <div class="row">
                <div class="col-md-12">
                    <h3>Processing...</h3>
                </div>
                <div class="col-md-12">
                    <div class="progress progress-bar-xl" style="margin-top: 3px">
                        <div class="progress-bar progress-bar-warning" role="progressbar" aria-valuenow="60" aria-valuemin="0" aria-valuemax="100" style="width: 0%;">0%</div>
                    </div>
                </div>
            </div>
            <div class="row">
                <!-- <div class="col-md-1">
                    <input id="start" value="Pause" class="input-sm btn btn-primary btn-gradient btn-block dark" style="line-height: 0.8em;padding: 0" type="submit">
                </div>
                <div class="col-md-1">
                    <input id="abort" value="Abort" class="input-sm btn btn-danger btn-gradient btn-block dark" style="line-height: 0.8em;padding: 0" type="submit">
                </div> -->
            </div>
        </div>
    </div>

js:
create a progress bar "class" jquery ui required if i recall.

<script type="text/javascript">

    function ProgressBar (selector) {
        var self = this;
        self.el = $(selector)
        self.text = self.el.find("h3");
        self.progress = self.el.find(".progress-bar");
        self.btnstart = self.el.find("#start");
        self.btnabort = self.el.find("#abort");
       
        self.value = 0;
        self.min = 0;
        self.max = 100;
        self.interval = 5000;
        self.running = false;
        self.timer = null;
        self.offset = 0;

        self.set_progress = function (value) {
            value = value || 0;
            var done = false;
            if (value>=self.max) {
                value = self.max
                done = true;
            };
            value = parseInt(value) + "%";
            self.progress.css("width", value);
            self.progress.text(value);
            if (done) self.done();
        };

        self.done = function (msg) {
            var msg = msg || "Process Completed"
            self.text.text(msg)
            console.log("done", self.timer);
            clearInterval(self.timer);
            self.running = false;
        };

        self.abort = function (e) {
            if (self.running) {

                self.value = 0;
                self.set_progress(0);
                self.text.text("Aborted");
                if (self.timer) {
                    clearInterval(self.timer);
                };
                self.running = false;
            };
        }

        #ask the server for the progress
        self.ask = function () {
            var ask = $.ajax({
                url: window.location.origin + "/default/getprogress"
                , type: "get"
            });

            ask.done(function (res, msg, xhr) {
                console.log(res);
                if (res=="-1") {
                    console.log("not running...?");
                    self.done();
                    return
                }
                // var data = res.split(";");

                var percent = parseInt(res[0]);
                var msg = res[1];
                // if (msg == "Error") {
                //    return self.done(msg);
                // };

                self.text.text(msg);
                // console.log("resp", res);
                self.value = percent + self.offset;
                if (percent == 80 && self.offset < 10) {
                    self.offset +=0.5;
                }
                if (percent == 80 && self.offset >= 10 && self.offset < 15) {
                    self.offset +=0.1;
                }
                self.set_progress(self.value);
            });

            ask.fail(function (xhr, status, error) {
                var resp = xhr.responseText;
                self.done("Error (see browser's console for details)");
                if (console) {
                    console.log(resp);
                };
                // console.log(resp);
                // console.log(status);
                // console.log(error);
                // clearInterval(self.timer);
                // self.running = false;

            })
        };

        self.start = function (e) {
            self.text.text("Processing...");
            if (!self.running) {
                console.log("running");
              
                self.value = 0;
                self.set_progress();
                self.ask();
                self.timer = setInterval(self.ask, self.interval);
                self.running = true
            } else {
                console.log("already runing");
            }
        }

        self.set_progress(self.value);
        // self.btnstart.on("click", self.start);
        // self.btnabort.on("click", self.abort);
    }

    var pbar = new ProgressBar("#pbar");
    pbar.start();

Note: this ProgressBar function is expecting a json response from the server with an array like [50, "Processing"]

The scheduler is more complex to setup i might need to do a video tutorial to exaplain it.

[chen...@gmail.com]

So, I spend the best part of the day playing with your example and finally I made it work.

Here is what I got, after making some minor modification to your code to make it work for me:

In my controller/playground.py

# Progress bar example:
# Thanks to Alfonso Serra
# See: https://groups.google.com/d/msg/web2py/zgSLxeg7avw/JvPidEBMAQAJ
import time
import threading

def update_progress(n):
    for i in range(n):
        msg = "Working"

        percent = 100*(i+1) / n

        progress_data = cache.ram('progress_data',
                                  lambda: dict(percent=percent, msg=msg),
                                  time_expire=0)
       
        print "progress:", progress_data
        time.sleep(1)

def progress():
    # when a form is submitted, change progress each second

    a_form = SQLFORM.factory(Field('some_field'))
   
    if request.post_vars:
        threading.Thread(target=update_progress, args=[20]).start()
    else:
        progress_data = cache.ram('progress_data',
                                  lambda: dict(percent=(-1), msg='Ready'),
                                  time_expire=0)
    return locals()

def getprogress():
    progress_data = cache.ram('progress_data', lambda: None, None)
    print "getprogress:", progress_data
    return progress_data

And in my view/playground/progress.html I have:

{{extend 'layout.html'}}

{{=a_form}}

<div class="container-fluid">

    <div id="pbar">
        <div class="row">
            <div class="col-md-12">

                <h3 id="pbar_status">Ready...</h3>

            </div>
            <div class="col-md-12">
                <div class="progress progress-bar-xl"
                     style="margin-top: 3px">
                    <div class="progress-bar progress-bar-warning"
                         role="progressbar"
                         aria-valuenow="60"
                         aria-valuemin="0"
                         aria-valuemax="100"
                         style="width: 0%;">0%</div>
                </div>
            </div>
        <

/div>
    </div>
</div>

<script type="text/javascript">
 function ProgressBar (selector) {

     self = this
     
     self.el = $(selector)

     self.text = self.el.find("#pbar_status");

     // ask the server for the progress

     self.ask = function () {
         var ask = $.ajax({

             type: "GET",
             url: window.location.origin + "{{=URL('getprogress.json')}}",
             dataType: "json",
         });
         
         ask.done(function (json) {
             console.log(json);
             var percent = parseInt(json.percent);
             var msg = json.msg;
             if (percent == -1) {
                 console.log(msg);
                 return;
             };
             self.text.text("Processing...");
             self.text.text(msg);

             self.value = percent + self.offset;

             self.set_progress(self.value);
         });

         ask.fail(function (xhr, status, error) {
             var resp = xhr.responseText;
             self.done("Error (see browser's console for details)");
             if (console) {
                 console.log(resp);
             };

         });
     };

     self.start = function (e) {

         if (self.running) {
             console.log("already runing");
         } else {

             console.log("running");
             
             self.value = 0;
             self.set_progress();
             self.ask();
             self.timer = setInterval(self.ask, self.interval);
             self.running = true

         };
     };

     self.set_progress(self.value);
 }

 // start here
 $(function() {

     var pbar = new ProgressBar("#pbar");
     pbar.start();

 })
</script>

Thank you so very much.

Cheers,
Chen.