Multiple Threading

[tommy]

I tried to run some tasks every certain minutes in the background for
my WEB2PY application. I created another thread, but my web page
screen was stuck there for ever. Can someone tell me how to write
multthreaded GUI application in web2py?

[mdipierro]

Not sure I understand the question.
Let's say you create a "test" application with a simple "default.py"
controller:

def index(): return "hello world"

This is already a multithreaded application. Every time an http
request for "http://localhost:8000/test/default/index" arrives, a new
thread is created by the web2py web server.
It is the web server that creates threads (at every request) and kills
them (when they take too long). The web server is often even smarter
than that and reuses existing threads.

threads created this way can share information using one of the
following mechanisms:
1) database (db=SQLDB(....))
2) cache (cache.disk, cache.ram, cache.memcache)
3) session (session.myvariable=3)
4) filesystem (open(os.path.join(request.folder,'private/
myfile.txt),'w').write('hello'))

You should not create threads manually by using thread and threading
modules.

Massimo

[billf]

I started to write a post saying "ok - can't start a new thread - so
how can I run a background task?" and then I really struggled to find
an example that couldn't be solved some other way.

The only example I came up with was a major processing task, maybe
some major calculation or db search/analysis, that is kicked off by a
request, runs in background and sets an indicator when completed.

1) what other background tasks do people envisage?

2) how should you do it in web2py?

[Keith Edmunds]

On Sun, 26 Oct 2008 11:55:08 -0700 (PDT), billf...@blueyonder.co.uk
said:

> 1) what other background tasks do people envisage?

A regular display update via Ajax (but that could probably be demanded by
the client rather than pushed)

> 2) how should you do it in web2py?

Pass.

--
Keith Edmunds

[Jose de Oliveira Filho]

Hi Bill,

How about COMET ("reverse ajax") applications, like they use in chat
clients such as meebo.com ? I guess those need background tasks by
nature.

Deodoro Filho

[mdipierro]

This app does that. You upload a video and it is converted in
background (like at youtube) in a separate process (not a thread, a
process although the process my have threads). The visitor can later
check the conversion was completed. It includes documentation:

http://mdp.cti.depaul.edu/appliances/default/show/38

Massimo

On Oct 26, 2:00 pm, Keith Edmunds <k...@midnighthax.com> wrote:
> On Sun, 26 Oct 2008 11:55:08 -0700 (PDT), billferr...@blueyonder.co.uk

[tommy]

Massimo,

Here is my code,

I use GUI button to start my service (StartEngine, in the background,
I want to retrieve the infomation from another application and save it
in my database for every couple of munites. after I starting my start
engine service, I can go to other screen. and the background service
keeps runing.


I can create the thread now, but got an error :SQLITE object created
in a thread can only used in that same thread.

def enginestart():
menu_setup()
# engine.start()
session.starttimer=True

symbols=db(db.symbols.id>0).select()

# for symbol in symbols:
# engine.addMarket(symbol.symbol_Name,
symbol.description,symbol.tick)


session.flash='engine stared'
session.enginestarted=True
db(db.historyprice.id >0).delete()
# tt=threading.Timer(2, gethistoryprice(db))
# tt.start()
t=Gethistoryprice()
t.start()
redirect(URL(r=request,c='engine',f='enginemaint'))
return dict() session.starttimer=True

symbols=db(db.symbols.id>0).select()

# for symbol in symbols:
# engine.addMarket(symbol.symbol_Name,
symbol.description,symbol.tick)


session.flash='engine stared'
session.enginestarted=True
db(db.historyprice.id >0).delete()
t=Gethistoryprice(db)
t.start()
redirect(URL(r=request,c='engine',f='enginemaint'))
return dict()



class Gethistoryprice(threading.Thread):
def __init__(self, db):
threading.Thread.__init__(self)

self.stopped=0
self.db=db


def run(self):
while not self.stopped:
mysym=self.db(db.symbols.id>0).select()
for symbol in mysym:
# p=engine.getlastprice(symbol)
p=random.randrange(10.00,60.00)
now=datetime.datetime.now()
db.historyprice.insert(symbol_Name=symbol.symbol_Name,
timestamp= now, price=p)

def stop(self):
self.stopped=1

> > Keith Edmunds- Hide quoted text -
>
> - Show quoted text -

[Tito Garrido]

Hi tommy,

I don't know if it's a good ideia to put this verification (session.enginestarted=True) in a session var. Because the client can close the browser, open the link again and start another thread...

I have here a small example where the client can start just one thread.

In a model:
db.define_table('test',SQLField('counter','integer'),SQLField("doing_something", "boolean", notnull=True, default='False'))

In a view: (default/test.html)
{{extend 'layout.html'}}
<h1>This is the default/test.html template</h1>
<form><input id='source' type='text' value={{=old}} readonly="readonly"></form>
<button id='botao', name='botao', onclick="ajax('add_more',[],'destination');"> DO SOMETHING! </button><br><br>

<div id="destination">...</div> <!-- results should appear here -->

In a controller:
def add_more():
    import time
    doing_something=db(db.test.id==1).select(db.test.doing_something)[0].doing_something # get the value of doing_something
    if not doing_something: # if it is = false
        db(db.test.id==1).update(doing_something=True) # doing something now...
        db.commit() # very important, commit it
        time.sleep(10) # doing something....
        old=db(db.test.id==1).select(db.test.counter)[0] # just adding... :)
        db(db.test.id==1).update(counter=old.counter+1) # just adding... :)
        db(db.test.id==1).update(doing_something=False) # Ok... done something...
        return 'OK! You can do it again!'
    else:
        return 'you are already doing something...'
  
def test():
    return dict(old=db(db.test.id==1).select(db.test.counter)[0].counter)

There is a detail in web2py_ajax.html that you must change:
function ajax(u,s,t) {
  var query="";
  for(i=0; i<s.length; i++) {
     if(i>0) query=query+"&";
     query=query+encodeURIComponent(s[i])+"="+encodeURIComponent(document.getElementById(s[i]).value);
  }
  $.ajax({async: false, type: "POST", url: u, data: query, success: function(msg) { document.getElementById(t).innerHTML=msg; } });
}

with async = false your client should be able to run just a single thread per time, so it avoids repetitious clicks opening a lot of threads.

I hope that this example helps you!

Regards,

Tito Garrido :)

--
Linux User #387870
.........____
.... _/_õ|__|
..º[ .-.___.-._| . . . .
.__( o)__( o).:_______

[billf]

Assuming an event causes the data in the "other" application to change
wouldn't it be easier for the "other" application to post the changes
to your web2py app when the event occurs? Having said that, I can
envisage a case where there are hundreds or thousands of updates per
couple of minutes and you only want a snapsot.

Looking at Massimo's videotest app, that seems to use the approach of
starting a process outside of web2py. If that is the case then that
process could execute a method after an interval that either:
a) sends your web2py app a request that causes it to retrieve the data
from the "other" application and update the web2py app database, or
b) the process gets the data and updates the web2py database directly
(unknown to your web2py app that just selects and gets the latest)

To stop the external process your web2py app could write to the db or
a file that is checked by the external process each time it wakes up.

Admittedly, none of the above explains how to create a background
process within web2py - maybe it is just not meant to be done by a
mere mortal.

[achipa]

Actually, I made a patch that allows execution of code in a parallel
thread after the page has been served. It can be scheduled (e.g.
cron), or just launched in a fire and forget fashion without affecting
the main app and the user (e.g. session cleaning, a long(er)
conversion task, etc). It runs on all platforms and under both
mod_wsgi and the stock cherrypy. I sent the patch to Massimo, but I
have had no response so far, I don't know if he disapproves of the
patch or simply did not have the time to review it. If there are brave
souls willing to try it out, I can attach it here, too.

[mdipierro]

I am sorry. I have a work deadline in two weeks and there are a few
patches from Bill that are complex and I need the time to study them
in some detail. They are not lost. Anyway, feel free to send me
periodic reminders using my personal email.

Massimo

[tommy]

Archipa, when you use the different thread to access same SQLITE
database, do you have any problem? It looks like SQLITE database can
be only accessed by one thread. How do you walk around it?

To all, thanks for your replies, I think my design has some issue.
Massimo's example code and Bill's suggestion made me think I need to
create another process in the background. Tito's suggestion is also
good. but I am so bad at Ajax.

Tommy

On Oct 26, 4:31 pm, achipa <attila.cs...@gmail.com> wrote:

> > > > - Show quoted text -- Hide quoted text -

[mdipierro]

SQLite locks the database. Only one thread can safely access it.

[achipa]

Note that databases generally dislike multithreaded access, my patch
won't help you with that. You can shift some the problem to the
database if you're using multiple connections, but with SQLite you
don't have that option (it's not 'lite' for no reason). The database
adapter that boasts to be the best with that regard would be a
postgresql + psycopg2 backend, but even that won't save you from logic
errors and deadlocks - data consistency is tricky business in
multithread land.

[Phyo Arkar]

I have not  read everything carefully yet .I am doing a system monitor ajax+web2py application , which monitor CPU and MEM at the same time and updates the page with jquery ajax effects. its pretty simple.So it is multi-threading in a webapp . to get response from multiple httprequest at the same time can be done like this.

NOTE: for this System monitor to work,  Requires libstatgrab and pystatgrab .

libstatgrab + pystatgrab works under Linux and Cygwin.

to

in controllers , default.py :

import statgrab

def index():
    return dict()


def cpumon():
   
    cpustat = statgrab.sg_get_cpu_percents()
    perc = int(round(100 - cpustat['idle']))
    return perc

def memon():
    
    #otno=random.randrange(0,100)
    memstat = statgrab.sg_get_mem_stats()
    memdiff = memstat['total'] - (memstat['cache'] + memstat['free'] )
    memfloat = float (memdiff) / float(memstat['total'])
    perc = int(round (memfloat * 100))
    return perc


in view . Index.html :

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
    "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html>
<head>
<title>Ajax + Web2py monitor</title>
<link rel="stylesheet" type="text/css" href="css/style.css"/>
<script type="text/javascript" src="js/jq.js"/>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
<meta name="author" content="Your name here" />
<meta name="Copyright" content="Copyright (c) Your copyright here 2005" />

<script language="JavaScript1.8">
// You can put Functions i
</script>



<script language="JavaScript1.8">
   
    var clockid = 0;
    function updatecpu()
    {
        if(clockid)
        {
            clearTimeout(clockid);
            clockid = 0;
        }
        cpueffect();
        clockid = setTimeout('updatecpu()',2000);
       
    }

    function cpueffect() {
       
        $.post(('cpumon'),function(data)
        {
            if (data < 6){
                $("#amount").animate(
                {
                    width: 6 + "%"           
                },1000);
                document.getElementById('amount').innerHTML=data + '%';
   
            } else {
                $("#amount").animate(
                {
                    width: data + "%"           
                },1000);
                document.getElementById('amount').innerHTML=data + '%';
            }


           
        });
       
    }
    $(document).ready(updatecpu());
</script>

<script language="JavaScript1.8">
   
    var clockid2 = 0;
        function updatemem()
        {
            if(clockid2)
            {
                clearTimeout(clockid2);
                clockid2 = 0;
            }
            memeffect();
            clockid2 = setTimeout('updatemem()',2000);
           
        }
       
        function memeffect()
        {
           
            $.post(('memon'),function(data)
            {
                $("#mem").animate(
                {
                    width: data + "%"           
                },1000);
                document.getElementById('mem').innerHTML=data + '%';
               
            });
           
        }
    $(document).ready(updatemem());
</script>

</head>
<body>

<h5><a href="#" onclick=blinds() >JxMon </a></h5>



<!-- ****************** BEGIN CPU Progress Bars ****************** -->

<h3><a href="#" onclick=blinds() >cpu: </a></h3>
<div class= "mon_cont" >
    <div class="cpumon_fill" id='amount'></div>
</div>

<h3><a href="#" onclick=blinds() >mem: </a></h3>
<div class= "mon_cont"  >
    <div class="memmon_fill" id='mem'></div>
</div>


</body>
</html>

[Phyo Arkar]

I forget some notes ..

NOTE: for this System monitor to work,  Requires libstatgrab and pystatgrab .


libstatgrab + pystatgrab works under Linux and Cygwin.

for linux under debian based distro. apt-get install libstatgrab to get statgrab library and get pystatgrab here :

http://www.i-scream.org/pystatgrab/


NOTE: layout.html is not used so jquery files and ajax files need to be included manually.

For short: Following 2 JQuery Function does the trick:

  $.post(('cpumon'),function(data)
        {
            if (data < 6){
                $("#amount").animate(
                {
                    width: 6 + "%"          
                },1000);
                document.getElementById('amount').innerHTML=data + '%';

}

This makes posts in parallel.

<script language="JavaScript1.8">
   

    var clockid = 0;
    function updatecpu()
    {
        if(clockid)
        {
            clearTimeout(clockid);
            clockid = 0;
        }
        cpueffect();
        clockid = setTimeout('updatecpu()',2000)

;
       
    }