scheduling multiple tasks

[Yebach]

Hello

In my web2py app i am using scheduler. So far I have one task scheduled which runs a subprocess when called from controler (an external exe file/application)

Now i want to add another task which will do some background work

My code in scheduler.py till now was

def runWoshiEngine(scriptId, path):

# import os, sys
# import time
import subprocess
print "runWoshiEngine in progress......"
p = subprocess.Popen(['woshi_engine.exe', scriptId], shell=True, stdout = subprocess.PIPE, cwd=path)
return dict(status = 1)


from gluon.scheduler import Scheduler
scheduler = Scheduler(db, heartbeat = 1)

so this way the scheduler started on client's request

after starting my app I run my scheduler with command

python web2py.py -K myapp

Now I want to add another function which will start every hour to do some background work.

What would you recommend and how to add it to scheduler because if i add anything to my line of code my initial task --> exe app is not started

thank you

best regards

[Dave S]

You're using the Popen because you're running something non-python, I take it.  And the  ".exe" suggests Windows environment.

I'm not sure why adding a line of code breaks your app start, but the recommended way of queuing the initial occurrence of a task that repeats periodically is to have a function in your controller file that has args (that makes it not be a URL target), and then use -S from the command line or from a startup file to invoke that function.

See <URL:https://groups.google.com/d/msg/web2py-developers/cI7R-9hex7k/PfTsGodYEwAJ>
for some sample code.  That's a PR that pending for the web2py book.

/dps

[Nico de Groot]

Looking at your code you posted earlier at http://stackoverflow.com/questions/34661782/web2py-function-not-triggered-on-user-request/34895532#34895532 ou have a task running the engine. This task is scheduled by adding it adding it to the scheduler. You can create a second task and add this task using a one-time controller, a script, or at application start time.

Nico de Groot

[Vid Ogris]

Ok thanx

How and where do I add one time controller, or application start?

 my task will be executed once per hour

2016-11-11 7:18 GMT+01:00 Nico de Groot <ndeg...@gmail.com>:

Looking at your code you posted earlier at http://stackoverflow.com/questions/34661782/web2py-function-not-triggered-on-user-request/34895532#34895532 ou have a task running the engine. This task is scheduled by adding it adding it to the scheduler. You can create a second task and add this task using a one-time controller, a script, or at application  start time.

Nico de Groot

--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to a topic in the Google Groups "web2py-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/web2py/FK1ygjNNjDU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to web2py+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Lep pozdrav 

Vid Ogris

[黄祥]

pls try : (you can improve it to another queue task by define another function in controller)

controllers/default.py

"""

for running scheduler

python web2py.py --nogui --no-banner -K woshiweb -D 0

1 hour = 3600 seconds # for period

10 minutes = 600 seconds # for timeout

"""

start_now = datetime.datetime.now()

stop_time_now = (start_now + datetime.timedelta(days = 1) )

def queue_task_0():

scheduler.queue_task('runWoshiEngine', prevent_drift = True, start_time = start_now, 

next_run_time = start_now, stop_time = stop_time_now,

repeats = 0, retry_failed = 1, period = 3600, timeout = 600

)

session.flash = T("Task 0 Queued")

redirect(URL('index.html') )

best regards,

stifan

[Vid Ogris]

The runWoshiEngine is started on user request. I have to get another one  working like every hour after the app was started.

Call it saveIdOut

in scheduler.py

from gluon.scheduler import Scheduler

def runWoshiEngine(scriptId, path):

    # import os, sys
    # import time
    import subprocess
    print "runWoshiEngine in progress......"
    p = subprocess.Popen(['woshi_engine.exe', scriptId], shell=True, stdout = subprocess.PIPE, cwd=path)
    return dict(status = 1)

def saveIdOut(a,b):
    print b
    print "haha " + a
    print "----"
    return dict(status = 1)

scheduler = Scheduler(db, tasks = dict(runWoshiEngine = runWoshiEngine,saveIdOut=saveIdOut ) ,heartbeat = 1)

I put 

task = scheduler.queue_task(saveIdOut, [1,2], start_time=now,  # datetime
                  stop_time=None,  # datetime
                  timeout = 60,  # seconds
                  repeats=6)

in my main controler 

And i always get failed in my scheduler table 

--

Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to a topic in the Google Groups "web2py-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/web2py/FK1ygjNNjDU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to web2py+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dave S]

On Friday, November 11, 2016 at 4:15:46 AM UTC-8, Yebach wrote:

Ok thanx

How and where do I add one time controller, or application start?

 my task will be executed once per hour

Was the example I pointed to not clear?  What questions did you have after reading it?  What could I explain better?

/dps
 

[Nico de Groot]

Hai Dave,

I ran your code and checked the field 'traceback' in the table 'scheduler_run' for the failing task. I found:

Traceback (most recent call last):
  File "/Users/ncdegroot/web2py/gluon/scheduler.py", line 315, in executor
    result = dumps(_function(*args, **vars))
  File "applications/welcome/models/scheduler.py", line 16, in saveIdOut
    print "haha " + a
TypeError: cannot concatenate 'str' and 'int' objects

The cause of this is the line in your task:

print 'haha' + b

When queing you give the task two integer parameters in [ 0,1]. In python you can't directly concatenate a string and an int. It should be

print 'haha'+str(i) 

 

With this change your tast is repeating 6 times as intended.

But I know it's only testcode. I  expect you get the hint: Just use the traceback field to check why a certain run of your task is failing. 

There is a nice plugin by Niphlod I'm using in my apps to queue and check tasks. Make it's easier. See https://github.com/niphlod/cs_monitor_plugin 

Nico de Groot

2016-11-11 19:59 GMT+01:00 Dave S <snide...@gmail.com>:

On Friday, November 11, 2016 at 4:15:46 AM UTC-8, Yebach wrote:

Ok thanx

How and where do I add one time controller, or application start?

 my task will be executed once per hour

Was the example I pointed not clear?  What questions did you have after reading it?  What could I explain better?

/dps
 

--

[Nico de Groot]

Correction:

Hi Vid,

[Vid Ogris]

Thank you guys. I finally made it. I did not quite understand the whole scheduler process but i am now getting there.

Thank you again

2016-11-14 2:50 GMT+08:00 Nico de Groot <ndeg...@gmail.com>:

Correction:

Hi Vid,

--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to a topic in the Google Groups "web2py-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/web2py/FK1ygjNNjDU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to web2py+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dave S]

On Tuesday, November 15, 2016 at 2:14:21 AM UTC-8, Yebach wrote:

Thank you guys. I finally made it. I did not quite understand the whole scheduler process but i am now getting there.

Thank you again

Congratulations!  It sometimes takes a while to get the right mental images, doesn't it?

/dps

[Vid Ogris]

SO the problem continues.

The task was running  and was completed for some dummy code.

Now i have to go trough a folder, read a file and update file content to database.

The task always times out and there are two files that need to be updated, non of them are. The code is executed, but there is no db update. Also it seems for loop is not completed. For only one file db._lastsql is printed in db. The statement in scheduler table is ok and if i run it it inserts data into db. Any suggestions?

My code

in scheduler.py 

def saveIdOut():
    try:
        #demo koda
        #poberem statuse
        db_status = db(db.scripts.sc_status == 11).select(db.scripts.id).as_list()
        #ggrem v mapo in pogledam katere skripte ki imajo status 11 se dejansko izvajajo
        for rec in db_status:
            scriptId = rec["id"]
            outPath  = os.path.join(request.folder, 'engine', 'e1', str(scriptId) + '.out')
            #preverim če obstaja datoteka
            if os.path.isfile(outPath):
                #če obstaja jo shranim v sc_engine_output
                #sparsamo out datoteko da jo lahko damo v tabelo result
                out = readOutFile(str(scriptId))
                #Vnesemo podatke v tabelo script
                db(db.scripts.id == scriptId).update(sc_engine_output = out)
                print db._lastsql

                print "out datoteka za scripto %s, arhivirana" % (str(scriptId))

        db.commit()
        return dict(status = 1)

    except Exception, e:
        exc_type, exc_obj, exc_tb = sys.exc_info()
        fname = os.path.split(exc_tb.tb_frame.f_code.co_filename)[1]
        print(exc_type, fname, exc_tb.tb_lineno)
        print e.__doc__
        print e.message
        return dict(status = 666)

in my controler

def save_outs():

    """
    for running scheduler
    python web2py.py --nogui --no-banner -K woshiweb -D 0

    1 hour = 3600 seconds # for period
    10 minutes = 600 seconds # for timeout
    """

    import datetime
    now = datetime.datetime.now()

    start_now = datetime.datetime.now()

    stop_time_now = (start_now + datetime.timedelta(days = 1))

    scheduler.queue_task(saveIdOut, start_time = start_now,

                   next_run_time = start_now, stop_time = stop_time_now,

                   repeats = 0, retry_failed = 2, period = 30, timeout = 15)

    print "Funkcija za arhiviranje outov AKTIVIRANA...."
    print "Glej scheduler tabele v postgresu"
    session.flash = T("Task save_outs Queued")
    redirect(URL('index.html'))

--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to a topic in the Google Groups "web2py-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/web2py/FK1ygjNNjDU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to web2py+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nico de Groot]

Hi Vid,

Two remarks at first sight.

In save_outs you initialize the variable now. But you queue the task using start_now. But the task is inserted, maybe just copy paste error in your post...

Is the function readOutFile working? Can you show It? It seems it gets stuck when called the second time. I would suggest to step through the code using the debugger, or use print statement to pinpoint the timeout.

Nico

Op za 19 nov. 2016 om 05:41 schreef Vid Ogris <vid....@gmail.com>

To unsubscribe from this group and all its topics, send an email to web2py+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Lep pozdrav 

Vid Ogris

--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to a topic in the Google Groups "web2py-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/web2py/FK1ygjNNjDU/unsubscribe.

To unsubscribe from this group and all its topics, send an email to web2py+un...@googlegroups.com.

[Vid Ogris]

There is no errors, the file is read etc. The task's status is always TIMEOUT, although the first time I call it is executed correctly.

Then I delete the values of the fields the task manipulates with and the second time around values are not inserted and also there is no error or anything in the tables.

any suggestiions?

To unsubscribe from this group and all its topics, send an email to web2py+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Lep pozdrav 

Vid Ogris

--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to a topic in the Google Groups "web2py-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/web2py/FK1ygjNNjDU/unsubscribe.

To unsubscribe from this group and all its topics, send an email to web2py+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to a topic in the Google Groups "web2py-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/web2py/FK1ygjNNjDU/unsubscribe.

To unsubscribe from this group and all its topics, send an email to web2py+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Vid Ogris]

Hello guys. Can somebody please help me with this. I am out of ideas and i need to sort this out. Thank you

[Nico de Groot]

If you want, pack your app or a example with same behavior using the admin interface as a w2p file and send it to the list.

No promises...

Nico


Op di 22 nov. 2016 om 12:47 schreef Vid Ogris <vid....@gmail.com>

To unsubscribe from this group and all its topics, send an email to web2py+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Lep pozdrav 

Vid Ogris

--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to a topic in the Google Groups "web2py-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/web2py/FK1ygjNNjDU/unsubscribe.

To unsubscribe from this group and all its topics, send an email to web2py+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to a topic in the Google Groups "web2py-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/web2py/FK1ygjNNjDU/unsubscribe.

To unsubscribe from this group and all its topics, send an email to web2py+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Lep pozdrav 

Vid Ogris

--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to a topic in the Google Groups "web2py-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/web2py/FK1ygjNNjDU/unsubscribe.

To unsubscribe from this group and all its topics, send an email to web2py+un...@googlegroups.com.