How to TERMINATE a RUNNING Scheduler Task?

[dlypka]

I need a way to terminate a specific Scheduled Task while it is RUNNING.
I tried using Admin Database Admin to update the scheduler_task.stop_time to a time close to now while it was running.

But the task continued for several more minutes and COMPLETED its normal 5 minute run.

The task calls time.sleep(300) to make it run for 5 minutes.

I guess the Scheduler is not looking at the db values every 3 seconds.
Is it just checking the in memory task object properties?

Is there a way to update the in memory object properties of a task while it is running?

Thanks.

[Niphlod]

once the task is inserted you should not change it's values unless its in the QUEUED status (technically the ASSIGNED works too, but it's NOT recommended).
there's no way for the scheduler to terminate a specific task once the task is started, unless you KILL the worker (setting the worker to TERMINATE will kill the worker as soon as the RUNNING task is finished).
PS: if you need to execute a task n times, use the repeat argument. using time.sleep(something) in a task has the side-effect of NOT returning to the main loop to execute potentially new QUEUED tasks (every scheduler process is allowed to process a single task at a time).
if you need to limit the time the task runs, use the timeout parameter.

If something is not clear please ask.

[dlypka]

Thanks for the quick reply.

I suppose a workaround is to have a separate worker for each task.

The I can TERMINATE the worker.

A suggestion: can you suggest some standard python scaffolding to include in each

task function to make it listen for a kill signal / message?

[Niphlod]

having a single worker for each task can be daunting (well, you could wrap web2py.py -K appname in a never-ending loop so it's restarted as soon as it gets killed)....

#!/bin/sh
while true
    do web2py/web2py.py -K yourapp
    echo 'killed, restarting in a bit'
    sleep 2
done

but I'm curious about your use-case. Why do you need to terminate a RUNNING task (that can't be accomplished using the timeout parameter)?

[dlypka]

timeout does not handle the case where it is discovered that the task needs to be stopped ASAP due to a business condition.

For example the remote server may have limited connections available so if a task runs into trouble, the process needs

to be killed ASAP to release the precious remote connection.  That is the case in my application.

Even with a while loop, there will have to be a way get the win32 process id of the python task for a given worker so the right one

can be killed.

Looks like a better solution is for the task to have its own manager thread that can listen for kill requests.

i.e. the scheduler tasks need to be written using a standard piece of scaffolding code to provide fine-grained task management.

[Niphlod]

"business condition" like ????
it seems a poor design to have a task, than watch it and if it "hangs for some reason" kill it abruptly not based on "task started a while ago"

PS: are you on windows ? signal management on win poses a lot of limitations.

[David Lypka]

a supervsior may decide to kill a task started by one of his team members.

--
 
---
You received this message because you are subscribed to a topic in the Google Groups "web2py-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/web2py/7-ZOS_In8IU/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to web2py+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Niphlod]

ok, seems more a goal of a workflow engine than a task queue, but I'll see what can be done.

[dlypka]

Well the functions to terminate will still have be manually initiated by an operator clicking a button.

And no workflow type of suspending / resuming a long running task is needed.

And no workflow designer and no workflow Activity classes such as State Activities needed.

Managers just need to know the tasks can be terminated anytime.

Meanwhile I am installing my Scheduler app on the corporate server tonight and the Manager will be pleased with what it does so far.

Thanks.

[Niphlod]

is it ok if we add, e.g. STOP_TASK to the list of worker statuses and STOP_TASK basically does what KILL does without actually killing the worker process ?

Having the process that is spawned to execute a task to spawn another thread to check on itself leads to transaction isolation problems.

Since tasks have a worker_name that for RUNNING is a reference to the worker row, you'll be able anyway to kill the task, but you'll do on the worker table instead of the scheduler_task one ....

On Monday, March 11, 2013 11:15:23 PM UTC+1, dlypka wrote:

Well the functions to terminate will still have be manually initiated by an operator clicking a button.

And no workflow type of suspending / resuming a long running task is needed.

And no workflow designer and no workflow Activity classes such as State Activities needed.

Managers just need to know the tasks can be terminated anytime.

Meanwhile I am installing my Scheduler app on the corporate server tonight and the Manager will be pleased will what it does so far.

Thanks.

On Monday, March 11, 2013 4:58:38 PM UTC-5, Niphlod wrote:

[Niphlod]

PS: the task will end in FAILED status

[David Lypka]

Wow. That will be nice.

Thanks.

On Mon, Mar 11, 2013 at 5:28 PM, Niphlod <nip...@gmail.com> wrote:

PS: the task will end in FAILED status

[Niphlod]

please test it (trunk version). I'm sending a patch to Massimo to include it in trunk.

[Niphlod]

ps: docs pretty self-explanatory, however ....
setting STOP_TASK on the scheduler_worker row will terminate the currently RUNNING task (if any)

given
myscheduler = Scheduler(db, ....)
in models,
in your controller you can call
actually_stopped = myscheduler.stop_task(15)

that will terminate task with id 15, while with

actually_stopped = myscheduler.stop_task('edeb8b54-dda8-4ff9-9585-2abb4a05c73a')

will terminate task with uuid edeb8b54-dda8-4ff9-9585-2abb4a05c73a

If the task is RUNNING it will be terminated --> execution will be set as FAILED by the worker as soon as the worker "reads" the STOP_TASK command (~3 secs), really_stopped will be 1
If the task is QUEUED, its stop_time will be set as to "now", the enabled flag will be set to False, and the status will be set to STOPPED, really_stopped will be 1
If the task is on any other task status, even if the reference passed to stop_task() is legit, nothing will be done, really_stopped will be None

[David Lypka]

OK will test it later tonight.

Thanks.

[David Lypka]

I ran one test and it did indeed stop the task.

scheduler_run.status = 'STOPPED'

scheduler_task.status = 'QUEUED'

scheduler_worker.status = 'ACTIVE'

return value from stop_task() was 1

So I got 'QUEUED' rather than 'FAILED' but that seems logical to me.

My task function body was:

def task_fabrun(cmdstring):   

    while True:

        time.sleep(.5) # time in seconds

    return cmdstring

Thanks.