How to implement a semaphore to fix race condition with multiple HTTP Get calls
Sal Lusso
Task: querySomeWebsite
A1: HTTP Get [ Server:Port:somewebsiteurl.com Path: Attributes: Cookies: User Agent: Timeout:30 Mime Type: Output File: Trust Any Certificate:Off Continue Task After Error:On ]
A2: If [ %HTTPR eq 200 ]
A3: Variable Set [ Name:%data To:%HTTPD Do Maths:Off Append:Off ]
...
If multiple such tasks are running with different URL's, I'm sometimes getting my %data variable populated with the results of another tasks's HTTP Get call.
The solution is obviously to implement a semaphore, but I'm not sure how to do that robustly in Tasker. I don't see atomic increment or test-and-set functions.
So I'm looking for suggestions on how to implement a general semaphore construct in Tasker.
I'm thinking that an action like this *might* work as an atomic test-and-set:
A1: Variable Set [ Name:%Lock To:%newval Do Maths:Off Append:Off ] If [ %Lock eq 0 ]
Based on this assumption, I have created a set of 3 tasks to implement a lock for HTTP calls. The lock has a timeout feature as well so that if a task fails during execution, it doesn't block all subsequent HTTP calls indefinitely. Here are the 3 tasks and an example of how to call them for a safe HTTP Get request:
NOTE: Set %CfgHttpTimeoutSecs to some value, like 30.
Task: getHttpLock
Run Both Together
A1: Variable Set [ Name:%lock_timeout To:%CfgHttpTimeoutSecs * 1000 + 5000 Do Maths:On Append:Off ]
<whileLoop>
A2: Wait Until [ MS:0 Seconds:3 Minutes:0 Hours:0 Days:0 ] If [ %TIMEMS > %HttpLock + %lock_timeout - 1 ]
A3: Variable Clear [ Name:%lock Pattern Matching:Off ]
A4: Perform Task [ Name:setHttpLock Priority:%priority Parameter 1 (%par1):%lock_timeout Parameter 2 (%par2): Return Value Variable:%lock Stop:Off ]
A5: If [ %lock > 0 ]
A6: Return [ Value:%lock Stop:On ]
A7: Else
A8: Goto [ Type:Action Label Number:1 Label:whileLoop ]
<whileLoopEnd>
A9: End If
Task: setHttpLock
A1: Variable Set [ Name:%lock_timeout To:%par1 Do Maths:Off Append:Off ]
A2: Variable Randomize [ Name:%random Min:1 Max:999 ]
A3: Variable Set [ Name:%random To:%random / 1000 Do Maths:On Append:Off ]
A4: Variable Set [ Name:%now To:%TIMEMS Do Maths:Off Append:Off ]
A5: Variable Set [ Name:%lock To:%now + %random Do Maths:On Append:Off ]
A6: Variable Set [ Name:%HttpLock To:%lock Do Maths:Off Append:Off ] If [ %now > %HttpLock + %lock_timeout - 1 ]
A7: If [ %HttpLock eq %lock ]
A8: Return [ Value:%lock Stop:On ]
A9: Else
A10: Return [ Value:0 Stop:On ]
A11: End If
(NOTE: setHttpLock has "Abort New Task" as its Collision Handling setting. When getHttpLock calls this task, it is built to retry when there is no result. This mechanism itself could perhaps be the basis of a sempahore implemention?)
Task: releaseHttpLock
Run Both Together
A1: Variable Set [ Name:%lock To:%par1 Do Maths:Off Append:Off ]
A2: Variable Set [ Name:%HttpLock To:0 Do Maths:Off Append:Off ] If [ %HttpLock eq %lock ]
Task: querySomeWebsite
A1: Perform Task [ Name:getHttpLock Priority:%priority Parameter 1 (%par1): Parameter 2 (%par2): Return Value Variable:%http_lock Stop:Off ]
A2: HTTP Get [ Server:Port:somewebsiteurl.com Path: Attributes: Cookies: User Agent: Timeout:%CfgHttpTimeoutSecs Mime Type: Output File: Trust Any Certificate:Off Continue Task After Error:On ]
A3: Variable Set [ Name:%httpr To:%HTTPR Do Maths:Off Append:Off ]
A4: Variable Set [ Name:%httpd To:%HTTPD Do Maths:Off Append:Off ]
A5: Perform Task [ Name:releaseHttpLock Priority:%priority Parameter 1 (%par1):%http_lock Parameter 2 (%par2): Return Value Variable: Stop:Off ]
A6: If [ %httpr eq 200 ]
A7: Variable Set [ Name:%data To:%httpd Do Maths:Off Append:Off ]
...
I'm interested in getting feedback on this solution or suggestions on the preferred way to implement a semaphore in Tasker.
Rich D
>
> I'm interested in getting feedback on this solution or suggestions on the preferred way to implement a semaphore in Tasker.
>
Here is what I would do.
Create 1 task that will do the http get and give it a user name. Then when you want to do a http get from any other task use the preform task action and send any necessary info in %par1 and %par2. And use the return action to get the info back. Just before the preform task action do a wait until action and wait until %TRUN !~ *,Name Of The HTTP get Task,*
The *,,* are important, see pattern matching in the guide.
%TRUN contains a comma separated list of all "User Named" running tasks
Pent
There were no local variables at the time it was implemented, sorry!
Pent
Sal Lusso
Are there any atomic constructs in Tasker though that address the need for semaphores?
1) For example, can this be assumed to be an atomic test-and-set?
A1: Variable Set [ Name:%Lock To:%newval Do Maths:Off Append:Off ] If [ %Lock eq 0 ]
3) Would Java calls to the atomic package functionality work? I.e., does Tasker always run everything in the context of one global JVM?
Any advice on these or other suggestions?
Thanks.
Pent
so a concurrently running task can slip its own action in, between doing HTTP Get and testing the result,
in this case.
Rich's solution should be just fine since there is only one task dealing with the %HTTPx variables
at any one time.
Pent
Sal Lusso
So, this action would act as an atomic test-and-set:
A1: Variable Set [ Name:%Lock To:%newval Do Maths:Off Append:Off ] If [ %Lock eq 0 ]
Based on that, I can simplify my functions a bit; I will repost with the code when done...
The drawbacks I see with Rich's solution are:
1) It doesn't scale well with n different HTTP Gets as n grows bigger... each one needs to check if any others are running. In other words, each call needs to know about the names of every other, and adding a new call requires modifying each call that already exists.
2) Or, if I try to make a generic HTTP Get wrapper task instead to avoid this, there is complexity in packing and unpacking the inputs and outputs since there can be more than two relevant inputs and one relevant output (which are Tasker's out-of-the-box limitations on perform task calls). I will end up with lots of duplicated [and hacky] code to handle inputs/outputs in each caller, more than the getLock and releaseLock calls I can otherwise use.
Juergen Gruen
interesting topic.
My workaround: I do not use HttpGet but do all my http requests with Javascript.
- only one js action at a certain time performing
- error handling is easier
- data handling is easier (json, XML, regex, string functions)
Juergen.
Sal Lusso
As promised, here is my updated, simpler recipe for implementing a lock semaphore to make HTTP Get actions safe in a multiple concurrent task environment... There are two tasks created named getHttpLock and setHttpLock.
STEP 1) Set a global %CfgHttpTimeoutSecs var to some value, like 30.
This will be used in the HTTP Get calls and will also ensure that any lock older than this (such as from a task that died) is treated as stale and won't block other tasks from making HTTP calls.
STEP 2) Create these two lock management tasks:
Task: getHttpLock
Run Both Together
<whileLoop>
A2: Wait Until [ MS:0 Seconds:3 Minutes:0 Hours:0 Days:0 ] If [ %TIMES > %HttpLock + %lock_timeout - 1 ]
A3: Variable Randomize [ Name:%random Min:1 Max:999 ]
A4: Variable Set [ Name:%random To:%random / 1000 Do Maths:On Append:Off ]
A5: Variable Set [ Name:%lock_time To:%TIMES Do Maths:Off Append:Off ]
A6: Variable Set [ Name:%lock To:%lock_time + %random Do Maths:On Append:Off ]
A7: Variable Set [ Name:%HttpLock To:%lock Do Maths:Off Append:Off ] If [ %lock_time > %HttpLock + %lock_timeout - 1 ]
A8: If [ %HttpLock eq %lock ]
A9: Return [ Value:%lock Stop:On ]
A10: Else
A11: Goto [ Type:Action Label Number:1 Label:whileLoop ]
<whileLoopEnd>
A12: End If
Task: releaseHttpLock
Run Both Together
A2: Variable Set [ Name:%HttpLock To:0 Do Maths:Off Append:Off ] If [ %HttpLock eq %lock ]
STEP 3) Enclose all HTTP Get actions in these two functions as in this example where the call is made and the global variables %HTTPR and %HTTPD are read before releasing the lock:
Task: querySomeWebsite
A1: Perform Task [ Name:getHttpLock Priority:%priority Parameter 1 (%par1): Parameter 2 (%par2): Return Value Variable:%http_lock Stop:Off ]
A2: HTTP Get [ Server:Port:somewebsiteurl.com Path: Attributes: Cookies: User Agent: Timeout:%CfgHttpTimeoutSecs Mime Type: Output File: Trust Any Certificate:Off Continue Task After Error:On ]
A3: Variable Set [ Name:%httpr To:%HTTPR Do Maths:Off Append:Off ]
A4: Variable Set [ Name:%httpd To:%HTTPD Do Maths:Off Append:Off ]
A5: Perform Task [ Name:releaseHttpLock Priority:%priority Parameter 1 (%par1):%http_lock Parameter 2 (%par2): Return Value Variable: Stop:Off ]
A6: If [ %httpr eq 200 ]
....
Rich D
> As promised, here is my updated, simpler recipe for implementing a lock semaphore to make HTTP Get actions safe in a multiple concurrent task environment... There are two tasks created named getHttpLock and setHttpLock.
I honestly am having difficulty following your approach, and trying to figure it with multiple iterations of the lock task running makes figuring all potential outcomes very difficult.
Personally I would not trust any task set to 'run both together' that sets a global variable for flow control.
I agree with your issues with my first approach. I would suggest instead of doing the http get in the child task just use the child task as the lock.
However, There is also a potential for it to fail. No matter what kind of test you do before a the preform task action there is a potential of failure. Consider the following with task 1 and task 2 having the same priority and alternating actions.
(Task 1) -Wait until %trun !~ *,Task Name,* // this action proves true and the next action will be preformed//
(Task 2) -Wait until %trun !~ *,Task Name,* // this action proves true and the next action will be preformed//
(Task 1) - preform task // task starts , if you use %priority+1 this will ensure the child runs to completion before task 1 or task 2 continue however once the child completes I believe the next action will be from task 2
(Task 2) - preform task
This could cause potential issues.
I believe the answer would be to incorporate the 'test' into the preform task action then have a response from the child (from the return action) to the parent to prove the preform task action was completed. something like this..
Http Get (781)
<start loop>
A1: Wait Until [ MS:0 Seconds:2 Minutes:0 Hours:0 Days:0 ] If [ %TRUN !~ *,Lock,* ]
A2: Perform Task [ Name:Lock Priority:%priority+1 Parameter 1 (%par1): Parameter 2 (%par2): Return Value Variable:%lock Stop:Off ] If [ %TRUN !~ *,Lock,* ]
A3: Goto [ Type:Action Label Number:1 Label:start loop ] If [ %lock !~ true ]
A4: HTTP Get [ Server:Port:%test Path: Attributes: Cookies: User Agent: Timeout:10 Mime Type: Output File: Trust Any Certificate:Off ]
A5: Variable Set [ Name:%set_your_variables To:%HTTPD Do Maths:Off Append:Off ]
A6: Stop [ With Error:Off Task:Lock ]
Lock (782)
A1: Return [ Value:true Stop:Off ]
A2: Wait [ MS:0 Seconds:0 Minutes:10 Hours:0 Days:0 ]
So now the preform task action will only run if the child task is not running (lock is off).
If this fails (lock is on) %lock will not get set and it will go back to the top of the loop.
If the preform action runs then %lock returns and matches 'true' and the http get will run. The child task continues running because of the wait action (so the lock remains on until the 'Stop task' action.
Sal Lusso
It's interesting how you're using a running named task as the lock itself. Looks to me like it should work reliably.
To try to explain how my approach works...
(NOTE: I made a typo in my last post, my two tasks are named getHttpLock and releaseHttpLock (not setHttpLock).)
Basically, I use an integer timestamp (%TIMES, seconds since the epoch) as the lock in the global variable %HttpLock. Locks older than the configured timeout are considered expired and can be ignored.
Critical to the design is the fact that all modifications to the global variable %HttpLock are safe because they are always done with atomic "check-and-set" calls:
An: Variable Set [ Name:%HttpLock To:%lock Do Maths:Off Append:Off ] If [ <%HttpLock matches expectations> ]
Walking through my getHttpLock task:
First, I wait until the lock is either released by another task (set to 0) or expires (when current time is 5 seconds past: the current lock value plus the configured HTTP timeout). In either of these cases, it will be true that: %TIMES > %HttpLock + %lock_timeout - 1. Remember that %HttpLock contains [basically] a %TIMES timestamp value for when it was checked out, or zero if not checked out. The 5 seconds is just an added buffer to give the task that owns the lock some extra time to cleanup gracefully:
Task: getHttpLock
Run Both Together
A1: Variable Set [ Name:%lock_timeout To:%CfgHttpTimeoutSecs + 5 Do Maths:On Append:Off ]
<whileLoop>
A2: Wait Until [ MS:0 Seconds:3 Minutes:0 Hours:0 Days:0 ] If [ %TIMES > %HttpLock + %lock_timeout - 1 ]
A3: Variable Randomize [ Name:%random Min:1 Max:999 ]
A4: Variable Set [ Name:%random To:%random / 1000 Do Maths:On Append:Off ]
A5: Variable Set [ Name:%lock_time To:%TIMES Do Maths:Off Append:Off ]
A6: Variable Set [ Name:%lock To:%lock_time + %random Do Maths:On Append:Off ]
Third, I try to atomically check-and-set the lock and then check if I set it successfully. If so, I return the %lock value to the caller. If not, I assume another task beat me to it and got the lock before I could, so I start the wait loop over:
A7: Variable Set [ Name:%HttpLock To:%lock Do Maths:Off Append:Off ] If [ %lock_time > %HttpLock + %lock_timeout - 1 ]
A8: If [ %HttpLock eq %lock ]
A9: Return [ Value:%lock Stop:On ]
A10: Else
A11: Goto [ Type:Action Label Number:1 Label:whileLoop ]
<whileLoopEnd>
A12: End If
When the lock owner is done, they call releaseHttpLock and pass in the lock value that they got when they checked it out. This way, a caller can't overwrite someone else's lock in the unlikely event that they took longer than they should have to complete and their lock was expired and a new lock was checked out by someone else. This function just does an atomic check-and-set to set the global lock value back to zero if the caller's lock value matches the current global lock value:
Task: releaseHttpLock
Run Both Together
A1: Variable Set [ Name:%lock To:%par1 Do Maths:Off Append:Off ]
Multithreaded programming is hard!
I see a couple shortcomings in my approach:
1) There is a 1 in a 999 chance that two tasks competing for the lock at the same time might get the same random number and not have a unique lock value name. But it seemed "good enough" to me to not add more complexity in the design. I could use the %caller1 name as part of the lock perhaps to better ensure uniqueness, but then my check-and-set calls become more complicated to implement than simple time comparisons.
2) The 5 second buffer assumes that the lock owner will have this much time after the HTTP timeout to read the global %HTTPD and %HTTPR variables into local variables and release the lock. If they take longer than this (maybe because the CPU is overworked), another task could theoretically get a new lock and start a new HTTP call.
Thinking about your approach further, the one shortcoming I see is similar to my #2... if the Lock task completes its Wait action but the caller isn't actually done, then a new caller can start a new HTTP call and interfere with it. With your example value of 10 minutes, this is unlikely, but then it introduces a big wait time for any new HTTP calls if for some reason the caller task dies. I'd probably try it with a wait value more like the 35 seconds I'm using. But basically, then it's the same problem as in mine, or I could use a 10 minute buffer (instead of 5 seconds) and have the same effect as your scenario. It's a tradeoff between safety and having a big latency that blocks all HTTP calls if the lock owner task dies.
But I think I like how you have less code overall and avoid the random collision vulnerability.
Sal Lusso
Although I liked your idea, it doesn't seem to be working for me. The calling task always blocks on the Perform Task action until the wait inside the Lock task is done:
A2: Perform Task [ Name:Lock Priority:%priority+1 Parameter 1 (%par1): Parameter 2 (%par2): Return Value Variable:%lock Stop:Off ] If [ %TRUN !~ *,Lock,* ]
I've attached my backup.xml file to see if it works for you.
I tried invoking it two ways...
Query1 directly calls the Perform Task and Stop actions, as you wrote it:
Task: Query1
<whileLoop>
A1: Wait Until [ MS:0 Seconds:2 Minutes:0 Hours:0 Days:0 ] If [ %TRUN !~R ,httpLock, ]
A3: Perform Task [ Name:httpLock Priority:%priority + 1 Parameter 1 (%par1): Parameter 2 (%par2): Return Value Variable:%lock Stop:Off ]
A5: Goto [ Type:Action Label Number:1 Label:whileLoop ] If [ %lock neq 1 ]
A7: HTTP Get [ Server:Port:https://google.com Path: Attributes: Cookies: User Agent: Timeout:15 Mime Type: Output File: Trust Any Certificate:Off Continue Task After Error:On ]
A8: Variable Set [ Name:%httpr To:%HTTPR Do Maths:Off Append:Off ]
A10: Stop [ With Error:Off Task:httpLock ]
A12: Flash [ Text:HTTPR: %httpr Long:On ]
Query2 performs the same functionality but encapsulates the lock logic in two functions named getHttpLock and releaseHttpLock (so that I can experiment with and adjust the logic globally):
Task: Query2
A1: Perform Task [ Name:getHttpLock Priority:%priority + 1 Parameter 1 (%par1): Parameter 2 (%par2): Return Value Variable:%http_lock Stop:Off ]
A2: HTTP Get [ Server:Port:https://google.com Path: Attributes: Cookies: User Agent: Timeout:15 Mime Type: Output File: Trust Any Certificate:Off Continue Task After Error:On ]
A3: Variable Set [ Name:%httpr To:%HTTPR Do Maths:Off Append:Off ]
A5: Perform Task [ Name:releaseHttpLock Priority:%priority + 1 Parameter 1 (%par1):%http_lock Parameter 2 (%par2): Return Value Variable: Stop:Off ]
A6: Flash [ Text:HTTPR: %httpr Long:On ]
I've attached the backup.xml file. Below is the full datadescr.txt file:
============
Task: Query1
<whileLoop>
A1: Wait Until [ MS:0 Seconds:2 Minutes:0 Hours:0 Days:0 ] If [ %TRUN !~R ,httpLock, ]
<TODO DELETE>
A2: Flash [ Text:%TIMES
Requesting lock for:
%caller1 Long:On ]
A3: Perform Task [ Name:httpLock Priority:%priority + 1 Parameter 1 (%par1): Parameter 2 (%par2): Return Value Variable:%lock Stop:Off ]
<TODO DELETE>
A4: Flash [ Text:%TIMES
Lock request FAILURE for:
%caller1
LOCK: %lock Long:On ] If [ %lock neq 1 ]
A5: Goto [ Type:Action Label Number:1 Label:whileLoop ] If [ %lock neq 1 ]
<TODO DELETE>
A6: Flash [ Text:%TIMES
Lock request SUCCESS for:
%caller1
LOCK: %lock Long:On ]
A7: HTTP Get [ Server:Port:https://google.com Path: Attributes: Cookies: User Agent: Timeout:15 Mime Type: Output File: Trust Any Certificate:Off Continue Task After Error:On ]
A8: Variable Set [ Name:%httpr To:%HTTPR Do Maths:Off Append:Off ]
A9: Variable Set [ Name:%httpd To:%HTTPD Do Maths:Off Append:Off ]
A10: Stop [ With Error:Off Task:httpLock ]
<TODO DELETE>
A11: Flash [ Text:%TIMES
Lock released for:
%caller1 Long:On ]
A12: Flash [ Text:HTTPR: %httpr
%httpd Long:On ]
Task: httpLock
A1: Return [ Value:1 Stop:Off ]
A2: Wait [ MS:0 Seconds:30 Minutes:0 Hours:0 Days:0 ]
Task: Query2
A1: Perform Task [ Name:getHttpLock Priority:%priority + 1 Parameter 1 (%par1): Parameter 2 (%par2): Return Value Variable:%http_lock Stop:Off ]
A2: HTTP Get [ Server:Port:https://google.com Path: Attributes: Cookies: User Agent: Timeout:15 Mime Type: Output File: Trust Any Certificate:Off Continue Task After Error:On ]
A3: Variable Set [ Name:%httpr To:%HTTPR Do Maths:Off Append:Off ]
A4: Variable Set [ Name:%httpd To:%HTTPD Do Maths:Off Append:Off ]
A6: Flash [ Text:HTTPR: %httpr
%httpd Long:On ]
Task: getHttpLock (232)
Run Both Together
<whileLoop>
A1: Wait Until [ MS:0 Seconds:2 Minutes:0 Hours:0 Days:0 ] If [ %TRUN !~R ,httpLock, ]
<TODO DELETE>
A2: Flash [ Text:%TIMES
Requesting lock for:
%caller1 Long:On ]
A3: Perform Task [ Name:httpLock Priority:%priority + 1 Parameter 1 (%par1): Parameter 2 (%par2): Return Value Variable:%lock Stop:Off ]
<TODO DELETE>
A4: Flash [ Text:%TIMES
Lock request FAILURE for:
%caller1
LOCK: %lock Long:On ] If [ %lock neq 1 ]
A5: Goto [ Type:Action Label Number:1 Label:whileLoop ] If [ %lock neq 1 ]
<TODO DELETE>
A6: Flash [ Text:%TIMES
Lock request SUCCESS for:
%caller1
LOCK: %lock Long:On ]
Task: releaseHttpLock
Run Both Together
<TODO DELETE>
A2: Flash [ Text:%TIMES
Lock released for:
%caller1 Long:On ]
Rich D
> Although I liked your idea, it doesn't seem to be working for me. The calling task always blocks on the Perform Task action until the wait inside the Lock task is done:
Oh crap.. I forgot there is a mechanism in place that ensures a child task with a higher priority then the parent will finish before the parent is allowed to continue. Without the parent/child relation whenever a wait is encountered, lower priority tasks are allowed to run (this is what I thought would happen).
So unfortunately we will have to introduce another task for this approach to. Work. The child task will have to start the lock task then when the child task stops the parent can continue.
Here is what I have so far..
Http Get (781)
<start loop>
A1: Wait Until [ MS:0 Seconds:2 Minutes:0 Hours:0 Days:0 ] If [ %TRUN !~ *,Lock*,* ]
A2: Perform Task [ Name:Lock Priority:%priority Parameter 1 (%par1): Parameter 2 (%par2): Return Value Variable:%lock Stop:Off ] If [ %TRUN !~ *,Lock*,* ]
A3: Goto [ Type:Action Label Number:1 Label:start loop ] If [ %lock !~ true ]
A4: [X] HTTP Get [ Server:Port:%test Path: Attributes: Cookies: User Agent: Timeout:10 Mime Type: Output File: Trust Any Certificate:Off ]
A5: Flash [ Text:HTTP GET Long:On ]
A6: Wait [ MS:0 Seconds:3 Minutes:0 Hours:0 Days:0 ]
A7: Variable Set [ Name:%set_your_variables To:%HTTPD Do Maths:Off Append:Off ]
A8: Stop [ With Error:Off Task:Lock 2 ]
A9: Flash [ Text:DONE Long:On ]
Lock (782)
A1: Return [ Value:true Stop:Off ]
A2: Flash [ Text:After return Long:On ]
A3: [X] Wait [ MS:0 Seconds:10 Minutes:0 Hours:0 Days:0 ]
A4: [X] Flash [ Text:After wait Long:On ]
A5: Perform Task [ Name:Lock 2 Priority:%priority-1 Parameter 1 (%par1): Parameter 2 (%par2): Return Value Variable: Stop:On ]
Lock 2 (784)
A1: Flash [ Text:Lock 2 Long:On ]
A2: Wait [ MS:0 Seconds:10 Minutes:0 Hours:0 Days:0 ]
A3: Flash [ Text:Lock 2 done Long:On ]
Rich D
Sorry if my original post was not clear. I should have stated it was just a untested idea. My second approach has not been thoroughly tested either. It is just my first thought on how to over come the parent/child relation problem. There could be a more elegant way to do this however my tasker time is very limited at the moment.
Rich D
Ok, I snuck in a little tasker time ... :)
Here is a better solution. You just need to have the lock task start another iteration of itself and set the collision to abort existing .
Again not entirely tested but seems to be working..
Http Get (781)
<start loop>
A1: Wait Until [ MS:0 Seconds:2 Minutes:0 Hours:0 Days:0 ] If [ %TRUN !~ *,Lock,* ]
A2: Perform Task [ Name:Lock Priority:%priority Parameter 1 (%par1): Parameter 2 (%par2): Return Value Variable:%lock Stop:Off ] If [ %TRUN !~ *,Lock,* ]
A3: Goto [ Type:Action Label Number:1 Label:start loop ] If [ %lock !~ true ]
A4: [X] HTTP Get [ Server:Port:%test Path: Attributes: Cookies: User Agent: Timeout:10 Mime Type: Output File: Trust Any Certificate:Off ]
A5: Flash [ Text:HTTP GET Long:On ]
A6: Wait [ MS:0 Seconds:3 Minutes:0 Hours:0 Days:0 ]
A7: Variable Set [ Name:%set_your_variables To:%HTTPD Do Maths:Off Append:Off ]
A8: Stop [ With Error:Off Task:Lock ]
A9: Flash [ Text:DONE Long:On ]
Lock (782)
Abort Existing Task
A1: Return [ Value:true Stop:Off ]
A2: Flash [ Text:After return Long:On ]
A3: Wait [ MS:0 Seconds:10 Minutes:0 Hours:0 Days:0 ] If [ %par1 ~ set ]
A4: Flash [ Text:After lock wait Long:On ]
A5: [X] Flash [ Text:After wait Long:On ]
A6: Perform Task [ Name:Lock Priority:%priority-1 Parameter 1 (%par1):set Parameter 2 (%par2): Return Value Variable: Stop:On ] If [ %par1 !~ set ]
Sal Lusso
So, I tried out this new approach, and though it wasn't *blocking* anymore, it seems that the child instance of the Lock task wasn't actually getting launched, which allowed me to run multiple concurrent calls in my test.
I had to change it up a bit and put the Perform Task before the Return (and change collision handling to Run Both Together as a result).
This seems to be working for me like this:
Task: getHttpLockTask
Run Both Together
<whileLoop>
A1: Wait Until [ MS:0 Seconds:3 Minutes:0 Hours:0 Days:0 ] If [ %TRUN !~R ,httpLock, ]
A2: Perform Task [ Name:httpLock Priority:%priority Parameter 1 (%par1): Parameter 2 (%par2): Return Value Variable:%lock Stop:Off ] If [ %TRUN !~R ,httpLock, ]
A3: Goto [ Type:Action Label Number:1 Label:whileLoop ] If [ %lock neq 1 ]
A4: Return [ Value:%lock Stop:On ]
Task: httpLock
Run Both Together
A1: If [ %par1 eq child ]
A2: Wait [ MS:0 Seconds:%CfgHttpTimeoutSecs + 15 Minutes:0 Hours:0 Days:0 ]
A3: Else
A4: Perform Task [ Name:httpLock Priority:%priority - 1 Parameter 1 (%par1):child Parameter 2 (%par2): Return Value Variable: Stop:Off ]
A5: Return [ Value:1 Stop:On ]
A6: End If
Task: releaseHttpLockTask (235)
Run Both Together
A1: Stop [ With Error:Off Task:httpLock ]
I've attached a project XML to run all this with some test tasks.
Sal Lusso
If you run "Query Timeout" and then "Query" immediately (in the test Lock.prj.xml file I uploaded in last post), you will see that Query correctly blocks until the httpLock task expires. Then Query finishes and "flashes" a notice that it kills the lock task... all good. BUT THEN, the Query Timeout task flashes that it is killing the lock task! If this were to happen in a real app context, it could kill another task's lock that it obtained. It's a design flaw I didn't spot before that the method of releasing a lock is globally killing any lock task that is running. This means you could release a lock even if it isn't yours.
This problem, plus the complexity that is now arising with all of the concurrent tasks that are getting spawned is making me think to return to my original approach which just manages the lock via a global variable. I have an idea to get rid of the random number collision possibility, even if it is very unlikely. Will share...
Sal Lusso
I made some modifications to your tasks and got it to work without blocking, but there is still the problem I noted in my last post that a long-running task that finishes unexpectedly late can kill the lock task of a task that ran after it. For this reason, plus the unwieldiness of needing so many tasks to run concurrently to implement the lock and the resulting difficulty to track their priorities and make sure everything is working correctly, I went back to my original idea of using a lock var but one that has a timestamp plus the caller's name in it. This way, each task can be sure it doesn't overwrite someone else's lock.
I have attached a project that provides a test harness to test each of the 4 recipes/solutions that have been presented here. To run it, just import the Lock Tests project and open the Configure Tests tasks. Set the %method var to one of the following to use the corresponding recipe/solution:
1) "Var1" -- my original variable-based lock that uses a timestamp and random "identifier" component.
2) "Var2" -- my latest approach using a variable-based lock with a timestamp and caller name identifier component. (Details to be provided in next post.)
3) "Task1" -- your latest posting using a lock task that launches a child task.
4) "Task2" -- my minor modifications to your latest posting that appear to solve the blocking issue when a task runs unexpectedly long.
After running the "Configure Tests" task, one can run the "Query Test" task to see how a short HTTP Get would behave (5 seconds long). The "Query Test Timeout" task runs a HTTP Get that simulates an unexpectedly long running task (75 seconds). In the test harness, 30 seconds is considered the max expected HTTP timeout.
The best test scenario is to run "Query Test Timeout" and then "Query Test" as quickly as possible one after the other. The Flash notifications show how the lock is behaving.
My "Task2" modifications to your recipe look like this:
Task: requestHttpLockTask2
Run Both Together
<whileLoop>
A1: Wait Until [ MS:0 Seconds:3 Minutes:0 Hours:0 Days:0 ] If [ %TRUN !~R ,httpLockTask2, ]
A2: Perform Task [ Name:httpLockTask2 Priority:%priority Parameter 1 (%par1): Parameter 2 (%par2): Return Value Variable:%lock Stop:Off ] If [ %TRUN !~R ,httpLockTask2, ]
<whileLoopEnd>
A3: Goto [ Type:Action Label Number:1 Label:whileLoop ] If [ %lock neq 1 ]
A4: Return [ Value:%lock Stop:On ]
Run Both Together
A1: If [ %par1 eq child ]
A2: Wait [ MS:0 Seconds:%CfgHttpTimeoutSecs + 15 Minutes:0 Hours:0 Days:0 ]
A4: Else
A5: Perform Task [ Name:httpLockTask2 Priority:%priority - 1 Parameter 1 (%par1):child Parameter 2 (%par2): Return Value Variable: Stop:Off ]
A6: Return [ Value:1 Stop:On ]
A7: End If
Task: releaseHttpLockTask2
Run Both Together
A1: Stop [ With Error:Off Task:httpLockTask2 ]
Sal Lusso
I have attached the supporting tasks to this post in a project file (Http_Lock.prj.xml).
Here is an example of how all HTTP Get calls in a Tasker project should be wrapped with calls to the requestHttpLock and releaseHttpLock tasks:
Task: Sample Http Get
A1: Perform Task [ Name:requestHttpLock Priority:%priority Parameter 1 (%par1): Parameter 2 (%par2): Return Value Variable:%http_lock Stop:Off ]
A2: HTTP Get [ Server:Port:https://www.google.com Path: Attributes: Cookies: User Agent: Timeout:%CfgHttpTimeoutSecs Mime Type: Output File: Trust Any Certificate:Off Continue Task After Error:On ]
A3: Variable Set [ Name:%httpr To:%HTTPR Do Maths:Off Append:Off ]
A5: Perform Task [ Name:releaseHttpLock Priority:%priority Parameter 1 (%par1):%http_lock Parameter 2 (%par2): Return Value Variable: Stop:Off ]
... Do whatever you need to
The requestHttpLock generates a lock value consisting of "%TIMEMS;%caller1". It then reads the current value of %HttpLock into a local variable and parses out the timestamp component. It compares the global lock timestamp to the current time, considering any lock more than 15 seconds older than a configured HTTP timeout (%CfgHttpTimeoutSecs) as stale and able to be ignored. If the existing %HttpLock is 0 (meaning no lock is set) or old enough to be ignored, it tries to set the lock to its new value in an atomic check-and-set call that also makes sure the global %HttpLock hasn't changed since it was read and parsed (step A7). This runs in a loop so that if the lock is taken, it waits a few seconds and tries again. When the lock is successfully set, it returns the new lock value to the caller. To release the lock, a task just calls releaseHttpLock, passing the lock value it obtained from requestHttpLock. The releaseHttpLock does a check-and-set to reset the lock to zero if the lock parameter received (%par1) matches the current %HttpLock value.
Task: requestHttpLock
Run Both Together
A1: Variable Set [ Name:%lock_timeout To:(%CfgHttpTimeoutSecs + 15) * 1000 Do Maths:On Append:Off ]
<whileLoop>
A2: Variable Set [ Name:%lock_time To:%TIMEMS Do Maths:Off Append:Off ]
A3: Variable Set [ Name:%lock To:%lock_time;%caller1 Do Maths:Off Append:Off ]
A4: Variable Set [ Name:%global_lock To:0 Do Maths:Off Append:Off ]
A5: Variable Set [ Name:%global_lock To:%HttpLock Do Maths:Off Append:Off ] If [ %HttpLock ~R ^[0-9]+; ]
A6: Variable Search Replace [ Variable:%global_lock Search:^[0-9]+ Ignore Case:Off Multi-Line:Off One Match Only:Off Store Matches In:%global_lock_time Replace Matches:Off Replace With: ]
A7: Variable Set [ Name:%HttpLock To:%lock Do Maths:Off Append:Off ] If [ %lock_time > %global_lock_time1 + %lock_timeout - 1 & %HttpLock eq %global_lock ]
A8: If [ %HttpLock eq %lock ]
A10: Else
A11: Wait [ MS:0 Seconds:3 Minutes:0 Hours:0 Days:0 ]
A12: Goto [ Type:Action Label Number:1 Label:whileLoop ]
<whileLoopEnd>
A13: End If
Task: releaseHttpLock
Run Both Together
Rich D
I have come up with the following solution
Nice work, that looks like a great solution and very creative. Probably to best to stick with that.
I believe I was able to get the %TRUN approach working as well. There seems to be a bug when trying to invoke another iteration of the same task from within that task. (The same problem you had and ended up going to a 'run both together' solution.
The solution is to put the call as the last action in task (weird huh..) That one took a while to figure out... :(
If I get some time I will test and post just for a alternative. I will add a identifier to solve the global lock release issue. On that note I noticed you. Are using %TIMESMS and %caller. I had been thinking that just using %TIMESMS as the time stamp and identifier would suffice. Theory being 2 actions could not be completed within 1 ms. What are your thoughts on that?
BTW.. are you using the run log for debugging? This is a invaluable tool. Menu / more / run log.
Sal Lusso
I noticed you. Are using %TIMEMS and %caller1. I had been thinking that just using %TIMEMS as the time stamp and identifier would suffice. Theory being 2 actions could not be completed within 1 ms. What are your thoughts on that?
That would certainly make things easier!!! But I wasn't sure how fast things could theoretically run and if there could be a collision within a millisecond. So I built it to be super-defensive at the cost of a bit more complexity.
BTW.. are you using the run log for debugging? This is a invaluable tool. Menu / more / run log.
No, but it would probably be worth my investigating. Debugging with these flash commands is not the easiest! Thanks.
Rich D
> That would certainly make things easier!!! But I wasn't sure how fast things could theoretically run and if there could be a collision within a millisecond. So I built it to be super-defensive at the cost of a bit more complexity.
Gotcha..
>> BTW.. are you using the run log for debugging? This is a invaluable tool. Menu / more / run log.
>
>
> No, but it would probably be worth my investigating. Debugging with these flash commands is not the easiest! Thanks.
>
Their is some good info in the docs about the run log but it has a few quirks as well.
1. All actions are logged when they are 'compleated' so for example a wait action will not be entered 'shown' until after the wait is over.
2. For some reason ( most likely same as above) when there is a 'perform task' action, the invoked task will be shown as running before the preform task action is shown.
Rich D
OK, here is the working tested %TRUN approach. I included a ID for the locking task. The lock task sets a global variable to %caller(:) and returns that data to the calling task. Then this global variable is tested before the unlock task is called.
I was not sure why you had the extra 'getHttpLockTask' instead of just calling 'httpLock' directly from 'Query'. My guess is for test in purposes. In any event I left
'getHttpLockTask' and tested. All seems to work as expected.
Query (789)
Run Both Together
A1: Variable Set [ Name:%CfgHttpTimeoutSecs To:30 Do Maths:Off Append:Off ]
A2: Perform Task [ Name:getHttpLockTask Priority:%priority Parameter 1 (%par1): Parameter 2 (%par2): Return Value Variable:%http_lock Stop:Off ]
A3: Flash [ Text:HTTP GET Long:On ]
A4: Wait [ MS:0 Seconds:15 Minutes:0 Hours:0 Days:0 ]
A5: Flash [ Text:Unlocking
%http_lock Long:On ] If [ %http_lock ~ %Current_Lock ]
A6: Flash [ Text:Unlock failed
%http_lock
%Current_lock Long:On ] If [ %http_lock !~ %Current_Lock ]
A7: Perform Task [ Name:releaseHttpLockTask Priority:%priority Parameter 1 (%par1):%http_lock Parameter 2 (%par2): Return Value Variable: Stop:Off ] If [ %http_lock ~ %Current_Lock ]
Query Timeout (790)
A1: Variable Set [ Name:%CfgHttpTimeoutSecs To:30 Do Maths:Off Append:Off ]
A2: Perform Task [ Name:getHttpLockTask Priority:%priority Parameter 1 (%par1): Parameter 2 (%par2): Return Value Variable:%http_lock Stop:Off ]
A3: Flash [ Text:HTTP GET TIMEOUT Long:On ]
A4: Wait [ MS:0 Seconds:15 Minutes:1 Hours:0 Days:0 ]
A5: Flash [ Text:Unlocking
%http_lock Long:On ] If [ %http_lock ~ %Current_Lock ]
A6: Flash [ Text:Unlock failed
%http_lock
%Current_lock Long:On ] If [ %http_lock !~ %Current_Lock ]
A7: Perform Task [ Name:releaseHttpLockTask Priority:%priority Parameter 1 (%par1):%http_lock Parameter 2 (%par2): Return Value Variable: Stop:Off ] If [ %http_lock ~ %Current_Lock ]
A8: [X] Perform Task [ Name:releaseHttpLockTask Priority:%priority Parameter 1 (%par1):%http_lock Parameter 2 (%par2): Return Value Variable: Stop:Off ]
httpLock (791)
Run Both Together
<TODO DELETE>
A1: Flash [ Text:%TIMES
Running httpLock: %par1 Long:On ]
A2: If [ %par1 eq child ]
A3: Flash [ Text:%caller(:) Long:On ]
A4: Wait [ MS:0 Seconds:%CfgHttpTimeoutSecs + 15 Minutes:0 Hours:0 Days:0 ]
A5: [X] Wait [ MS:0 Seconds:2 Minutes:0 Hours:0 Days:0 ]
A6: Variable Clear [ Name:%Current_Lock Pattern Matching:Off ]
A7: Flash [ Text:Lock time expired Long:On ]
A8: Else
A9: Return [ Value:%caller(:) Stop:Off ]
A10: Variable Set [ Name:%Current_Lock To:%caller(:) Do Maths:Off Append:Off ]
A11: Perform Task [ Name:httpLock Priority:%priority - 1 Parameter 1 (%par1):child Parameter 2 (%par2): Return Value Variable: Stop:On ]
A12: End If
releaseHttpLockTask (792)
Run Both Together
A1: Stop [ With Error:Off Task:httpLock ]
<TODO DELETE>
A2: Flash [ Text:%TIMES
Lock released for:
%caller1 Long:On ]
getHttpLockTask (793)
Run Both Together
<TODO DELETE>
A1: [X] Flash [ Text:%TIMES
Requesting lock for:
%caller1 Long:On ]
<whileLoop>
A2: Wait Until [ MS:0 Seconds:3 Minutes:0 Hours:0 Days:0 ] If [ %TRUN !~R ,httpLock, ]
A3: Perform Task [ Name:httpLock Priority:%priority Parameter 1 (%par1): Parameter 2 (%par2): Return Value Variable:%lock Stop:Off ] If [ %TRUN !~R ,httpLock, ]
<TODO DELETE>
A4: [X] Flash [ Text:%TIMES
Lock request FAILURE for:
%caller1
LOCK: %lock Long:On ] If [ %lock neq 1 ]
A5: Goto [ Type:Action Label Number:1 Label:whileLoop ] If [ %lock !Set ]
<TODO DELETE>
A6: [X] Flash [ Text:%TIMES
Lock request SUCCESS for:
%caller1
LOCK: %lock Long:On ]
A7: Return [ Value:%lock Stop:On ]
Rich D
I cleaned this version up for others. It now calls the lock directly from the 'Query' task. Here is a explanation.
This uses a running task ( httpLock ) for the lock. Instead of setting and checking a global variable this tests the tasker system variable %TRUN. If the task is running then the lock is set.
Query (789)
Run Both Together
A1: Variable Set [ Name:%CfgHttpTimeoutSecs To:30 Do Maths:Off Append:Off ]
// this sets the time out for the lock in case the calling task does not release the lock for some unforseen reason. //
<whileLoop>
A2: Wait Until [ MS:0 Seconds:3 Minutes:0 Hours:0 Days:0 ] If [ %TRUN !~R ,httpLock, ]
// this will loop until the task ' httpLock ' is not running//
A3: Perform Task [ Name:httpLock Priority:%priority Parameter 1 (%par1): Parameter 2 (%par2): Return Value Variable:%http_lock Stop:Off ] If [ %TRUN !~R ,httpLock, ]
// this starts the 'httpLock' task and sets the return variable, only if the task is not running//
A4: Goto [ Type:Action Label Number:1 Label:whileLoop ] If [ %http_lock !Set ]
// this is a handshake from the lock task. Because there is a very slight chance that another task has set the lock in between actions A2 and A3. It checks the return variable is set. If it is not set that means the perform task action has failed and it will start the wait loop again //
A5: Flash [ Text:Test --HTTP GET action Long:On ]
A6: Wait [ MS:0 Seconds:15 Minutes:0 Hours:0 Days:0 ]
// this is to simulate the 'HTTP GET' action. //
A7: Perform Task [ Name:releaseHttpLockTask Priority:%priority Parameter 1 (%par1):%http_lock Parameter 2 (%par2): Return Value Variable: Stop:Off ] If [ %http_lock ~ %Current_Lock ]
// this starts the release task 'only' if %Current_Lock matches the value of this particular task. This test is necessary in case the lock has expired and has been reset by another task IE this will only cancel a lock that was set by this iteration of this task. There are 2 parts to the %Current_Lock. Fist is the %caller ID which identifies the calling task name and second is %TIMEMS at which the lock was set. This is to separate it from concurrent iterations of the same calling task. //
-------------------------
This is the lock task. To get this approach to work it is necessary to have both the child and parent task run together. There is currently a special parent / child relationship that prohibits the parent from running until the child task (of equal or greater priority) is completed (this includes even if the child encounters a wait action). To get around this it is necessary for the child to start another iteration of itself.
httpLock (791)
Abort Existing
A1: Variable Set [ Name:%Current_Lock To:%caller(:)|%TIMEMS Do Maths:Off Append:Off ] If [ %par1 !~ child ]
// This sets %Current_Lock as described above. It will only set on the first instance of the task //
A2: If [ %par1 ~ child ]
// checks to see if this is the second instance of the task and starts the time out wait //
A3: Flash [ Text:Lock set for--
%Current_Lock Long:On ]
A4: Wait [ MS:0 Seconds:%CfgHttpTimeoutSecs + 15 Minutes:0 Hours:0 Days:0 ]
// This is the time out wait. In case the lock is not properly released in x min //
A5: Variable Clear [ Name:%Current_Lock Pattern Matching:Off ]
A6: Flash [ Text:Lock time expired Long:On ]
A7: Else
A8: Return [ Value:%Current_Lock Stop:Off ]
// this returns the lock code to the calling task to confirm the lock and set the lock code in the caller //
A9: Perform Task [ Name:httpLock Priority:%priority - 1 Parameter 1 (%par1):child Parameter 2 (%par2): Return Value Variable: Stop:On ]
// Starts the second iteration of the lock task and stops the first iteration to release the 'parent/ child' condition. **At this point in time there appears to be a bug in taskers task scheduling. This action will only work correctly if it is the last action in the task. This only applys to a task starting another iteration of itself with the collision set to 'abort Existing'** //
A10: End If
// the 'end if' action does not seem to trigger the bug but other actions will //
releaseHttpLockTask (792)
Run Both Together
A1: Stop [ With Error:Off Task:httpLock ]
// stops the lock task which releases the lock //
<TODO DELETE>
A2: Flash [ Text:%TIMES
Lock released for:
%par1 Long:On ]
Unedited descriptions..
Query (789)
Run Both Together
A1: Variable Set [ Name:%CfgHttpTimeoutSecs To:30 Do Maths:Off Append:Off ]
<whileLoop>
A2: Wait Until [ MS:0 Seconds:3 Minutes:0 Hours:0 Days:0 ] If [ %TRUN !~R ,httpLock, ]
A3: Perform Task [ Name:httpLock Priority:%priority Parameter 1 (%par1): Parameter 2 (%par2): Return Value Variable:%http_lock Stop:Off ] If [ %TRUN !~R ,httpLock, ]
A4: Goto [ Type:Action Label Number:1 Label:whileLoop ] If [ %http_lock !Set ]
A5: Flash [ Text:Test --HTTP GET action Long:On ]
A6: Wait [ MS:0 Seconds:15 Minutes:0 Hours:0 Days:0 ]
A7: Perform Task [ Name:releaseHttpLockTask Priority:%priority Parameter 1 (%par1):%http_lock Parameter 2 (%par2): Return Value Variable: Stop:Off ] If [ %http_lock ~ %Current_Lock ]
httpLock (791)
Abort Existing Task
A1: Variable Set [ Name:%Current_Lock To:%caller(:)|%TIMEMS Do Maths:Off Append:Off ] If [ %par1 !~ child ]
A2: If [ %par1 ~ child ]
A3: Flash [ Text:Lock set for--
%Current_Lock Long:On ]
A4: Wait [ MS:0 Seconds:%CfgHttpTimeoutSecs + 15 Minutes:0 Hours:0 Days:0 ]
A5: Variable Clear [ Name:%Current_Lock Pattern Matching:Off ]
A6: Flash [ Text:Lock time expired Long:On ]
A7: Else
A8: Return [ Value:%Current_Lock Stop:Off ]
A9: Perform Task [ Name:httpLock Priority:%priority - 1 Parameter 1 (%par1):child Parameter 2 (%par2): Return Value Variable: Stop:On ]
A10: End If
releaseHttpLockTask (792)
Run Both Together
A1: Stop [ With Error:Off Task:httpLock ]
<TODO DELETE>
A2: Flash [ Text:%TIMES
Lock released for:
%par1 Long:On ]
Sal Lusso
The original reason I created the getHttpLock and releaseHttpLock task methods was for encapsulation... so that I could change the method implementations easily if I found any problems without affecting the calling task instances. Yes, it was useful for testing as well when I created a test harness using a dynamic version of these methods that would invoke one of several possible implementations based on a configuration.
Rich D
> Cool. Thanks for coming back to update your idea, Rich D.
It was more of a stubborn need to finish what I started... :)
>
> The original reason I created the getHttpLock and releaseHttpLock task methods was for encapsulation... so that I could change the method implementations easily if I found any problems without affecting the calling task instances.
That makes sense...
BTW.. I noticed the project I posted with the getHttpLock still had the 'httpLock' task collision set to 'run both together'. That should be changed to 'abort existing' .