Does PeriodicCallback ensure no parallel executions?

[David]

One question regarding the PeriodicCallbacks: Is the callback triggered while a previous triggered one is still running or does it ensure that there is only one callback running at a time? As far as I can see the later one is true, but I want to be sure!

[A. Jesse Jiryu Davis]

PeriodicCallback does not ensure only one callback is running. It just executes the callback every N seconds (roughly). If that callback initiates a chain of callbacks that's still in progress N seconds later, PeriodicCallback doesn't know or care, it just executes your callback again.

IMO a simpler and more reliable pattern is:

@gen.coroutine
def periodic():

    loop = IOLoop.current()

    while True:

        start = loop.time()

        yield do_some_coroutine()  # Do work.

        duration = loop.time() - start

        yield gen.Task(loop.add_timeout, max(N - duration, 0))

This runs do_some_coroutine(), only one instance at a time, every N seconds or as fast as possible if do_some_coroutine() takes over N seconds to run.

On Thu, Apr 18, 2013 at 12:08 PM, David <davidne...@gmail.com> wrote:

One question regarding the PeriodicCallbacks: Is the callback triggered while a previous triggered one is still running or does it ensure that there is only one callback running at a time? As far as I can see the later one is true, but I want to be sure!

--
You received this message because you are subscribed to the Google Groups "Tornado Web Server" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python-tornad...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Russ Weeks]

But, doesn't the fact that all these callbacks are running in the context of the IOLoop ensure that no two callbacks will ever be run in parallel?

-Russ

[Ben Darnell]

The IOLoop is single threaded, so only one callback is running at a time.  As long as your PeriodicCallback is synchronous, you're fine.  If you're kicking off an asynchronous routine, PeriodicCallback doesn't know anything about that so it will happily start up another copy the next time the timer is hit.  In that case you'd need an async-aware equivalent of PeriodicCallback like the one Jesse posted.

If your callback runs (synchronously) for longer than the PeriodicCallback's interval, we skip runs rather than run twice in a row next time.

-Ben

On Thu, Apr 18, 2013 at 12:08 PM, David <davidne...@gmail.com> wrote:

One question regarding the PeriodicCallbacks: Is the callback triggered while a previous triggered one is still running or does it ensure that there is only one callback running at a time? As far as I can see the later one is true, but I want to be sure!

--

[A. Jesse Jiryu Davis]

Like Ben said. To clarify for those who aren't familiar with coroutines:

def start_fetch():

    client = AsyncHTTPClient()

    client.fetch('http://example.com', callback=on_fetch)

def on_fetch(response):

    print response

PeriodicCallback(start_fetch, callback_time=1).start()

If example.com is slow and the response is passed to on_fetch more than 1 second after start_fetch begins it, the PeriodicCallback doesn't know that, and it will start another start_fetch 1 second later anyway.

[Serge S. Koval]

No, by default PeriodicCallback is not overrun safe.

You can customize it by making PeriodicCallback aware about wrapped asynchronous function, check this post: https://groups.google.com/d/msg/python-tornado/p6uuyf9ULF4/3NF6BLCoDZUJ

[David]

Thank you for the detailed feedback. My question was of course unprecise. It obviously cannot run parallel if it is synchronous. One further question on the synchronous case:

Assume the callback_time is N and a callback is triggered and runs N+1 seconds. Is the next callback triggered at about

1. 2 * N seconds (after the first started),

2. N + 1 seconds (after the first started) or
4.  2 * N + 1 seconds (after the first started)?

[Serge S. Koval]

PeriodicCallback schedules next execution after callback function is exited. 

Here's sequence: N, 2*N + T1, 3*N + T1 + T2, ...

N - callback time

Tx - time spent in the callback function

If function does not block IOLoop (it is asynchronous), then you can think that its immediate execution time T is close to 0. First run will be in N seconds, second run in 2*N and so on.

If function blocks IOLoop for T seconds for each run, then those seconds will add up and PeriodicCallback won't be as precise as requested.

If you don't want to start next run before previous finished, you need to use customized version of the PeriodicCallback.

--

[Ben Darnell]

On Fri, Apr 19, 2013 at 7:18 AM, Serge S. Koval <serge...@gmail.com> wrote:

PeriodicCallback schedules next execution after callback function is exited. 

Here's sequence: N, 2*N + T1, 3*N + T1 + T2, ...

N - callback time

Tx - time spent in the callback function

This used to be true, but not any more.  The time spent in the callback (Tx) is now subtracted from the timeout to keep the start-to-start time as close to N seconds as possible.  So if you have a 60-second periodic callback started at 12:00:00, it will run at 12:01:00, 12:02:00, and so on, no matter how long each call takes as long as it is less than 60 seconds.  If it takes more than that, callbacks will be skipped and then the next callback will be run at the next regular time (i.e. if the first callback takes 61 seconds, the second one will be run at 120 seconds).

-Ben

[Serge S. Koval]

I see. Good to know, overlooked this in change log.

[aliane abdelouahab]

what is the callback is a function that takes for time, say if
datetime.datetime.now().hour == 18 then publish something, and then
PeriodicCallback is set to 24 hours, and what if it can do it at this
time (for example there is no internet at 18h or the server goes
down) ?

On 19 avr, 14:27, Ben Darnell <b...@bendarnell.com> wrote:

> On Fri, Apr 19, 2013 at 7:18 AM, Serge S. Koval <serge.ko...@gmail.com>wrote:
>
> > PeriodicCallback schedules next execution after callback function is
> > exited.
>
> > Here's sequence: N, 2*N + T1, 3*N + T1 + T2, ...
> > N - callback time
> > Tx - time spent in the callback function
>
> This used to be true, but not any more.  The time spent in the callback
> (Tx) is now subtracted from the timeout to keep the start-to-start time as
> close to N seconds as possible.  So if you have a 60-second periodic
> callback started at 12:00:00, it will run at 12:01:00, 12:02:00, and so on,
> no matter how long each call takes as long as it is less than 60 seconds.
>  If it takes more than that, callbacks will be skipped and then the next
> callback will be run at the next regular time (i.e. if the first callback
> takes 61 seconds, the second one will be run at 120 seconds).
>
> -Ben
>
>
>
>
>
>
>
>
>
> > If function does not block IOLoop (it is asynchronous), then you can think
> > that its immediate execution time T is close to 0. First run will be in N
> > seconds, second run in 2*N and so on.
>
> > If function blocks IOLoop for T seconds for each run, then those seconds
> > will add up and PeriodicCallback won't be as precise as requested.
>
> > If you don't want to start next run before previous finished, you need to
> > use customized version of the PeriodicCallback.
>

> > On Fri, Apr 19, 2013 at 1:33 PM, David <davidnelles...@gmail.com> wrote:
>
> >> Thank you for the detailed feedback. My question was of course unprecise.
> >> It obviously cannot run parallel if it is synchronous. One further question
> >> on the synchronous case:
> >> Assume the callback_time is N and a callback is triggered and runs N+1
> >> seconds. Is the next callback triggered at about
> >> 1. 2 * N seconds (after the first started),
> >> 2. N + 1 seconds (after the first started) or
> >> 4.  2 * N + 1 seconds (after the first started)?
>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "Tornado Web Server" group.
> >> To unsubscribe from this group and stop receiving emails from it, send an
> >> email to python-tornad...@googlegroups.com.

> >> For more options, visithttps://groups.google.com/groups/opt_out.

[Ben Darnell]

On Fri, Apr 19, 2013 at 6:45 PM, aliane abdelouahab <alabde...@gmail.com> wrote:

what is the callback is a function that takes for time, say if
datetime.datetime.now().hour == 18 then publish something, and then
PeriodicCallback is set to 24 hours, and what if it can do it at this
time (for example there is no internet at 18h or the server goes
down) ?

A 24h PeriodicCallback will run 24h from the time the server started; there's no way to tie it to a particular real-world time.  If you want to be able to retry sooner on failure, you'll need to keep track of the status (e.g. in a database).  I would suggest storing the next scheduled time in a database and use a PeriodicCallback that runs every hour.  Most of the time it will just look at the database, see that the time hasn't come yet, and return immediately.  When it finishes successfully, write the next day's time into the database.

-Ben

[aliane abdelouahab]

ah, thank you, i'll try to implelement it :)

On 23 avr, 04:13, Ben Darnell <b...@bendarnell.com> wrote:
> On Fri, Apr 19, 2013 at 6:45 PM, aliane abdelouahab <alabdeloua...@gmail.com

[Ben Darnell]

On Tue, Mar 20, 2018 at 9:17 AM Luca Palombella <luca.pal...@gmail.com> wrote:

i don't understand this line : "yield gen.Task(loop.add_timeout, max(N - duration, 0))".

In more recent versions of Tornado, you can also write this

   yield gen.sleep(max(N - duration, 0))

The max() function call simply ensures that you don't get a negative number (I think gen.sleep() will actually work just fine anyway if you give it a negative number). So if N=5 seconds and do_some_coroutine takes 2 seconds, this will sleep for 3 seconds before looping to run do_some_coroutine again(). 

The end result is that as long as do_some_coroutine takes less than 5 seconds, one call will start every 5 seconds. If it ever takes longer, the next call will start immediately after the previous call finishes. 

-Ben

 

Can someone explain me please? If duration is less than N=5 seconds (for example 2 seconds) so when is the next task of do_some_coroutine()? 3 seconds?

Trying the code it seems that start immediately, without waiting 3 seconds...

thanks

For more options, visit https://groups.google.com/d/optout.