akka scheduler when scheduling tasks to be executed after a long delay

[Steve Buzzard]

We're currently running into an issue when we have an Akka actor - in this case an FSM actor - that acts as a scheduler for starting and stopping Free Previews as part of a cable system application.  These events are ultimately scheduled via a setTimer.  We noticed in testing that when scheduling a free preview start many months in advance (which in turn would cause a very long delay for the timer task) that we hit a point after which the timer task would execute immediately.  At first we looked to ensure we weren't accidentally causing an overflow issue where we might be creating durations with negative values but quickly determined that we were not - the code is pretty simple (the endEvent in the snipplet below uses joda-time Interval's and interval.getEnd.getMillis is a long milliseconds since epoch as is obviously System.currentTImeMillis):

...

import scala.concurrent.duration._

...

 private def addPendingEndedEvent(fp: FreePreview, data: Data): Unit = {

    trace(s"FreePreviewScheduler.addPendingEndedEvent with fp: $fp.")

    for (i <- fp.interval) setTimer(fp.id.toString, FreePreviewEndedTimerEvent(fp.id, data.updateId), fromNow(i.getEnd), false)

  }

  private def fromNow(time: DateTime) = {

    val fromNowDuration = (time.getMillis - System.currentTimeMillis).milliseconds

    trace(s"FreePreviewScheduler.fromNow: now to $time is $fromNowDuration")

    fromNowDuration

  }

We're running with akka 2.2.1.

I'll work on a unit test to reproduce this and will post it here.

In trying to track down the issue, I noticed this line of code in the akka scheduler - https://github.com/akka/akka/blob/master/akka-actor/src/main/scala/akka/actor/Scheduler.scala#L307

 tickNanos here is akka.scheduler.tick-duration config value in nanosecond form.  The default - which we are using - is 50 milliseconds (50000 nanoseconds).   I determined that our delays -  which as I mentioned can go several months for some tasks - could indeed have a nanosecond value that when divided by 50000 would still cause a 32 bit int value to overflow its sign bit and go negative.   Changing the akka.scheduler.tick-duration to a larger value - we tried 500ms - caused the issue to go away.  Clearly for such a task as this, that value is fine, as is the work around we put in place (to cap the delay at a shorter maximum value and requeue the timer if we have not yet reached the timer for the particular event).    But I wanted to make you guys aware of this issue, if it is indeed an issue (or perhaps to point me to where I might be going wrong - not an infrequent occurrence! :-) ).

[Steve Buzzard]

Update: I forgot a few zeros for my millis/nano conversion numbers when typing my note below (and tick-duration now defaults to 10ms rather than 50 as it did once upon a time).  But it doesn't change the underlying issue.

[Roland Kuhn]

Hi Steve,

the scheduler used and provided by an ActorSystem is optimized for short but frequent timeouts, e.g. ReceiveTimeout of actors or network event timeouts. The timer service you need will have a different focus, most probably it will also have to be persistent in some form. This is to say that I see two issues:

– scheduling an event too far into the future should yield an exception instead of firing early

– scheduling tasks months ahead should not be done using system.scheduler

When sending a message to an actor from an external scheduling service you should think about how that service will obtain a reference to the ActorSystem or the ActorRef when the time comes; the system might have been restarted due to a hardware failure or similar.

Another possibility is to persist the schedule (e.g. in a database) and have an actor wake up periodically to poll it and transfer the long-term tasks into the system’s short-term scheduler, for example on a daily basis.

Regards,

Roland

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/groups/opt_out.

Dr. Roland Kuhn
Akka Tech Lead
Typesafe – Reactive apps on the JVM.
twitter: @rolandkuhn

[Steve Buzzard]

Thanks for the reply, Roland.  We do in fact persist this schedule to the database and it would be in any case be highly unlikely that the system would remain up for the length of time in question not to mention - as you state - failures, etc.  When the system is up and running these schedules are dynamic and event driven and in most cases are short in duration.  I also certainly understand that the timers are really meant for short duration events, which is why I'm surprised that there isn't a cap on the length of time for a scheduled event and an error if this is exceeded.  I'd be fine with that.  These scheduled items are reconstituted upon system start-up and are very small in number.  I have since changed the logic so that for events that are not scheduled to fire for a lengthy time to have a much shorter maximum time cap and to re-queue if they have not reached their scheduled start / end.  

You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/OBmHObGJYa0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to akka-user+...@googlegroups.com.