Taskqueue ETA *sometimes* 1 hour off in AppEngine

[Robert _]

I'm experiencing a very tricky bug or "feature" in Taskqueues regarding tasks set with an ETA.  Tasks are being executed 1 hour later than expected *in some instances* (inconsistently).  I think this behavior started after DST.

In my experience with a handful of scheduled tasks (e.g. development), it works fine.  I'm using UTC timestamps in my Tasks set with the "etaMillis()" method in the Taskqueue API.  Things execute just when expected.  

Each night, however, I have thousands of tasks that queue up.  This causes multiple AppEngine instances to fire up (and I think this is important--the root cause of the inconsistent behavior).  Only during load am I seeing this issue.  *Some* tasks--are executing exactly 1 hour behind when they are expected to.  My AppEngine region is us-central.

Task queues are a wonderful feature of AppEngine.  Anyone else experience this with high volumes of scheduled tasks?  What's going on?  Different app engine instances with different time zone settings?  

I'm thinking a potential alternative will be to specify countdown instead... maybe this will resolve the issue?

[Robert _]

As a quick follow-on I should note that this is not an issue with my local machine or development environment.  Tasks are being enqueued *from the exact same Compute Engine machine* in all scenarios.  I have done the work to isolate this issue.  The timing error is either introduced by the mechanism that pushes the task into the queue or by the queue mechanism itself.

[Robert _]

For those of you who might run into this in the future.. I've gleaned that task ETA is the "soonest" a task will execute.  Under load those times start to slip.  Even after jacking queue processing rates way up I'm finding huge delays build up over time.  I'm going to need a new scheduling strategy.  For high throughput workloads, be warned: executing tasks with an ETA is not reliable.  Perhaps there is a way to spin up additional appengine instances... but for my purposes I'm going to write some code to determine if a task is "late" when I execute it and re-schedule it appropriately so my tasks remain spaced at a proper distance.  

[George (Cloud Platform Support)]

Hello Robert, 

The actual description of the option is: "etaMillis(long etaMillis)  Sets the approximate absolute time to execute." on the "com.google.appengine.api.taskqueue" documentation page, "Method Summary" sub-chapter. Your option sets an approximate value, that may increase under heavy load, as you already noticed. 

You are right, taking control of the execution time for your tasks with a dedicated strategy and appropriate re-scheduling should avoid these huge delays.