Delays in cron tasks

[Taengoo Taengstagram]

One of my apps have tasks configured to run every minute in cron.yaml. 

I've noticed for quite some time that the tasks are not being scheduled/executed with the frequency specified. There seem to be delays > 1 minute as seen in the task queue (the task queue is not congested and often have nothing running when I check). When I examine the logs, the minute tasks are being executed on average once every 2 minutes instead.

Any clues why this is happening? I have other apps with once per minute tasks scheduled and those do not have the same problems (the delay if any is often just a few seconds).

[Jordan (Cloud Platform Support)]

Hi Taengoo,

The Task Queue executes tasks the same way as a normal client would when attempting to access your App Engine service; by sending an HTTP GET request to your service's URL endpoint. Task Queues are therefore affected by the same conditions as any normal request to your app would, such as latency due to traffic overload.

To confirm if this is the cause of the delays, you can take a look at the App Engine Dashboard ‘latency’ and ‘loading latency’ graphs (located under the ‘Summary’ dropdown) for your project and service that is receiving the requests from your cron job. If you see visible spikes in latency during the time of the cron delays (by selecting the appropriate time span in the console), it is recommended to perform the following:

• Ensure your app.yaml or appengine-web.xml scaling settings are configured to handle your expected traffic by:
• Setting a high enough value of min idle instances to preemptively warm up instances of your service to quickly handle new requests (aka tasks).
• Increasing your instance class to a more powerful machine type to handle tasks more quickly.
• Ensuring that threadsafe is enabled and that you are using the default value of 8 for concurrent requests, which allows for your instances to handle more than one task at a time.
• If your service is also accepting requests from other clients that is not your Cron job, and you require to have your task executed on time, you can also consider deploying a separate service to be used strictly to handle tasks for this Cron job only.

You can also implement an alert with Stackdriver Monitoring using the task_attempt_delays metric type to notify you if there is any delay in your task executions. Since you have confirmed that your other app is handling your Cron tasks without any latency in ETA scheduling, this shows how app variability in latency and traffic are a very likely cause. Note that the scheduled ETA specifies the absolute earliest time, and not the actual time of execution due to the above mentioned expectations.

If after performing the above you are still noticing delays, I ask that you share your 'app.yaml' or 'appengine-web.xml' scaling settings for the service and version running your tasks, and your 'cron.yaml' job settings for the job seeing the delays.