TaskQueue Issues

[Prateek Malhotra]

My tasks are increasing in retry count but I have no logs showing they ever executed. My backends are shutting down pre-maturely (that process these tasks) and again, no errors or anything in the logs explaining why. Not sure if this is a backend or taskqueue issue. Here is the latency graph for task queues:

[Vinny P]

Hello Prateek,

Are you running on Master/Slave or HRD? I recall similar occurrences happening for me on M/S before I migrated.

If you're running on HRD, a few other questions:

1. Can you provide a screenshot of your logs? I realize that you said your tasks are showing no logs, but there could be a "diamond in the rough" that you're missing.

2. What logging level have you specified in log4j? 

3. Have you changed your application code recently? If so, can we see source code?

My first thought is that your backend is the problem here. The fact that the retry count is increasing means that AppEngine is (probably!) executing tasks just fine, but your backend is failing and responding with error codes. What is your backend doing? Does it depend on any non-GAE APIs?

-----------------

-Vinny P

Technology & Media Advisor

Chicago, IL

@GOV on AppDotNet: https://alpha.app.net/gov

[Prateek Malhotra]

(Reposting without signature)

I am using HRD. No changes to code, we've been doing the same thing for months. I remember this happening in the past, tasks just failing randomly with no logs. They eventually go through, but the instances usually restart before the tasks finish. CPU/Memory usage look fine the entire life of the process I run.

I use python so there's no log4j but I use the default settings, so all logging messages, including debug, make it into my log. The problem seems to have gone away, tasks are executing fine again and my backends aren't shutting down.

It may have been a small issue with the task queue service. My tasks are assigned to specific back end instances so the fact that they kept failing and the backend instances shutdown and come back shows that the task was having an error connecting to the backend which caused it to just shutdown due to inactivity, and finally when the task queue started connecting again, the instances came back up.

Thank you,

Prateek