Urgent: task queue creation errors all over the logs

[Bay]

Around 15 minutes ago app engine began throwing errors on task queue creation. This is the error:

1. 1.     taskqueue.Queue('presence-write').add(taskqueue.Task(url='/worker/writeuserpresence', params=dict(profile_id="%s" % profile.key().id())))
        File "/base/python27_runtime/python27_lib/versions/1/google/appengine/api/taskqueue/taskqueue.py", line 1637, in add
          self.__AddTasks(tasks, transactional, self.__FillAddPushTasksRequest)
        File "/base/python27_runtime/python27_lib/versions/1/google/appengine/api/taskqueue/taskqueue.py", line 1680, in __AddTasks
          raise _TranslateError(e.application_error, e.error_detail)
      TransientError
      

A transienterror (please try again later).

This has already resulted in hundreds of errors to my users.

Nothing can be seen from the system status page - unfortunately. It seems like it is not effectively monitoring all aspects of the app engine service?

Can somebody please fix this error?

[PK]

We see the same thing. This needs urgent attention and postmortem.

PK

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/NHIwThOV8rcJ.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Martin Ceperley]

Same here.

[Bay]

Further to this the report production issue has disappeared from admin console. Reporting service crucial errors such as this in a community google group is hardly the optimal solution. Please allow a much easier solution.

[Bay]

Seems to have now been resolved. Questions remain:

1) How did this happen? It must have affected thousands of users in total, knowing that the task queue function is highly utilized

2) Why did the system status not reveal the error?

[PK]

You can still add a production issue, I attach the URL.

http://code.google.com/p/googleappengine/issues/entry?template=Production%20issue

I just added production issue 8571 for this, star it and add your appid...

PK
http://www.gae123.com

On Dec 18, 2012, at 10:26 AM, Bay <troe...@gmail.com> wrote:

> Further to this the report production issue has disappeared from admin console. Reporting service crucial errors such as this in a community google group is hardly the optimal solution. Please allow a much easier solution.
>

> --
> You received this message because you are subscribed to the Google Groups "Google App Engine" group.

> To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/XoseW4u4MrMJ.

[Bay]

Well maybe its just me but in my experience the referred issue-tracker is not followed on a minute-to-minute basis and therefore unsuitable for anything other than feature requests...

[Johan Euphrosine]

The issue has been Acknowledged on the public issue tracker:

https://code.google.com/p/googleappengine/issues/detail?id=8563

And the incident has been advertised on the status site:

https://code.google.com/status/appengine/detail/taskqueue/2012/12/18#ae-trust-detail-taskqueue-add-latency

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/AaWDWkP7I6MJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
Johan Euphrosine (proppy)
Developer Programs Engineer
Google Developer Relations

[PK]

Hi Bay,

my experience is similar...

Could somebody from Google comment if this URL is still the best place for production issues for those of us who do not have premier support yet?

Thanks,

PK

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/AaWDWkP7I6MJ.

[Johan Euphrosine]

The following message has been posted to the status site:

"""

The problem with the Google App Engine Task Queue API should be resolved as of 10:40 AM US/Pacific time. Applications should now be able to add tasks normally, although some applications may continue to see elevated latency for a short while. We apologize for the inconvenience and thank you for your patience.      

"""

[Bay]

And what was the problem?

The tracker issue you are referring to is 26 hours (!) old. Does this mean it took you guys 26 hours to realise this issue? Is this normal, and how are we supposed to interpret this lack of information?

[Christina Ilvento]

Hi All,

We had an issue this morning from about 9:30 to 10:40 this morning that caused transient errors when adding tasks to task queue (noted on our status site: http://code.google.com/status/appengine/detail/taskqueue/2012/12/18#ae-trust-detail-taskqueue-add-latency). This issue was picked up by our regular monitoring and was likely unrelated to the production issue from yesterday linked above. I'm working with our team to figure out if the issue reported yesterday is related or part of a separate incident (or if it was, in fact, a transient issue).

Please let us know if you're still having problems with task queue by reporting an issue in our issue tracker.

Thanks,

Christina

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/X46QOoHqcssJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--

Christina Ilvento | Google App Engine | cilv...@google.com | (650)-201-9399