Stable Go app gone wild 3-4 days ago (Error code 123)

[radomir]

Hi,

Something happened to my paid Go application about 3-4 days ago. I didn't change code recently and it was working nicely/stable earlier. Now, instance lifetime is showrt and latencies sky-rocketed even for the simplest handler returning fixed text. All I see in the logs for such requests is message:

  Process terminated because the request deadline was exceeded. (Error code 123)

and there are no my log messages so I think execution didn't event start.

Check the screenshots from the Dashboard bellow showing latencies and instance activity. For some reason AppEngine shows an instance as "active" while handlers are not really CPU busy - the service mostly talks to remote web services. (The remote service is not an issue as tasks eventually complete successfully (and quickly) on retry.) When working normally, latency was about one second but now requests times out (tasks after 10 minutes)!

Any ideas? I'd appreciate if someone from Google could check what's going (my app id is bugsio-atlas-addons).

Thanks,

Radomir