Alert triggered for 500 responses, but no 500 responses in logs

[Alan deLespinasse]

Hi, I'm trying to debug an alert I have set up. It's a pretty low volume service, App Engine standard environment, Python 3.x.

It's supposed to raise an alert any time there's a 500 response or higher on a query. I've occasionally been getting alerts, but then have been unable to find any 5xx errors in the logs.

The alert configuration looks like this:

---

I don't remember why I set the function to "rate" instead of "count", but as far as I can tell it shouldn't make a difference. The threshold is zero, so it should trigger if (and, I would hope, only if) there are any 500x errors

I didn't have the "loading = false" filter when I first set it up (sometime last year), but I got a few random alerts when nothing seemed wrong, and the "loading = false" filter seemed to fix it at the time. I was never clear on why (I assume "loading" means that the request was handled by a new instance that was not yet done booting up when the request came in, or something like that).

It's only in the last couple months, I think, that I've started getting these spurious alerts again.

Am I doing something wrong?

[Alan deLespinasse]

Hmm, ok, with help from this post I was able to find the 500 error. It was on /_ah/warmup and didn't show as an error or warning in the logs. I'm not sure why it would have returned an error (all that endpoint does is return a 200, so the instance must have just failed to boot somehow, but there's nothing useful in the logs).

Now I'm trying to figure out how I could filter out errors on warmup calls, since I don't think there's anything I could possibly do to prevent these errors. I would have thought that would count as a "loading" response if anything did, but I guess not.

[Osvaldo Lopez Acuña]

This video can help you to avoid common alerting mistakes. You can also try this Troubleshooting log-based alerts. If alert isn’t working try this troubleshooting.

There’s a known issue where sometimes app log entries are not correlated with the request log. This happens the first time your app receives a request and any other time App Engine writes status messages to your app's log. For more information, see https://issuetracker.google.com/issues/138365527.