Python Standard logging can deadlock at import time (with work-around)

[Evan Jones]

If you see rare "stuck" requests that either hit the overall request deadline (on frontend instances), or seem to hang "forever" on backend instances, there is a very small chance you are running into this bug we found in App Engine Standard's logging library. The brief summary is if you call logging at import time while other threads are also logging, it is possible that you can deadlock your requests. We were running into this on rare occasions, that recently started happening more often (a few times a day).

We have a workaround that we've been using for the past few days, and it seems to solve the problem. If you run into this, I'd love to hear about it.

Issue: https://issuetracker.google.com/issues/65016348

Description and workaround: https://github.com/evanj/app-engine-log-deadlock

[Kenworth (Google Cloud Platform)]

Since you already filed a public issue tracker for your issue, I recommend all replies and posts be directed towards that forum until a resolution is reached. 

[Attila-Mihaly Balazs]

Thank you Evan for publicly posting this (and an especially big thank you for publishing the detailed investigation *and* work-around).

Contrary to the official Google party-line, posting to the issue-tracker is near-useless for us mere-non-googlers:

- the new issue tracker is all AJAX-y and not crawled by Google
- its search is sub-par
- people are too quick in marking things "private" on it, so frequently one would see a link (even from this group) and then not be able to read the contents

Comparatively, this group (with its admittedly crummy interface) is crawled by Google and people like myself can use it to stay up-to-date with potential issues they might see in AppEngine production.

Thank you again!
Attila

[Evan Jones]

That is exactly why I posted here: I'm hoping on the small chance someone runs into this issue, they will find my post and it might help them resolve the problem in less time than it took us! Cheers!

Evan