Python 3: how to deserialize App Engine application logs from StackDriver Logging API?

[Ryan B]

Hi again! As part of migrating to Python 3, I need to migrate from logservice to the StackDriver Logging API. I have google-cloud-logging installed, and I can successfully fetch GAE application logs with eg:

>>> from google.cloud.logging_v2 import LoggingServiceV2Client
>>> entries = LoggingServiceV2Client().list_log_entries(('projects/brid-gy',),
         filter_='resource.type="gae_app" AND protoPayload.@type="type.googleapis.com/google.appengine.logging.v1.RequestLog"')
>>> print(next(iter(entries)))
proto_payload {
  type_url: "type.googleapis.com/google.appengine.logging.v1.RequestLog"
  value: "\n\ts~brid-gy\022\0018\032R5d..."
}
...

This gets me a LogEntry with text application logs in the proto_payload.value field. How do I deserialize that field? I've found lots of related mentions in the docs - links below - but nothing pointing me to a google.appengine.logging.v1.RequestLog protobuf generated class anywhere that I can use, if that's even the right idea. Has anyone done this?


https://googleapis.dev/python/logging/latest/gapic/v2/types.html#google.cloud.logging_v2.types.LogEntry.proto_payload

https://cloud.google.com/logging/docs/reference/v2/rpc/google.appengine.logging.v1#requestlog

https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#FIELDS.proto_payload

https://cloud.google.com/logging/docs/api/v2/resource-list#resource-indexes

https://groups.google.com/d/topic/google-cloud-dev/Xr6tZxOIaJY/discussion

[Ryan B]

Interesting. Dustin Ingram (Google DevRel) copied this question to StackOverflow, and answered it there, but not here. Is that standard for this group?

In any case, thanks Dustin! Except your answer to switch to the v1 StackDriver Logging API and use LogEntry.to_api_repr() doesn't work for me. I get:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/ryan/src/bridgy/local/lib/python2.7/site-packages/google/cloud/logging/entries.py", line 357, in to_api_repr
    info["protoPayload"] = MessageToDict(self.payload)
  File "/Users/ryan/src/bridgy/local/lib/python2.7/site-packages/google/protobuf/json_format.py", line 168, in MessageToDict
    return printer._MessageToJsonObject(message)
  File "/Users/ryan/src/bridgy/local/lib/python2.7/site-packages/google/protobuf/json_format.py", line 202, in _MessageToJsonObject
    return methodcaller(_WKTJSONMETHODS[full_name][0], message)(self)
  File "/Users/ryan/src/bridgy/local/lib/python2.7/site-packages/google/protobuf/json_format.py", line 314, in _AnyMessageToJsonObject
    sub_message = _CreateMessageFromTypeUrl(type_url, self.descriptor_pool)
  File "/Users/ryan/src/bridgy/local/lib/python2.7/site-packages/google/protobuf/json_format.py", line 389, in _CreateMessageFromTypeUrl
    'Can not find message descriptor by type_url: {0}.'.format(type_url))
TypeError: Can not find message descriptor by type_url: type.googleapis.com/google.appengine.logging.v1.RequestLog.

Searching turns up a few GitHub issues showing that people maintaining Google Cloud Python libraries have struggled with this for years. I got lost in them and haven't yet managed to find a conclusion:

https://github.com/googleapis/google-cloud-python/issues/2572

https://github.com/googleapis/google-cloud-python/issues/2674

etc.

In case it helps, here are the relevant packages I have installed in my virtualenv:

google-api-core                        1.14.3
google-api-python-client               1.7.11
google-auth                            1.6.3
google-auth-httplib2                   0.0.3
google-cloud-core                      1.0.3
google-cloud-logging                   1.14.0
googleapis-common-protos               1.6.0
oauth2client                           4.1.3

[George (Cloud Platform Support)]

Hello Ryan, 

You are right, posting to stackoverflow is the right thing to do for such a subject. You are now able to follow replies and suggestions in the new stackoverflow thread. This discussion group is oriented more towards general opinions, trends, and issues of general nature touching the app engine. For coding and program architecture, as well as de-serializing App Engine application logs from StackDriver Logging API, you may be better served in dedicated forums such as stackoverflow, where experienced programmers are within reach and ready to help.

[dustin...@google.com]

Not sure why but my original response in this thread did not make it to the list. Here's what I wrote:

You can use the LogEntry.to_api_repr() function to get a JSON version of the LogEntry.

You'll need to use the latest version of the client as well. Here's a snippet that might help:

>>> from google.cloud.logging import Client

>>> entries = Client().list_entries(
... ('projects/brid-gy',),
... filter_='resource.type="gae_app" AND protoPayload.@type="type.googleapis.com/google.appengine.logging.v1.RequestLog"')) 

>>> entry = next(iter(entries))
>>> entry.to_api_repr()

[Dustin Ingram]

You'll need to use the latest version of the client as well. Here's a snippet that might help:

>>> from google.cloud.logging import Client
>>> entries = Client().list_entries(
...     ('projects/brid-gy',),
...     filter_='resource.type="gae_app" AND protoPayload.@type="type.googleapis.com/google.appengine.logging.v1.RequestLog"'))        
>>> entry = next(iter(entries))
>>> entry.to_api_repr()

On Thu, Nov 7, 2019 at 10:01 AM Dustin Ingram <dustin...@google.com> wrote:

You can use the LogEntry.to_api_repr() function to get a JSON version of the LogEntry.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/a6665355-a7e6-42d4-96c2-82090d457adf%40googlegroups.com.

[Dustin Ingram]

You can use the LogEntry.to_api_repr() function to get a JSON version of the LogEntry.

On Thu, Nov 7, 2019 at 9:36 AM Ryan B <goo...@ryanb.org> wrote:

--

[Ryan B]

thanks guys! as mentioned, LogEntry.to_api_repr() is currently broken for this use case with the latest version of google-cloud-logging, 1.14.0. i've posted details on https://github.com/googleapis/google-cloud-python/issues/2572#issuecomment-551199704 (which is unfortunately closed right now).

To unsubscribe from this group and stop receiving emails from it, send an email to google-a...@googlegroups.com.