Error when using google.cloud.logging with logging.v1.RequestLog?

[Julian Bunn]

Right now we cannot use the request_logs feature for downloading our GAE logs (see my other thread on that problem!), so I thought I would try the newer google.cloud features, and get modern :-)

Here is the simple test I tried:

from google.cloud import logging

from google.cloud.logging import DESCENDING

def main():

    client = logging.Client() 

    for entry in client.list_entries(order_by=DESCENDING):

        print entry

if __name__ == '__main__':

    main()

When executing this (after setting up the gcloud credentials), there seems to be a problem parsing the log entries:

python GetLogsCloud.py

Traceback (most recent call last):

  File "GetLogsCloud.py", line 25, in <module>

    main()

  File "GetLogsCloud.py", line 18, in main

    for entry in client.list_entries(order_by=DESCENDING):

  File "c:\python27\lib\site-packages\google\cloud\iterator.py", line 219, in _items_iter

    for item in page:

  File "c:\python27\lib\site-packages\google\cloud\iterator.py", line 163, in next

    result = self._item_to_value(self._parent, item)

  File "c:\python27\lib\site-packages\google\cloud\logging\_gax.py", line 488, in _item_to_entry

    resource = MessageToDict(entry_pb)

  File "c:\python27\lib\site-packages\google\protobuf\json_format.py", line 133, in MessageToDict

    return printer._MessageToJsonObject(message)

  File "c:\python27\lib\site-packages\google\protobuf\json_format.py", line 164, in _MessageToJsonObject

    return self._RegularMessageToJsonObject(message, js)

  File "c:\python27\lib\site-packages\google\protobuf\json_format.py", line 196, in _RegularMessageToJsonObject

    js[name] = self._FieldToJsonObject(field, value)

  File "c:\python27\lib\site-packages\google\protobuf\json_format.py", line 230, in _FieldToJsonObject

    return self._MessageToJsonObject(value)

  File "c:\python27\lib\site-packages\google\protobuf\json_format.py", line 162, in _MessageToJsonObject

    return methodcaller(_WKTJSONMETHODS[full_name][0], message)(self)

  File "c:\python27\lib\site-packages\google\protobuf\json_format.py", line 266, in _AnyMessageToJsonObject

    sub_message = _CreateMessageFromTypeUrl(type_url)

  File "c:\python27\lib\site-packages\google\protobuf\json_format.py", line 341, in _CreateMessageFromTypeUrl

    'Can not find message descriptor by type_url: {0}.'.format(type_url))

TypeError: Can not find message descriptor by type_url: type.googleapis.com/google.appengine.logging.v1.RequestLog.

I'm wondering what the issue is here ... is it because of the type of logging we use on GAE?

Thanks!

[Nick (Cloud Platform Support)]

Hey Julian,

I've been able just now to run the same code fine. What might you mean by "the type of logging we use on GAE"? Are you doing anything particularly odd there?

It's worth mentioning here that this forum isn't meant for technical support but rather for general discussion of the platform, services, design patterns, etc. This question should have been posted to stackoverflow.com on a relevant Cloud Platform / App Engine tag. We monitor Stack Overflow regularly, so we'll be able to answer there, although we can perhaps help debug this a little here before you move to post there.

Cheers,

Nick
Cloud Platform Community Support

[Julian Bunn]

Hi Nick,

Thanks ... I do try StackOverflow from time to time for these sorts of technical issues, but have very limited success. Perhaps I'm using bad tags. See for example a logging related question I posted there on Friday about JSON credentials for logging https://stackoverflow.com/questions/43004904/accessing-gae-log-files-using-google-cloud-logging-python

Regarding the v1.RequestLog error that's the topic of this thread: we do not use StackDriver logging in our GAE deployment, so I wonder whether cloud.logging only supports StackDriver logs?

Otherwise, I am at a loss as to why google.cloud.logging can't understand the log format it fetches from our GAE deployment?

We've now been without access to our log data for almost a week, so I'm getting quite desperate to find a solution either with request_logs or cloud.logging :-)

Thanks for your help.

Julian

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/M01jjjNc1dQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/cd373f85-74d3-4dab-97d7-2693bf1dc6e2%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Nick (Cloud Platform Support)]

Hey Julian,

I've observed the identical error when using the filter 'resource.type="gae_app" AND resource.labels.module_id="default"', where I'd deployed a standard environment app on "default".

It appears this is caused by the fact that the standard environment doesn't log to Stackdriver v2 logging, as you observed. The docs show all v2 endpoints are what the library interfaces with. Nonetheless, it should be able to read V1 type logs as well, or at least we can consider this a feature request.

In the meantime, the following command pattern can be used, presenting no problem with V1 logs:

gcloud beta logging read ${LOG_FILTER}

I've created a Public Issue Tracker issue which you can follow while we work on this.

Cheers,

Nick
Cloud Platform Community Support

[Julian Bunn]

Thanks, Nick!

I confirm that the command "gcloud beta logging read" retrieves our logs! So this looks very promising ...

Is there a way to call this interface similarly to:

from google.cloud import logging

from google.cloud.logging import DESCENDING

def main():

    client = logging.Client() 

    for entry in client.list_entries(order_by=DESCENDING):

        print entry

if __name__ == '__main__':

    main()

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/9ae8bdb3-fe84-43d0-b58e-4b2894ec8158%40googlegroups.com.

[Nick (Cloud Platform Support)]

Hey Julian,

You have two options here: either to wait for the google.cloud library to get support for V1 logs, or adopt a slightly different approach to writing the python script. You could use the python subprocess module to call the gcloud command (keeping in mind that this is part of the beta command group, which can change on updates without warning (the only warning being the release notes)), sending the output into a string, and then parsing the entries that way. This, while it would mean you'd have your logs in a python environment (with all the leverage that brings) however would not use the google.cloud library.

Cheers,

Nick
Cloud Platform Community Support

[Julian Bunn]

Hi Nick,

Thanks - I had just started doing exactly what you suggest! :-)

I crave your indulgence on one other issue ... when I run a gcloud beta logging command with a LOG_FILTER clause on a Unix system, it works fine. For example:

 gcloud beta logging read 'timestamp<="2017-03-23T00:00:00Z" AND timestamp>="2017-03-22T00:00:00Z"' --limit=2

However, on Windows, the exact same command doesn't work - I suspect the single and double quotes are part of the problem:

 gcloud beta logging read 'timestamp<="2017-03-23T00:00:00Z" AND timestamp>="2017-03-22T00:00:00Z"' --limit=2

The filename, directory name, or volume label syntax is incorrect.

(A simple "gcloud beta logging read" works just fine on Windows.)

I've tried various ways of escaping the quotes, but nothing appears to work.

Should I post this on StackOverflow?

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/775cc91d-5d04-4c2b-bec6-139157889b71%40googlegroups.com.

[Nick (Cloud Platform Support)]

Hey Julian,

It seems more likely to be an issue with the shell in Windows rather than the SDK, but of course if you can determine that it's not the shell, then you'd want to post this as an issue to the Public Issue Tracker with all information necessary for us to attempt to observe this on our own systems. Stack Overflow is a good choice when you aren't sure if it's a platform issue or just your own code / configuration.

As for the quoting issue, I'm no windows expert but I thought I did remember something about quote escaping being weird in CMD. Lo and behold, there's a Stack Overflow Q&A about this, maybe your solution is in there? It seems that the way to escape a quote is to just put two quotes, but if it was that simple, the answer wouldn't be so long. Happy reading!

Cheers,

Nick
Cloud Platform Community Support

On Wednesday, March 29, 2017 at 4:07:34 PM UTC-4, Julian Bunn wrote:

Hi Nick,