Extract only log lines from stackdriver using gcloud?

[Jonas Tingeborn]

Is there a way to get the gcloud command to output just the textPayload attribute values?

I haven't managed to figure out a good way of how to extract only the log lines that an app submitts to stackdriver, without all the metadata associated with the log entry, of which the actual log line is just one of many attributes.

Currently I've had to resort to some shell-fu in order to get the log output in a form akin to what logging locally to file would have yielded.

To get logs from a given docker container in container engine I presently do something like the following:

gcloud beta logging read '<query-expression>' --format list | grep ' textPayload: ' | cut -c17- > relevant_log_lines

PS. thanks for providing a CLI for stackdriver, very handy indeed.

[Nicholas (Google Cloud Support)]

Thanks for sharing your questions and situation with the community!

There currently is no way of getting only the textPayload attribute of each log entry from the command line by itself.  Your use of shell-fu is an effective workaround and a very common approach for this sort of command line task.  I might consider using the yaml format instead of list as it the output is somewhat cleaner making it perhaps less prone to grep regular expression errors.  Other output formats can be found in the gcloud reference.

If you would like to avoid using grep and see this feature implemented in the gcloud beta logging component (keeping in mind that it is currently in Beta), feel free to file a new feature request on the gcloud SDK public issue tracker.  Be sure to include as much information and business case as possible.

Hope this helps.

[jo...@tomologic.com]

Thanks for the clarification of the feature set currently implemented Nicholas. I'll open an ER as described.

/ Jonas

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/RDIJ5x-m-1U/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/0e779563-80a1-4d5f-b1ba-e355d89b6d9e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nicholas (Google Cloud Support)]

For what purpose are you using the gcloud beta logging component parsing logs?  Perhaps, understanding more about your use case, we may be able to suggest a more tailored solution for your goals.  For instance, what do you then do with relevant_log_lines?

On Wednesday, June 22, 2016 at 5:15:47 AM UTC-4, Jonas Tingeborn wrote:

[Jonas Tingeborn]

I'm just viewing it as an option for getting consolidated logs from our GCP and GKE resources, mainly as a workaround to GKE discarding the docker container output rather quickly.

Something similar to, and for the same purposes one uses docker-compose logs [-f], kubectl logs [-f] or kubectl get events [-w]
That is, a consolidated log event view of some system slice, of interest for ad-hoc discovery analysis.

A concrete example that prompted my exploration of beta-logging and stackdriver is that logs in GKE vanish "quickly". From the stackdriver documentation it seems data is kept there for 30 days, which is sufficient for our need regarding verifying changes we make to our own components deployed on your hosted platform, behaves as expected at all levels. Also for answering question about events that happened outside the time-span of your GKE log retention policy.

In short, to get hold of the consolidated logs that are a week or a couple old.

We use ELK elsewhere, but nothing beats having a CLI that easily facilitates ad-hoc exploration using "grep, perl, custom scripts etc", using plain log files (or unix streams), something that's also familiar to all developers in the company (no new web UI to learn).

Also, Google deploys its version of k8s configured with stackdriver as the default log target, so it seemed appropriate to try leveraging that log store and its tools rather than setting up some other infrastructure for the same purpose, stuff which would then need to be maintained by us (cost and complexity). Path of least resistance it seemed.

I'd be glad to hear of other options, and especially for which purpose the gcloud logging feature was created (the main use cases you envisioned).

[Nicholas (Google Cloud Support)]

If those with whom you work are most comfortable with command line tools, grep is definitely a very appropriate hands-on tool for the job.

If however, you'd like to aggregate log data or otherwise perform various operations on it, you might consider exporting the logs to BigQuery.  This process is described in Exporting Logs.  Within a dataset, one can specify a default table data expiration duration in days after which the table will be deleted.

Any new table created in this dataset will be automatically deleted the specified number of days after creation. This is useful for temporary data which does not need to be preserved.

This can cut down some on some storage costs depending how much you're using.  In addition, many of the regular expressions you might be using with grep to parse your log text payloads can be used in BigQuery to separate payloads into multiple columns (see Regular expression functions).

Hope this helps!

On Wednesday, June 22, 2016 at 5:15:47 AM UTC-4, Jonas Tingeborn wrote:

[Adrien Delessert]

I've been using the grep-for-textPayload approach for a while which works for most things, but if you ever wind up needing to incorporate multiple keys from each log message I stumbled on a decent approach via jq, which gives you access to all of the keys and a format string syntax.

For example to add back a timestamp to each log line:

gcloud beta logging read '<filter-query>' --format=json | jq -rnc --stream 'fromstream(1|truncate_stream(inputs)) | @text "\(.timestamp) \(.textPayload)"'

Or just extracting the textPayload in what feels like a more predictable manner than grep/cut if the formatting ever changes:

gcloud beta logging read '<filter-query>' --format=json | jq -rnc --stream 'fromstream(1|truncate_stream(inputs)) | .textPayload'

Very much agreed re 'nothing beats having a CLI that easily facilitates ad-hoc exploration using "grep, perl, custom scripts etc"'.

My apologies for digging up an old discussion, but I hope this is helpful to anyone who finds this thread in the future!

Wednesday, June 22, 2016 at 5:15:47 AM UTC-4, Jonas Tingeborn wrote: