Grouping GKE stdout container logs?

570 views
Skip to first unread message

TKO

unread,
Jul 17, 2019, 2:01:57 PM7/17/19
to Google Stackdriver Discussion Forum
I originally started my GCP experience on AppEngine, where the out of the box stackdriver experience is fantastic. I'm trying to group logs from my managed GKE containers as specified in the docs. I've got every field set except one -- it seems that the fluentd defaults do not modify the logName field sent to stdout. My logs always end up with the logName set to projects/[PROJECT-ID]/logs/app According to the docs, child logs must have a different logName than the parent log entry. I know grouping can be accomplished via the client libs, but the client libs require communicating with the google oauth apis which would be painful to setup in my current environment.

Is there any way to group logs in the stackdriver UI for folks using stdout logging on GKE? Is there something I didn't see in the docs?

Igor Peshansky

unread,
Jul 19, 2019, 1:15:58 PM7/19/19
to TKO, Google Stackdriver Discussion Forum
Hi,

It would be great to get more info on what you're trying to do and which docs you've been following. With the Stackdriver Kubernetes monitoring experience, the resource labels can be used to group the logs by pod and container name. You can also group by pod annotations, by using the labels prefixed with "k8s-pod-label/".
        Igor

On Wed, Jul 17, 2019 at 2:01 PM TKO <nelso...@gmail.com> wrote:
I originally started my GCP experience on AppEngine, where the out of the box stackdriver experience is fantastic. I'm trying to group logs from my managed GKE containers as specified in the docs. I've got every field set except one -- it seems that the fluentd defaults do not modify the logName field sent to stdout. My logs always end up with the logName set to projects/[PROJECT-ID]/logs/app According to the docs, child logs must have a different logName than the parent log entry. I know grouping can be accomplished via the client libs, but the client libs require communicating with the google oauth apis which would be painful to setup in my current environment.

Is there any way to group logs in the stackdriver UI for folks using stdout logging on GKE? Is there something I didn't see in the docs?

--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.
---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/6c44216a-c1b2-4de2-a65f-6b1abd11d312%40googlegroups.com.

TKO

unread,
Jul 22, 2019, 11:10:19 AM7/22/19
to Google Stackdriver Discussion Forum
Hey Igor, 

To add some clarity:
What I'd like to accomplish is this:

Whenever a request comes in to my application via HTTP, a log is created -- then each log created within the context of the HTTP request is viewable in the StackDriver logging web UI as a child of the initial request log. On AppEngine this is the default behavior. What you end up with in the StackDriver UI is an expandable tree-view of logs that are grouped by request. 

I'd prefer to stick with Legacy Stackdriver and writing to STDOUT if possible.

Thanks for the reply, and let me know if theres any more context I can add.

Igor Peshansky

unread,
Jul 22, 2019, 11:29:06 AM7/22/19
to TKO, Google Stackdriver Discussion Forum
Hi,

Just FYI, the legacy experience will no longer be available on later GKE versions (though it's still available on versions up to and including 1.14).

Both the legacy and the new Stackdriver experience support detecting and parsing single-line JSON written to the container stdout. As you noted, the httpRequest field will be parsed out of the JSON and sent as part of the log entry. So you have all the tools to get the same httpRequest data in your GKE logs as you would in your GAE logs.

However, the Logs Viewer does not group entries by httpRequest, but rather by the embedded trace id, which GAE automatically generates for each request. You can embed a trace id in your container logs as well, by setting the logging.googleapis.com/trace field in the generated JSON, and you should get the same behavior as for GAE logs.

Hope this helps,
        Igor

--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.
---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/89aa62f4-23fd-45b5-872c-f54aa6f2fd80%40googlegroups.com.

TKO

unread,
Sep 20, 2019, 6:12:16 PM9/20/19
to Google Stackdriver Discussion Forum
Igor thanks for your help so far,

Even setting the trace hasn't correlated the logs for me. Since my last message I've moved my project over to Stackdriver Monitoring. Multiple logs have the same trace, but since the logs are coming from the same container they have the same logName. My gut feeling is that the logName needs to be different to group the logs in the UI as specified in the AppEngine docs I listed in my last post. Any tips / ideas to troubleshoot?

On Monday, July 22, 2019 at 9:29:06 AM UTC-6, Igor Peshansky wrote:
Hi,

Just FYI, the legacy experience will no longer be available on later GKE versions (though it's still available on versions up to and including 1.14).

Both the legacy and the new Stackdriver experience support detecting and parsing single-line JSON written to the container stdout. As you noted, the httpRequest field will be parsed out of the JSON and sent as part of the log entry. So you have all the tools to get the same httpRequest data in your GKE logs as you would in your GAE logs.

However, the Logs Viewer does not group entries by httpRequest, but rather by the embedded trace id, which GAE automatically generates for each request. You can embed a trace id in your container logs as well, by setting the logging.googleapis.com/trace field in the generated JSON, and you should get the same behavior as for GAE logs.

Hope this helps,
        Igor

On Mon, Jul 22, 2019 at 11:10 AM TKO <nelso...@gmail.com> wrote:
Hey Igor, 

To add some clarity:
What I'd like to accomplish is this:

Whenever a request comes in to my application via HTTP, a log is created -- then each log created within the context of the HTTP request is viewable in the StackDriver logging web UI as a child of the initial request log. On AppEngine this is the default behavior. What you end up with in the StackDriver UI is an expandable tree-view of logs that are grouped by request. 

I'd prefer to stick with Legacy Stackdriver and writing to STDOUT if possible.

Thanks for the reply, and let me know if theres any more context I can add.

--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdriver-discu...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-discussion+unsub...@googlegroups.com.

Xi Pu

unread,
May 16, 2020, 9:25:04 AM5/16/20
to Google Stackdriver Discussion Forum
Hi, I have met the same problem as yours. I run my application in GKE container and use glog for logging. My logs are written to stderr by default. I would like to group logs in one request like parent-child format. It seems the only way to do that is by setting the different logName. However all logNames are the same because all logs are written to stderr. Have you found a way to correlate the logs? Thanks a lot and look forward to hearing back!

Xi

Igor Peshansky

unread,
May 16, 2020, 10:17:12 AM5/16/20
to Xi Pu, Google Stackdriver Discussion Forum
Hi,

There are a number of articles on the topic (see [1], [2], [3]). The log name shouldn't matter, as long as it's the same and other fields match up.

Specifically, you have to set both traceId and httpRequest.utl. You can specify both by emitting JSON payloads as documented in [4].

Hope this helps.

[4] https://cloud.google.com/logging/docs/agent/configuration#process-payload
        Igor
-- sent from a mobile device, please excuse tyops and omissns

Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.

--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/c80d56e3-73e4-4396-ba93-81caf4bd27b7%40googlegroups.com.

Xi Pu

unread,
May 16, 2020, 3:00:37 PM5/16/20
to Google Stackdriver Discussion Forum
Hi Igor,
Thanks for your quick response! I have read these articles before. But as [2] mentioned, "the child must have a different log name than the parent". In my experiment, I have set the request url in parent and set the same traceID in parent and child. But child log won't show in the nest of parent log. After I modify the log name of child log, the child log appears in the nested. Is there anyway to solve this issue besides setting the different logName? Thanks a lot!

Xi


On Saturday, May 16, 2020 at 7:17:12 AM UTC-7, Igor Peshansky wrote:
Hi,

There are a number of articles on the topic (see [1], [2], [3]). The log name shouldn't matter, as long as it's the same and other fields match up.

Specifically, you have to set both traceId and httpRequest.utl. You can specify both by emitting JSON payloads as documented in [4].

Hope this helps.

[4] https://cloud.google.com/logging/docs/agent/configuration#process-payload
        Igor
-- sent from a mobile device, please excuse tyops and omissns

Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdriver-discu...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-discussion+unsub...@googlegroups.com.

--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdriver-discu...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-discussion+unsub...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages