Google Stackdriver not showing struct entry log as expected from Google Kubernetes Engine

[chris...@calipsa.io]

I am using Stackdriver Logging for Python and a Python logger at the same time. I am using the function google.cloud.logging.logger.log_struct (https://gcloud-python.readthedocs.io/en/latest/logging/logger.html) to log a JSON to the Stackdriver.

I am able to view the logs as expected in the log viewer with the selected resource Global when I am running my script using a Google Compute Engine VM instance. The struct I am logging is recorded properly in jsonPayload.

However, when the logging is coming from a Google Kubernetes Engine, the logger view does not show the structs I passed, but rather what is printed on stdout. How do I make sure I observe the same behaviour from the Google Compute Engine VM instance and a Google Kubernetes Engine?

This is a snippet showing how I am doing the logging:

import google.cloud.logging
import logging

logging_client = google.cloud.logging.Client()
# connects the logger to the root logging handler
cloud_logger = logging_client.logger('email_adaptor')


formatter = logging.Formatter(
            '%(asctime)s - %(levelname)s - %(message)s - %(lineno)d - %(filename)s')
# get the logger a name
logger = logging.getLogger('email_adaptor')
# set a level for the logger
logger.setLevel(logging.DEBUG)

# make the logger write on stdout
stdout_alarm_log = logging.StreamHandler(sys.stdout)
stdout_alarm_log.setFormatter(formatter)
logger.addHandler(stdout_alarm_log)


struct = {'message':'Processed Alarm', 'valid': True}
cloud_logger.log_struct(struct, severity='INFO')

logger.info(str(struct))

This is an example of what I get on the STDOUT on both the VM instance and the Kubernetes Engine:

2018-10-26 12:30:20,008 - INFO - Processed Alarm {'valid': True} - 79 - log.py
INFO:email_adaptor:Processed Alarm {'valid': True}

This is what I see under the resource Global in the Google Log Viewer (the logs are ones from my VM instance, and do not correspond to the example I gave in the snippet code): 

This is what I see under the resource Google Kubernetes Engine: The struct do not show, instead I see what is printed on STDOUT: 

[Himanshu Bansal]

In Stackdriver Console panel  -> log Ingestion ... you can set exclusion rules to exclude logs which you don't want.

[Chris Tegho]

Hi Himanshu,

Yes that is fine, but what I am asking for is to get the logs, and more specifically the struct entries to show  when I am logging from a Kubernetes Engine. Currently, I do not get the logs to view as expected (if you look at the images I attached in the email).

On Mon, Oct 29, 2018 at 11:43 AM Himanshu Bansal <skillhima...@gmail.com> wrote:

In Stackdriver Console panel  -> log Ingestion ... you can set exclusion rules to exclude logs which you don't want.

--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.
---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.
To post to this group, send email to google-stackdr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/68cb2bd3-a3cb-4049-a60d-a1216ddbb5dd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Igor Peshansky]

Hi, Chris,

There is no default configuration on GKE or GCE that would parse the format your logger outputs to stdout.

Looks like you actually have two loggers in your example. One is using a library that writes directly into the Stackdriver logging API, and the other writes to stdout. I suspect the log entry you provide as the desired result comes from the former. Using the same exact library on GKE should work the same way.

Since the Stackdriver library calls in your code write against "global", it's very likely you'll find those structured log entries from your GKE containers if you filter for the "Global" resource.

As for stdout on GCE, unless you've redirected your stdout to a file and added custom configuration to your VM to find file, it will not be ingested at all. On GKE, however, stdout and stderr are ingested by default, and those are the logs you're seeing against "GKE Container". If you plan to write structured logs to stdout/stderr and access them in Stackdriver, the only structured format that the logging agent will accept is JSON. You must configure your logger to output JSON for it to be picked up as a structured log entry. More information at https://cloud.google.com/logging/docs/agent/configuration#process-payload .

        Igor
-- sent from a mobile device, please excuse tyops and omissns

To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/CAAjG-wvgFrX_xFThdDX-6%2BP8MTsfXJigCeUrGxKsziY61vk0_g%40mail.gmail.com.

[Chris Tegho]

Hi Igor,

Thanks so much for all the details. You are right, somehow I did not notice the logs coming from my GKE containers under the resource Global, and I assumed they were coming from a VM instance. They are showing the JSON objects I am logging under the jsonPayload as expected.

Thanks for pointing out the logging to stdout is ingested by default and shows under GKE container resource.

Best,

Chris