Stackdriver logs not showing up in GKE

[Craig Martin]

I am unable to see log messages that are sent from my GKE clusters using Golang. They work fine when running locally but not from the container running in GKE. Clearly something is misconfigured in GKE but I don't see any errors but not really sure where to look. Any insight or places to check would be very useful.

Below is my code and my cluster scopes (if it helps).

Thank you.

oauthScopes:

- https://www.googleapis.com/auth/cloud-platform
- https://www.googleapis.com/auth/compute
- https://www.googleapis.com/auth/datastore
- https://www.googleapis.com/auth/devstorage.full_control
- https://www.googleapis.com/auth/devstorage.read_only
- https://www.googleapis.com/auth/logging.write
- https://www.googleapis.com/auth/monitoring
- https://www.googleapis.com/auth/monitoring.write
- https://www.googleapis.com/auth/pubsub
- https://www.googleapis.com/auth/service.management.readonly
- https://www.googleapis.com/auth/servicecontrol
- https://www.googleapis.com/auth/source.full_control
- https://www.googleapis.com/auth/sqlservice.admin
- https://www.googleapis.com/auth/trace.append

Code: 

func LogMessage(logLevel ReddiyoLoggingSeverity, message, domain, transactionID string) {

    ctx := context.Background()
    // Creates a client.
    client, err := logging.NewClient(ctx, loggingData.ProjectID)
    if err != nil {
        log.Fatalf("Failed to create client: %v", err)
    }

    // Selects the log to write to.
    logger := client.Logger(loggingData.LogName)

    labels := make(map[string]string)
    labels["transactionID"] = transactionID
    labels["domain"] = domain

    var logSeverity logging.Severity
    switch logLevel {
    case debug:
        logSeverity = logging.Debug
    case info:
        logSeverity = logging.Info
    case warning:
        logSeverity = logging.Warning
    case reddiyoError:
        logSeverity = logging.Error
    case critical:
        logSeverity = logging.Critical
    case emergency:
        logSeverity = logging.Emergency
    default:
        logSeverity = logging.Warning
    }
    logger.Log(logging.Entry{
        Payload:  message,
        Severity: logSeverity,
        Labels:   labels})

    // Closes the client and flushes the buffer to the Stackdriver Logging

    // service.

    if err := client.Close(); err != nil {
        log.Fatalf("Failed to close client: %v", err)
    }
}

[Marc Unangst]

Hi Craig,

It's hard to know exactly what's happening in your situation, but here are a few suggestions:

- I assume this is probably intended to be a simple repro case, but best practice is to share a Logger across many log entries, rather than creating and destroying it for each log message. This allows the client to batch entries and flush them asynchronously. client.Close() is supposed to wait for any buffered entries to be flushed, but it's possible that quickly creating/destroying the client is preventing that somehow.

- Are your projectID and logName correct? Is the logName a legal value?

- When you say you don't see any errors, where are you looking? GKE should automatically collect stdout and stderr from each container and send them to Stackdriver Logging, so I would look at the stdout/stderr log IDs for your container to see if there are any errors being printed by the logging client library.

- Another idea would be to try temporarily replacing the logger.Log call with logger.LogSync, which will try to write the entry immediately, and return an error if there's a problem. Because of the buffering that the logger does by default, errors flushing log entries may not appear immediately, or may not be returned from Close. (Close can only return one error, so it will only return the last error from the logger.)

- It may be easier to emit logs to stdout/stderr and rely on GKE's built-in logging support, rather than using the Stackdriver Logging client library to send the logs. If you do this, you can emit JSON strings, which will be parsed to structured payloads (jsonPayload) instead of being sent as text. You can also set certain special fields in the JSON to populate LogEntry fields like severity and trace.

Unfortunately we can't really provide individualized support in this forum, so if you're not able to get it working with these ideas, I'd suggest opening a Cloud Support ticket and we'll be able to continue troubleshooting from there. Thanks!

-Marc Unangst

Eng. Manager, Stackdriver Logging

--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.
---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.
To post to this group, send email to google-stackdr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/69566c3e-3cce-40da-b700-d70f8e437eb0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.