Expected fluentd JSON format

[Chris Foresman]

I've been trying to update our request logging middleware for falcon to collect data about each request and log it to Stackdriver using the ContainerEngineHandler in the GCP Python module. Right now I set the logging message to json.dumps(log_data), but then it just shows up as the message of the log line and not structured data. Docs and comments in the code constantly refer to "Format the message into JSON expected by fluentd", but it never explicitly states what that format is or where I can find documentation for it. I tried guessing from the Ruby code of the fluentd plugin, but I'm not well verse in Ruby and I couldn't make heads or tails of it.

What I would like to do is to write the JSON data as some kind of structured element in the log payload so that we can filter and possibly create metrics around the values in those logs. But I see no clear way to do that. Any ideas?

[Igor Peshansky]

Does https://cloud.google.com/logging/docs/agent/configuration#process-payload help?

        Igor

On Tue, May 29, 2018 at 5:43 PM Chris Foresman <fore...@gmail.com> wrote:

I've been trying to update our request logging middleware for falcon to collect data about each request and log it to Stackdriver using the ContainerEngineHandler in the GCP Python module. Right now I set the logging message to json.dumps(log_data), but then it just shows up as the message of the log line and not structured data. Docs and comments in the code constantly refer to "Format the message into JSON expected by fluentd", but it never explicitly states what that format is or where I can find documentation for it. I tried guessing from the Ruby code of the fluentd plugin, but I'm not well verse in Ruby and I couldn't make heads or tails of it.

What I would like to do is to write the JSON data as some kind of structured element in the log payload so that we can filter and possibly create metrics around the values in those logs. But I see no clear way to do that. Any ideas?

--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.
---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.
To post to this group, send email to google-stackdr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/ff63e814-b136-4d31-b49f-705fe03f6d20%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Chris Foresman]

Maybe? I'm not really sure, but I think this is implying that I can just build a JSON document with pretty much anything in it, and if anything matches up to the "special" fields they'll be stripped out and inserted in special places, and whatever is left over is left in jsonPayload. Is that right? That sounds like what I'm looking for. 

[Igor Peshansky]

That's exactly correct. Also, you can enable "detect_json" in the output plugin config (the "match" clause), which will parse serialized JSON written into text logs (e.g., stdout). This option is on by default in some environments (e.g., GKE). 

        Igor
-- sent from a mobile device, please excuse tyops and omissns

[Chris Foresman]

That's the environment we're running in, so perfect. Thank you. 

On Tue, May 29, 2018 at 10:00 PM Igor Peshansky <ig...@google.com> wrote:

That's exactly correct. Also, you can enable "detect_json" in the output plugin config (the "match" clause), which will parse serialized JSON written into text logs (e.g., stdout). This option is on by default in some environments (e.g., GKE). 

        Igor
-- sent from a mobile device, please excuse tyops and omissns

On Tue, May 29, 2018, 10:31 PM Chris Foresman <fore...@gmail.com> wrote:

Maybe? I'm not really sure, but I think this my inis implying that I can just build a JSON document with pretty much anything in it, and if anything matches up to the "special" fields they'll be stripped out and inserted in special places, and whatever is left over is left in jsonPayload. Is that right? That sounds like what I'm looking for. 

[Chris Foresman]

Hrmmm, either I'm doing something wrong, or the documentation is perhaps inaccurate. I've made my own ContainerEngineHandler class that adds httpResponse, operation, and sourceLocation fields to the structured JSON log output to standard error where it is picked up by fluentd. However, the resulting LogEntry records in Stackdriver don't appear to have stripped all the "special" fields from the JSON payload. Even when it did in the case of httpRequest, only some of the fields were transferred over. You can see that operation and sourceLocation are still inside the jsonPayload. Furthermore, presumably because it is not parsing the operation data, I'm not getting related log entries grouped together in the stackdriver UI.

Any ideas on what I'm missing?

Here's a sample:

{

    httpRequest: {

        remoteIp: "xxx.xxx.xxx.xxx"    

        requestMethod: "GET"    

        requestUrl: "http://redacted.example.com/v1/htx/poll-orders"    

        responseSize: "45"    

        status: 200    

        userAgent: "python-requests/2.18.4"

    }

    insertId: "198oijhg1cy6myp"   

    jsonPayload: {

        auth_user: "Sample User"    

        httpRequest: {

            protocol: "HTTP/1.1"     

            serverIp: "xxx.xxx.xxx.xxx"

        }

        message: "200 GET     /v1/htx/poll-orders"    

        operation: {

            first: true     

            id: "16fba0ef6f424d4cb5f418bf35c1d381"

        }

        request: null    

        response: "{"data": "Poller successful.", "error": null}"    

        sourceLocation: {

            file: "/usr/local/lib/python3.6/site-packages/middleware/logger.py"     

            function: "process_response"     

            line: 82

        }

        thread: 139630283294464

    }

    labels: {

        compute.googleapis.com/resource_name: "fluentd-gcp-v2.0.10-5n4fp"    

        container.googleapis.com/namespace_name: "pod"    

        container.googleapis.com/pod_name: "pod-6cdb5bfb8d-nskhl"    

        container.googleapis.com/stream: "stderr"

    }

    logName: "projects/project-id-000000/logs/pod"   

    receiveTimestamp: "2018-06-05T19:20:17.953129022Z"   

    resource: {

        labels: {

            cluster_name: "prodref"     

            container_name: "orp"     

            instance_id: "000000000000000"     

            namespace_id: "orp"     

            pod_id: "pod-6cdb5bfb8d-nskhl"     

            project_id: "project-id-000000"     

            zone: "us-central1-c"

        }

        type: "container"

    }

    severity: "INFO"   

    timestamp: "2018-06-05T19:20:13.571447849Z"

}

On Tue, May 29, 2018 at 10:09 PM, Chris Foresman <fore...@gmail.com> wrote:

That's the environment we're running in, so perfect. Thank you. 

On Tue, May 29, 2018 at 10:00 PM Igor Peshansky <ig...@google.com> wrote:

That's exactly correct. Also, you can enable "detect_json" in the output plugin config (the "match" clause), which will parse serialized JSON written into text logs (e.g., stdout). This option is on by default in some environments (e.g., GKE). 

        Igor
-- sent from a mobile device, please excuse tyops and omissns

On Tue, May 29, 2018, 10:31 PM Chris Foresman <fore...@gmail.com> wrote:

Maybe? I'm not really sure, but I think this my inis implying that I can just build a JSON document with pretty much anything in it, and if anything matches up to the "special" fields they'll be stripped out and inserted in special places, and whatever is left over is left in jsonPayload. Is that right? That sounds like what I'm looking for. 

On Tue, May 29, 2018 at 5:34 PM Igor Peshansky <ig...@google.com> wrote:

Does https://cloud.google.com/logging/docs/agent/configuration#process-payload help?

        Igor

On Tue, May 29, 2018 at 5:43 PM Chris Foresman <fore...@gmail.com> wrote:

I've been trying to update our request logging middleware for falcon to collect data about each request and log it to Stackdriver using the ContainerEngineHandler in the GCP Python module. Right now I set the logging message to json.dumps(log_data), but then it just shows up as the message of the log line and not structured data. Docs and comments in the code constantly refer to "Format the message into JSON expected by fluentd", but it never explicitly states what that format is or where I can find documentation for it. I tried guessing from the Ruby code of the fluentd plugin, but I'm not well verse in Ruby and I couldn't make heads or tails of it.

What I would like to do is to write the JSON data as some kind of structured element in the log payload so that we can filter and possibly create metrics around the values in those logs. But I see no clear way to do that. Any ideas?

--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 

Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdriver-discu...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.
---

You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-discussion+unsub...@googlegroups.com.

To post to this group, send email to google-stackdriver-discussion@googlegroups.com.

[Manuele Cavalli-Sforza]

I'm seeing the same thing. Any follow ups here? 

To post to this group, send email to google-stackdr...@googlegroups.com.