Colon (:) replaced by arrow (=>) in output

[Hector Oswaldo Caballero]

Hi again,

Another noob question. I'm deploying fluent-bit and fluentd to a K8s cluster to forward logs to an external server.

I'm interested in getting the journald and /var/log/messages logs for the moment so I set up Fluent-Bit config like that:

[SERVICE]
        Flush         1
        Daemon        off
        Parsers_File  parsers.conf

    [INPUT]
        Name            tail
        Tag               logger.syslog
        Path              /var/log/messages
        Parser            docker
        DB                /var/log/fluentbit.db
        Buffer_Chunk_Size 64k
        Buffer_Max_Size   128k
        Skip_Long_Lines   On
        Mem_Buf_Limit     5MB
        Refresh_Interval  5

    [INPUT]
        Name          systemd
        Tag             logger.systemd
        Path            /var/log/journal

    [OUTPUT]
        Name            forward
        Match           *
        Host            {{.Values.service.name}}-{{.Values.fluentd.app}}-svc.{{.Release.Namespace}}
        Port            {{.Values.fluentd.port}}


parsers.conf:
    [PARSER]
        Name        syslog
        Format      regex
        Regex       ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
        Time_Key    time
        Time_Format %b %d %H:%M:%S

    [PARSER]
        Name         docker
        Format       json
        Time_Key     time
        Time_Format  %Y-%m-%dT%H:%M:%S.%L
        Time_Keep    On
        # Command       |  Decoder  | Field | Optional Action   |
        # ==============|===========|=======|===================|
        Decode_Field_As    escaped     log

Before I was using the syslog parser and had the original syslog messages transformed to:

{"log":"{ \"msg\": \"microcode: CPU0 sig=0x206a1, pf=0x1, revision=0x1\", \"rawmsg\": \"microcode: CPU0 sig=0x206a1, pf=0x1, revision=0x1\", \"timereported\": \"2018-10-30T23:25:46.883690+00:00\\\\\", \"hostname\": \"node-1\", \"syslogtag\": \"kernel:\", \"inputname\": \"imjournal\", \"fromhost\": \"node-1\", \"fromhost-ip\": \"127.0.0.1\", \"pri\": \"6\", \"syslogfacility\": \"0\", \"syslogseverity\": \"6\", \"timegenerated\": \"2018-10-30T23:25:46.883690+00:00\", \"programname\": \"kernel\", \"protocol-version\": \"0\", \"structured-data\": \"-\", \"app-name\": \"kernel\", \"procid\": \"-\", \"msgid\": \"-\", \"uuid\": null, \"$!\": { \"PRIORITY\": \"6\", \"_BOOT_ID\": \"9c260ada251e43e5b3c99493f204a496\", \"_MACHINE_ID\": \"470e4674b4574ae7b2279a5ee4481600\", \"_HOSTNAME\": \"localhost\", \"_TRANSPORT\": \"kernel\", \"SYSLOG_FACILITY\": \"0\", \"SYSLOG_IDENTIFIER\": \"kernel\", \"_SOURCE_MONOTONIC_TIMESTAMP\": \"2078442\", \"MESSAGE\": \"microcode: CPU0 sig=0x206a1, pf=0x1, revision=0x1\" } }"}

By using the docker parser I'm able to get the original message unescaped. The issue, though, is that the colon is replaced by an arrow:

{"msg"=>"pci 0000:00:03.0: reg 0x10: [io  0xc060-0xc07f]", "rawmsg"=>"pci 0000:00:03.0: reg 0x10: [io  0xc060-0xc07f]", "timereported"=>"2018-10-31T20:44:05.637391+00:00", "hostname"=>"node-1", "syslogtag"=>"kernel:", "inputname"=>"imjournal", "fromhost"=>"node-1", "fromhost-ip"=>"127.0.0.1", "pri"=>"7", "syslogfacility"=>"0", "syslogseverity"=>"7", "timegenerated"=>"2018-10-31T20:44:05.637391+00:00", "programname"=>"kernel", "protocol-version"=>"0", "structured-data"=>"-", "app-name"=>"kernel", "procid"=>"-", "msgid"=>"-", "uuid"=>nil, "$!"=>{"_BOOT_ID"=>"fc2b68275efa48fea941beffb564b8c3", "_MACHINE_ID"=>"b2118a24bf3545ccac1f1e104cddbc8d", "_HOSTNAME"=>"localhost", "_TRANSPORT"=>"kernel", "SYSLOG_FACILITY"=>"0", "SYSLOG_IDENTIFIER"=>"kernel", "PRIORITY"=>"7", "_KERNEL_SUBSYSTEM"=>"pci", "_KERNEL_DEVICE"=>"+pci:0000:00:03.0", "_UDEV_SYSNAME"=>"0000:00:03.0", "_SOURCE_MONOTONIC_TIMESTAMP"=>"582211", "MESSAGE"=>"pci 0000:00:03.0: reg 0x10: [io  0xc060-0xc07f]"}}

which makes the message an invalid JSON. In post-treatment I can just do the replacement to get it to valid JSON again, but I'm trying to do it automatically when parsing the original message. Is that possible? I've read the documentation but cannot find a way to do so.

Thanks

[Eduardo Silva]

the output you are seeing is not JSON, is a human-readable version of the msgpack data, if you are using stdout plugin make sure to enable the option "format json_lines", that will do a proper JSON formatting.

--
You received this message because you are subscribed to the Google Groups "Fluent-Bit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluent-bit+...@googlegroups.com.
To post to this group, send email to fluen...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fluent-bit/599d4a3c-2d04-4bdb-a486-a265657ed4af%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Eduardo Silva

Open Source, Treasure Data

http://www.treasuredata.com/opensource

 

[Hector Oswaldo Caballero]

Ah, OK, that could explais the issue. In reality, I'm using a custom Fluentd plugin that sends the output to an external UDP/TCP server, so the issue should be in the way I'm handling the formatting of the msgpack data. I'll check from that side.

Thanks for the hint.