Escaped characters in Json output

[David Kleiner]

Hello,

I'm having hard time with getting a proper json output from strings with escaped characters.

For example:

% cat a.sh

#!/bin/bash

-----

set -x

a=123

if [[ $a != '1234' ]]

then

echo no

fi

echo "123"

-----

$ cat a.log

+ a=123

+ [[ 123 != \1\2\3\4 ]]

+ echo no

+ echo 123

$ ./a.sh >> a.log 2>&1

relevant part:

$ fluent-bit -R parsers.conf -i tail  -p path=/tmp/a.log -o stdout

[1] tail.0: [1537474360.000000000, {"log"=>"+ [[ 123 != \1\2\3\4 ]]"}]

To be accepted as a valid JSON, it should look like this:

[1] tail.0: [1537474360.000000000, {"log"=>"+ [[ 123 != \\1\\2\\3\\4 ]]"}]

=======

Any suggestions?  

Thank you,

David

[Eduardo Silva]

hi David, 

note that the stdout plugin do not print out messages in JSON format, it just convert from msgpack (binary representation) to a readable format which is not JSON. Since v0.14.3 the stdout plugin supports a new configuration property called format where you can specify it prints out JSON with: -o stdout -p format=json_lines

let me know if that works for you,

best

--
You received this message because you are subscribed to the Google Groups "Fluent-Bit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluent-bit+...@googlegroups.com.
To post to this group, send email to fluen...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fluent-bit/94c50934-b211-4595-9da1-3ecfd385134e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Eduardo Silva

Open Source, Treasure Data

http://www.treasuredata.com/opensource

 

[David Kleiner]

Hi Eduardo,

Thank you for your response, here's the actual problematic case I was trying to reproduce.

Log file (the line from kubernetes logs) - source

{"log":"+ [[ stage != \\d\\e\\v ]]\n","stream":"stderr","time":"2018-09-24T17:53:50.205588946Z"}

Fluent-bit config:

------------

[SERVICE]

    Parsers_File /tmp/parsers.conf

[INPUT]

    Name tail

    Parser docker

    Path /tmp/j.log

[OUTPUT]

    Name kafka

    Match *

    Brokers kafka-client:9092

    Topics mike-test

    Format json

------------

With 

[PARSER]

    Name         docker

    Format       json

    Time_Key     time

    Time_Format  %Y-%m-%dT%H:%M:%S.%L

    Time_Keep    On

    # Command      |  Decoder | Field | Optional Action

    # =============|==================|=================

    Decode_Field_As   escaped    log

Kafka:

    {"@timestamp":1537811630.205589, "log":"+ [[ stage != de\v ]]\n", "stream":"stderr", "time":"2018-09-24T17:53:50.205588946Z"}

With 

    Decode_Field_As   escaped_utf8    log

Kafka (note the extra backslash, \\\v):

   {"@timestamp":1537811630.205589, "log":"+ [[ stage != \\d\\e\\\v ]]\\n", "stream":"stderr", "time":"2018-09-24T17:53:50.205588946Z"}

Without decoding (correct, since docker produces valid JSON):

  

   {"@timestamp":1537811630.205589, "log":"+ [[ stage != \\\\d\\\\e\\\\v ]]\\n", "stream":"stderr", "time":"2018-09-24T17:53:50.205588946Z"}

--------

Thank you,

David

[Eduardo Silva]

hi David, 

I think the proper way to address the conversion is to use the following parser:

[PARSER]

    Name         docker-utf8

    Format       json

    Time_Key     time

    Time_Format  %Y-%m-%dT%H:%M:%S.%L

    Time_Keep    On

    # Command      |  Decoder | Field | Optional Action

    # =============|==================|=================

    Decode_Field_As   escaped            log             do_next

    Decode_Field_As   escaped_utf8    log

I tested that locally and looks fine, please confirm..

best, 

--

You received this message because you are subscribed to the Google Groups "Fluent-Bit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluent-bit+...@googlegroups.com.
To post to this group, send email to fluen...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/fluent-bit/68efe51a-dcd2-4a53-bb62-46cb634cc713%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.