[Multiline parser] Problem with Java multiline parsing on K8S platform

[Marc Kamerbeek]

I'm trying for days now to get my multiline fluent-bit java log parser to work. I guess i'm close now, but no luck so far.

My fluentbit configuration:

    [SERVICE]

        Flush        1

        Daemon       Off

        Log_Level    info

        Parsers_File parsers_custom.conf

    [INPUT]

        Buffer_Chunk_Size 400k

        Buffer_Max_Size 5MB

        DB /var/log/containers/fluent-bit.db

        Mem_Buf_Limit 5MB

        Multiline On

        Multiline_Flush 5

        Name tail

        Parser docker

        Parser_Firstline java_multiline

        Path /var/log/containers/*_online-*.log

        Refresh_Interval 5

        Tag kube.spring.*

    [FILTER]

        K8S-Logging.Parser On

        Kube_URL https://${KUBERNETES_SERVICE_HOST}:443

        Match kube.*

        Merge_JSON_Log On

        Name kubernetes

        tls.verify Off

    [PARSER]

        Format regex

        Name spring

        Regex (?<apptime>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}\.\d{3})\s+\[(?<process>[^\s]+)\]\s+(?<priority>[^\s]+)\s+(?<class>[^\s]+)\s(\[(?<mdc>[^\]]*)\])?\s+-\s+(?<message>.*)

        Time_Format %Y-%m-%d %H:%M:%S.%L

        Time_Key apptime

    [PARSER]

        Format regex

        Name java_multiline

        Regex (?<apptime>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}\.\d{3})

        Time_Format %Y-%m-%d %H:%M:%S.%L

        Time_Keep true

        Time_Key apptime

    [PARSER]

        Format json

        Name docker

        Time_Format %Y-%m-%dT%H:%M:%S %z

        Time_Key time

    [OUTPUT]

        Name  es

        Match *

        Host  141.XX.XXX.XX

        Port  9200

        Logstash_Format On

        Logstash_Prefix  kubernetes-2-tst

        Include_Tag_Key true

        Retry_Limit False

My java stacktrace:

2018-03-12 13:11:07.858 [http-nio-8080-exec-1] WARN  org.springframework.cloud.netflix.zuul.filters.post.SendErrorFilter [] - Error during filtering

com.netflix.zuul.exception.ZuulException: Connect to connection-service:80 [connection-service/10.233.9.171] failed: connect timed out

        at org.springframework.cloud.netflix.zuul.util.ZuulRuntimeException.<init>(ZuulRuntimeException.java:33) 

        at org.springframework.cloud.netflix.zuul.filters.route.SimpleHostRoutingFilter.run(SimpleHostRoutingFilter.java:207) 

        at com.netflix.zuul.ZuulFilter.runFilter(ZuulFilter.java:112) 

        at com.netflix.zuul.FilterProcessor.processZuulFilter(FilterProcessor.java:193)

        at com.netflix.zuul.FilterProcessor.runFilters(FilterProcessor.java:157)

        at com.netflix.zuul.FilterProcessor.route(FilterProcessor.java:118)

        at com.netflix.zuul.ZuulRunner.route(ZuulRunner.java:96)

        at com.netflix.zuul.http.ZuulServlet.route(ZuulServlet.java:116)

        at com.netflix.zuul.http.ZuulServlet.service(ZuulServlet.java:81)

        at org.springframework.web.servlet.mvc.ServletWrappingController.handleRequestInternal(ServletWrappingController.java:157)

        at org.springframework.cloud.netflix.zuul.web.ZuulController.handleRequest(ZuulController.java:44)

        ....and some more lines

I tested the first line parser on http://rubular.com/ and it matches the time.

But for some reason it ends up in Elasticsearch as separate lines.

I'm using the dev version 0.13 tag 0.10. I'm also using the logging.parser annotation to tell what parser should be used for the pod.

Any help would be appreciated.

[Eduardo Silva]

hi Marc, 

I've tested the following setup which works:

-- fluent-bit.conf --

[SERVICE]
    Flush        1
    Daemon       Off
    Log_Level    info

    Parsers_File parser_java.conf

[INPUT]
    Name             tail
    Path             java.log
    Multiline        on
    Parser_Firstline java_multiline

[OUTPUT]
    Name  stdout

    Match *

-- EOF --

-- Java parser file --

[PARSER]

    Format regex

    Name java_multiline

    Regex (?<apptime>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}\.\d{3})

    Time_Format %Y-%m-%d %H:%M:%S.%L

    Time_Keep true

    Time_Key apptime

-- EOF -- 

-- Log file example --

2015-03-12 13:11:07.858 [http-nio-8080-exec-1] WARN  org.springframework.cloud.netflix.zuul.filters.post.SendErrorFilter [] - Error during filtering

-- EOF -- 

Please find the attached config files and sample log, let me know how it goes

best

--
You received this message because you are subscribed to the Google Groups "Fluent-Bit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluent-bit+unsubscribe@googlegroups.com.
To post to this group, send email to fluen...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fluent-bit/a8e177e6-0517-47d4-a371-d296185c424a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Eduardo Silva

Open Source, Treasure Data

http://www.treasuredata.com/opensource

 

[Panchuk Bogdan]

This config does not solve the problem Marc Kamerbeek (as well as most other people need to solve):
parsing docker logs (with JSON parser) first and then applying multi-line Parser_Firstline to its contents...
Fluent Bit doc explicitly states, that if Multiline option is On for "tail" input, Parser is not used.
Unfortunately I can not find any example, how to use JSON parser with Multiline...

пятница, 16 марта 2018 г., 18:11:41 UTC+2 пользователь Eduardo Silva написал:

To unsubscribe from this group and stop receiving emails from it, send an email to fluent-bit+...@googlegroups.com.

To post to this group, send email to fluen...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fluent-bit/a8e177e6-0517-47d4-a371-d296185c424a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Don Bowman]

not directly an answer to this, but the only way I made the java logging work properly with its huge stack traces was to switch to json logging from log4j config.

then it comes as a single 'line' in the docker output and fluent-bit parses it correctly since its json.

To view this discussion on the web visit https://groups.google.com/d/msgid/fluent-bit/4988e21f-0357-47f7-b22d-237d00d287c9%40googlegroups.com.

[Marc Kamerbeek]

Thats exactly how I solved the problem. Its even better for cpu usage. No parsing needed anymore. All the 1000+ pods are now logging in json. 

Op di 26 mrt. 2019 om 13:09 schreef Don Bowman <d...@agilicus.com>:

You received this message because you are subscribed to a topic in the Google Groups "Fluent-Bit" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/fluent-bit/XoOibAxGGeI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to fluent-bit+...@googlegroups.com.

To post to this group, send email to fluen...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/fluent-bit/CAM8ZkJja-90qZvzMTB2Q%2BQcGftFUNyS538nSEzgdqnpu0SbDHA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Marc Kamerbeek
It Ferset 35
8754 LB Makkum
Email: marckam...@gmail.com
Mobiel: +31 6 18114205

[Eduardo Silva]

would you please share some steps and config done to accomplish this? I would like to add that info to our documentation.

To view this discussion on the web visit https://groups.google.com/d/msgid/fluent-bit/CAKMFkQ%2BSC6BUsmps0sEFfWnpgAdd1qS33niUB6cxrg5_NE%3DL_w%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Eduardo Silva
Principal Engineer  | Arm
. . . . . . . . . . . . . . . . . . . . . . . . . . . 
m. +506 70138007
Arm.com
Treasuredata.com

[Don Bowman]

I used:

```<?xml version="1.0" encoding="UTF-8" ?> <Configuration status="WARN"> <Appenders> <Console name="Console" target="SYSTEM_OUT"> <JSONLayout compact="true" eventEol="true" properties="true"/> </Console> </Appenders> <Loggers> <Root level="info"> <AppenderRef ref="Console"/> </Root> <Logger name="org.apache.druid.jetty.RequestLog" additivity="false" level="WARN"> <AppenderRef ref="Console"/> </Logger> </Loggers> </Configuration>```

Another user on slack used:

```

<?xml version="1.0" encoding="UTF-8"?>

<configuration>

    <appender name="json" class="ch.qos.logback.core.ConsoleAppender">

        <layout class="ch.qos.logback.contrib.json.classic.JsonLayout">

            <jsonFormatter

                    class="ch.qos.logback.contrib.jackson.JacksonJsonFormatter">

                <prettyPrint>false</prettyPrint>

            </jsonFormatter>

            <timestampFormat>yyyy-MM-dd' 'HH:mm:ss.SSS</timestampFormat>

            <appendLineSeparator>true</appendLineSeparator>

        </layout>

    </appender>

    <root level="INFO">

        <appender-ref ref="json"/>

    </root>

</configuration>

```

[Eduardo Silva]

since I know nothing about Java environment, where that content should be placed ?

[Don Bowman]

the user will have a log4j.xml file somewhere on 'classpath', those files are the content of it.

https://stackoverflow.com/questions/29387007/does-log4j-support-json-format

[Eduardo Silva]

got it thanks, I will share a draft soon

[Marc Kamerbeek]

This is a configuration for logback in combination with Spring Framework. It can also be used for non Spring projects with some changes.

Place a logback.xml file on your classpath with the following content:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <include resource="org/springframework/boot/logging/logback/defaults.xml" />
    <include resource="org/springframework/boot/logging/logback/console-appender.xml" />
    <appender name="consoleAppender" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="net.logstash.logback.encoder.LogstashEncoder">
            <fieldNames>
                <timestamp>application_timestamp</timestamp>
            </fieldNames>
            <!-- This 'fixes' double quote problem for elastic -->
            <jsonFactoryDecorator class="net.logstash.logback.decorate.CharacterEscapesJsonFactoryDecorator">
                <escape>
                    <targetCharacterCode>34</targetCharacterCode>
                    <escapeSequence>'</escapeSequence>
                </escape>
            </jsonFactoryDecorator>
            <!-- This shortens extremely long stacktraces. Not always needed --> 
            <throwableConverter class="net.logstash.logback.stacktrace.ShortenedThrowableConverter">
                <maxDepthPerThrowable>30</maxDepthPerThrowable>
                <maxLength>2048</maxLength>
                <shortenedClassNameLength>20</shortenedClassNameLength>
                <exclude>sun\.reflect\..*\.invoke.*</exclude>
                <exclude>net\.sf\.cglib\.proxy\.MethodProxy\.invoke</exclude>
                <rootCauseFirst>true</rootCauseFirst>
                <inlineHash>true</inlineHash>
            </throwableConverter>
        </encoder>
    </appender>
    <root level="INFO">
        <!-- Services testing locally to have readable logging -->
        <springProfile name="!kubernetes">
            <appender-ref ref="CONSOLE" />
        </springProfile>
        <!-- Running services in kubernetes to have json logging --> 
        <springProfile name="kubernetes">
            <appender-ref ref="consoleAppender"/>
        </springProfile>
    </root>
</configuration>

 Needed dependency for logstash encoder (logback is shipped with springboot).

<dependency>
   <groupId>net.logstash.logback</groupId>
   <artifactId>logstash-logback-encoder</artifactId>
   <version>5.3</version>
</dependency>

 

[Eduardo Silva]

thanks everyone, we will write a document about this.

--

You received this message because you are subscribed to the Google Groups "Fluent-Bit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluent-bit+...@googlegroups.com.
To post to this group, send email to fluen...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/fluent-bit/abc02805-c4f2-4474-8854-e29a39093d71%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.