Fluentd Grok multiline parsing in filter?

[Marco Pas]

Hi there,

i am trying to parse multiline data coming from a Java Application but I seem to be stuck in trying to get parse multiline data.

fluentd.conf

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

<filter demo.**>

    @type parser

    format multiline_grok

    <grok>

        pattern %{LOGLEVEL:log_level} %{JAVACLASS:logger} - %{GREEDYDATA:message}

    </grok>

    <grok>

        pattern %{JAVACLASS:logger}: %{GREEDYDATA:message}

    </grok>

    multiline_start_regexp /^\s/

    reserve_data yes

    key_name log

</filter>

<match **>

    @type stdout

</match>

Some sample log data is as follows:

ERROR org.apache.tomcat.jdbc.pool.ConnectionPool - Unable to create initial connections of pool.
org.postgresql.util.PSQLException: The connection attempt failed.
   at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:249)
   at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
   at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:127)

I tried all kinds of combinations for the multiline regex but i seem to be unable to grab multiple lines.

Any hint on how this can be done? 

Thanks in advance!

[Mr. Fiber]

okimoto-san,

Could you check this question?

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[okimoto]

Hi, I'm maintainer of fluent-plugin-grok-parser.

I've checked your configuration in my environment.
But I'm not familiar with Java.

Main.java:

import java.util.HashMap;
import java.util.Map;
import org.fluentd.logger.FluentLogger;

public class Main {

    private static FluentLogger LOG = FluentLogger.getLogger("demo");

    /**
     * @param args
     */
    public static void main(String[] args) {
        Map<String, Object> data = new HashMap<String, Object>();
        String message =
                "ERROR org.apache.tomcat.jdbc.pool.ConnectionPool - Unable to create initial connections of pool.\n"
                + "org.postgresql.util.PSQLException: The connection attempt failed.\n"
                + "  at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:249)\n"
                + "  at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)\n"
                + "  at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:127)";
        data.put("log", message);
        LOG.log("logger", data);
    }
}

Fluentd output:

2016-05-13 14:51:39 +0900 demo.logger: {"log":"ERROR org.apache.tomcat.jdbc.pool.ConnectionPool - Unable to create initial connections of pool.\norg.postgresql.util.PSQLException: The connection attempt failed.\n  at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:249)\n  at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)\n  at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:127)","log_level":"ERROR","logger":"org.apache.tomcat.jdbc.pool.ConnectionPool","message":"Unable to create initial connections of pool.\norg.postgresql.util.PSQLException: The connection attempt failed.\n  at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:249)\n  at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)\n  at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:127)"}

BTW, you can use fluent-plugin-concat If you want to concatenate multiple lines messages separated in multiple events.

Thanks.

2016年5月12日木曜日 22時04分54秒 UTC+9 Marco Pas:

[Marco Pas]

Thanks for pointing me to the Fluentd Concat plugin!

[Fahimeh Ashrafy]

Hello I have the same problem with php log file and rabbit mq

<source>
  @type forward
  port 25289
</source>
<filter *.*.*.*.php>

    @type parser

    format multiline_grok

    grok_pattern \[%{DATA:raw_timestamp}\] %{DATA:name} %{WORD:loglevel} %{GREEDYDATA:message}

    multiline_start_regexp /(Stack trace:)|(^#+)|(^\"\")|(  thrown+)|(^\s)/

    reserve_data yes

         key_name message    

</filter>
<match **>
  @type stdout
</match>

one of the log file is like:

[01-Jun-2016 12:16:26 UTC] PHP Fatal error:  Uncaught exception 'PDOException' with message 'SQLSTATE[HY000]: General error: 2006 MySQL server has gone away' in /var/www/website/includes/database/database.inc:2171
Stack trace:
#0 /var/www/website/includes/database/database.inc(2171): PDOStatement->execute(Array)
#1 /var/www/website/includes/database/database.inc(683): DatabaseStatementBase->execute(Array, Array)
#2 /var/www/website/includes/database/select.inc(1265): DatabaseConnection->query('SELECT w.*\nFROM...', Array, Array)
#3 /var/www/website/sites/all/modules/partotech/cron2/pt_cron2.module(709): SelectQuery->execute()
#4 /var/www/website/sites/all/modules/partotech/cron2/pt_cron2.worker.inc(278): pt_cron2_load_worker('14')
#5 [internal function]: pt_cron2_worker_shutdown('14', 1464766262)
#6 {main}
  thrown in /var/www/website/includes/database/database.inc on line 2171

it just consider first line 

[01-Jun-2016 12:16:26 UTC] PHP Fatal error:  Uncaught exception 'PDOException' with message 'SQLSTATE[HY000]: General error: 2006 MySQL server has gone away' in /var/www/website/includes/database/database.inc:2171




 could you please help me?

Thanks a lot