time_format problem with tail multi-line

[Lei Chen]

I have the following source:

<source>

  type tail

  path /var/log/multi-line.log

  format_firstline /^(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\,\d{3})/

  format multiline

  format1 /^(?<time>d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\,\d{3}) (?<level>[^\s]+) (?<message>.*)/

  time_format %Y-%m-%d %H:%M:%S,%L

  pos_file /var/log/td-agent/multi.line.pos

  tag hostname.multi.line

  read_from_head true

</source> 

This is to parse multi-line stack traces like this:

2015-08-02 18:42:15,367 ERROR [org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/someuri].[spring-rest]] [ajp-8289-12] [StandardWrapperValve.java:246] Servlet.service() for servlet spring-rest threw exception

java.io.IOException: Not in GZIP format

        at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:141)

...

I get the following warning from starting td-agent:

2015-08-03 08:50:54 +0000 [warn]: fluent/engine.rb:81:block in run_configure: parameter 'time_format' in <source>

  type tail

  path /var/log/multi-line.log

  format_firstline /^(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\,\d{3})/

  format multiline

  format1 /^(?<time>d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\,\d{3}) (?<level>[^\s]+) (?<message>.*)/

  time_format %Y-%m-%d %H:%M:%S,%L

  pos_file /var/log/td-agent/multi.line.pos

  tag hostname.multi.line

  read_from_head true

</source> is not used.

I'm not sure why there is warning with time_format, I passed this through fluenttular and it worked OK there.  How do I correctly configure fluentd source for above stack trace logs?

The output is to ES, and I don't see any output there:

<match *.**>

  type elasticsearch

  host localhost

  port 9200

  include_tag_key true

  tag_key @log_name

  logstash_format true

  flush_interval 10s

</match>

[Mr. Fiber]

The following line is the cause.

https://github.com/fluent/fluentd/blob/e569d89a8c3006c6d8884181bc0f18b12a30004b/lib/fluent/parser.rb#L172

RegexpParser wraps multiline's Config::Element and it causes inconsistent unused parameter state.

time_format is set correctly so this is not critical.

Could you open an issue on fluentd repository?

I will fix it later.

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mr. Fiber]

BTW, you did typo in configuration.

> format1 /^(?<time>d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\,\d{3}) (?<level>[^\s]+) (?<message>.*)/

Need \ before d{4}. This is why "I don't see any output there:"

On Mon, Aug 3, 2015 at 5:57 PM, Lei Chen <chen...@gmail.com> wrote:

[Lei Chen]

yes, i fixed that later on.  i didn't realize fluentd didn't have support for sub-second time resolution -- i hope this can be added soon under issue 461.

thanks

--
You received this message because you are subscribed to a topic in the Google Groups "Fluentd Google Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/fluentd/7k-aG_YL9mI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to fluentd+u...@googlegroups.com.