Fluentd tail plugin not reading very lengthy files

[Maithri V M]

I have setup fluentd to read dump of the log file using tail plugin with custom format. Though it is able to parse successfully, it is failing to parse very lengthy files (size~300,000 lines). While POS file gets updated every time fluentd is restarted, only the first few set of lines (~150) are being parsed.,hence the stored out file contains duplicated records of the same set, instead of progressing it further upon repetition. Please find the below code:

<source>
  type tail
  path ~/axon.log
  pos_file ~/axonlog.log.pos

  read_from_head true
  read_lines_limit 10000

  format multiline
  format_firstline /^(?<host>([^\n])*):(?<path>(.*))#/
  format1 /(?<host>([^\n])*):(?<path>(.*))# (?<command>([^\n])*)([\n](?<output>(.|\n)*))?/

  tag axonlog
</source>

<match axonlog>
  type copy
  <store>
    type file
    path ~/axon_out.log
  </store>
  <store>
    @type elasticsearch
    logstash_format true
    flush_interval 10s # for testing
  </store>
</match>

I followed similar issue as found here, and hence tested both log out file as well as elastic search output reflect the same behavior. Please let me know if anyone else has come across this issue.

[Maithri V M]

Besides, I missed to mention that there are multiple output log (stored with copy) files created, which are all duplicates.

[Mr. Fiber]

> only the first few set of lines (~150) are being parsed.,hence the stored out file contains duplicated records of the same set

We need more detailed information to investigate the problem.

- How many duplicated records from one original line? 1 : 2 or 1 : random?

- duplicated means time and record are same or time is different but records are same?

> multiple output log (stored with copy) files created

It means stored json in out_file file contains multilines?

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Maithri V M]

Thanks for the quick turn around!

Basically the source log file contains >300,000 lines, and I'm expecting fluentd to parse it all upon start, as it's configured with "read_from_head" true. However, it's only parsing 150 lines and frames 9 valid log events into both log out(json) as well as elastic search.
Now I assuming I'll have to restart the fluentd to mark it to proceed further with log file processing(?!) please validate my understanding. When I restart again, it again parses the same first 150 lines and generates the replica of the log events as generated earlier, hence creating the duplicates. (creates 4-5 out.log files at times.)

Note: That 9th event is very huge record starting from line #70 till #250,000, and fluentd ends up having that 9th event with the data till line#9. There are some valid log events only after line#250,000 which never gets through fluentd. Never crosses further.

[Lance N.]

If I was modeling this, I would write a variant of the multiline plugin. It would:

1) emit each line in the multiline as a separate record.

2) add 2 counters to each record

2a) the first counter increments for every new multiline record

2b) the second counter increments for every line in the multiline record, and resets to 0 when a new multiline record starts.

This way, it is not necessary for Fluentd to store 250k lines of text in memory at once.

[Maithri V M]

Thanks, Lance for the hint!

It would be great if you could redirect me to any sample code to demonstrate the said implementation, I'm quite new to this fluentd.
What would be the max. # of lines a file can have for fluentd to index it at once?

[Lance Norskog]

You're welcome. I haven't done any Ruby coding, so can't help you. As the max # of lines of code, it's about how much memory it takes to hold the text. I would limit it to hundreds. I think multiline was written to help with stack traces, which are rarely more than a few hundred.

Lance

--
You received this message because you are subscribed to a topic in the Google Groups "Fluentd Google Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/fluentd/TPZg1f-Qlpg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to fluentd+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Lance Norskog
lance....@gmail.com
Redwood City, CA

[Mr. Fiber]

I tried simplified 250000 multiline logs and it worked.

Here is my config:

<source>
  @type tail
  path /path/to/axon.log
  pos_file /path/to/axonlog.log.pos

  read_from_head true
  read_lines_limit 10000
  format multiline

  format_firstline /\d{4}-\d{1,2}-\d{1,2}/
  format1 /^(?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}) \[(?<thread>.*)\] (?<level>[^\s]+)(?<message>.*)/

  tag axonlog
</source>

<match axonlog>

  @type file
  path /path/to/out.log
  flush_at_shutdown true
</match>

I generate java like stacktrace log with following Ruby script.

puts "2016-3-03 14:27:33 [main] INFO  Main - Start"
puts "2016-3-03 14:27:34 [main] ERROR Main - Exception javax.management.RuntimeErrorException: null"
250000.times {
  puts "    at Main.main(Main.java:16) ~[bin/:na]"
}
puts "2016-3-03 14:27:40 [main] INFO  Main - End"

So basically, the in_tail with multiline works well with large multiline logs.

Maybe, your regexp can't handle your logs correctly?

 

I can't reproduce duplicated record issue with restart.

[Maithri V M]

Hello Mr.Fiber,

Here I have updated the format to address those formatting errors and now I'm not getting those format exceptions. However, its not working when a line size grows beyond 70,000 lines for one record. Meaning, when a records' data spans more than 70K lines, it simply ignores that record (not logged into output file/elastic search) and same thing gets printed in the fluentd var log without mentioning whats the problem with those lines.

I have attached the sample logs that I am trying (the one with 70k lines which works fine and the other with 72k where is starts failing). Here is the code with updated format:

<source>

   type tail

   path /home/maithri/elasticsearch-2.3.0/axon.log

   pos_file /home/maithri/elasticsearch-2.3.0/axonlog.pos

   read_from_head true

   read_lines_limit 10000

format multiline

format_firstline /^(?<host>(.*)):(?<path>(.*))# /

format1 /(?<host>([^:])*):(?<path>([^#])*)# (?<command>([^\n])*)([\n](?<output>(.|\n)*))?/

   tag axonlog

</source>

<match axonlog>

  type copy

  <store>

    type file

    path /home/maithri/elasticsearch-2.3.0/axon_out.log

  </store>

  <store>

    @type elasticsearch

    logstash_format true

    flush_interval 10s # for testing

  </store>

</match>

It would be great if you could explain if there is any threshold on the #of lines for a given record.

Another important thing to consider is that, I have noticed the following log during one of the runs, but wasn't really consistently being reported.

2016-05-26 11:48:16 +0530 [warn]: Size of the emitted data exceeds buffer_chunk_limit.

2016-05-26 11:48:16 +0530 [warn]: This may occur problems in the output plugins ``at this server.``

2016-05-26 11:48:16 +0530 [warn]: To avoid problems, set a smaller number to the buffer_chunk_limit

2016-05-26 11:48:16 +0530 [warn]: in the forward output ``at the log forwarding server.``

2016-05-26 11:48:16 +0530 [warn]: Size of the emitted data exceeds buffer_chunk_limit.

2016-05-26 11:48:16 +0530 [warn]: This may occur problems in the output plugins ``at this server.``

2016-05-26 11:48:16 +0530 [warn]: To avoid problems, set a smaller number to the buffer_chunk_limit

2016-05-26 11:48:16 +0530 [warn]: in the forward output ``at the log forwarding server.``

2016-05-26 11:48:16 +0530 [info]: Connection opened to Elasticsearch cluster => {:host=>"localhost", :port=>9200, :scheme=>"http"}

Looking forwards for your valuable inputs.

--
You received this message because you are subscribed to a topic in the Google Groups "Fluentd Google Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/fluentd/TPZg1f-Qlpg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to fluentd+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Cheers,
My3

[Maithri.vm]

Hello there, just checking if you got everything that was needed to prove my findings..

Thanks and regards,

Maithri V M

Sent from my iPhone

<axon_notworking.log.zip>

<axon_working.log.zip>