Question about milisecond support with Java logs for example

[Cvijan Uros]

Hello, 

I saw couple of threads regarding this issue, not sure if i quite understand it. There are milliseconds in the time part of the log, and i parse it as such, but in kibana there are only ,000 at the end. 

Is there a clean way for me to tail the log, get the full time with the millisecond, forward parsed log to fluentd, and then send it to elasticsearc, but with the time-stamp having the millisecond.

I have seen this issue, but that works with present time, and not the time from the log:

https://github.com/uken/fluent-plugin-elasticsearch/issues/39

For ie, if I start put new configuration of fluentd, and want to import logs from the files that happened yesterday, i dont get times from the log, but i get current time. 

I installed everything 10 days ago, so i guess all the versions are the newest ones. 

Best regards,  

Uros

[Mr. Fiber]

Using keep_time_key in in_tail and time_key in out_elasticsearch may resolve your problem.

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Cvijan Uros]

Hi Masahiro, 

Thanks for the fast reply. When i do it like that, I do get time key reserved with full milli-seconds, logs get to the fluentd collector, in there i put time_key time in leasticsearch plugin, but it apperas that it threats it as string. And then when i try to look it up in Kibana, it it not treating that index as a time stamp, but i have @timestamp as a string, like a part of the index. Please see attach. Should i put some type in in_tail plugin for time? Or maybe I am missing something. 

Flentd forwarder conf, this is just testing...

<source>

 type tail

 read_from_head true

 path /home/logs/catalina.log

 keep_time_key true

 format multiline

 format_firstline /^(?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2},\d{1,3})/

 format1 /(?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2},\d{1,3}) \[(?<thread>[^ ]*)\] (?<level>[^\s]+)(?<message>.*)/

 time_format %Y-%m-%d %H:%M:%S,%L

 tag testing

</source>

<match testing>

 type copy

  <store>

   type stdout

  </store>

 <store>

  type forward

 <server>

    name collector

    host 192.168.186.34

    port 24224

    weight 60

  </server>

 </store>

</match>

Fluentd collector conf: 

<source>

 type forward

 port 24224

 bind 0.0.0.0

</source>

<match testing>

 type copy

 <store>

   type stdout

 </store>

 <store>

  type elasticsearch

  host localhost

  port 9200

  logstash_format true

  type_name nginx

  logstash_prefix test

  logstash_dateformat %Y.%m

  time_key time

 </store>

</match>

So what I would like to achieve is to have everything as it is for the default version, just instead of ,000 for milliseconds to have the exact value in the @timestamp.  

Or if that is not possible, leave @timestamp as is, but add new field in the record with the exact value of my date/time which I can sort by, but with the above configuration, my time is string and it seems that it is not sorting it the way it should. 

Thank you, 

Uros

[Mr. Fiber]

From my test, the type of @timestamp is date: https://gyazo.com/3b18af83359ff6e37a870e1e994c2490

# conf

<source>
  type tail
  path /path/to/test.csv
  format csv
  read_from_head true
  keep_time_key true
  keys time,event,message
  tag es.test
</source>

<match es.**>
  type elasticsearch
  buffer_chunk_limit 50m
  logstash_format true
  flush_at_shutdown true
  flush_interval 2s
  reload_on_failure true
  time_key time
</match>

# test.csv

2016-03-29T16:30:54.631+01:00,TRANS0,boodschap

# versions

- Elasticsearch 2.1.0

- fluentd v0.12.22

- fluent-plugin-elasticsearch 1.4.0

[Yasin Amadmia]

Hi,

I am hitting the same issue. 

We have a Java app where we get logs in the format 

"INFO  2016-04-19 19:53:16,086000 [localhost-startStop-1] [Reflections] Reflections took 89 ms to scan 1 urls, producing 9 keys and 16 values"

I have used tail plugin and extracted the relevant bits out as below

<source>

  type tail

  tag some_tag

  time_format %Y-%m-%d %H:%M:%S,%L

  format multiline

  format_firstline /(?:INFO|WARNING|WARN|ERROR|CRITICAL|CRIT)/

  format1 /(?<level>INFO|WARN|WARNING|CRITICAL|CRIT|ERROR)  (?<time>\S+ \S+) (?<message>.*)/

  keep_time_key true

  pos_file /var/log/td-agent/some_log.pos

  path /opt/tomcat/logs/console.log

</source> 

<match **.**>

    type elasticsearch

    host 10.6.20.210

    port 9200

    index_name fluentd

    type_name fluentd

    logstash_format true

    logstash_prefix logstash

    time_key time

</match>

When I set time_key time, within Kibana, ON my logstash index, it is not able to find any Time-Field name. I suspect, time is coming out as 'String'.

When I comment that line out, I can see time filed as "time:2016-04-19 19:53:16,086000" but "@timestamp" still has Milliseconds as 000 so it is not able to order the events correctly. We have many events that happen at same second and need to order them by millisecond to make sense.

I also tried setting the filed as timestamp instead of time within filter as below but still don't get any Time-Field name.

format1 /(?<level>INFO|WARN|WARNING|CRITICAL|CRIT|ERROR)  (?<timestamp>\S+ \S+) (?<message>.*)/

I want that @timestamp should be the time of the event including millisecond, and want to surface the logs in Kibana but currently stuck at it.

Thanks.

[Mr. Fiber]

> I want that @timestamp should be the time of the event including millisecond, and want to surface the logs in Kibana but currently stuck at it.

This seems not fluentd issue.

Elasticsearch can't handle your time value, 2016-04-19 19:53:16,086000,  as a timestamp.

I tried two format time values without mapping setting.

- 2016-04-19T19:54:16.086: Elasticsearch uses date type for this value

- 2016-04-19 19:53:16,086000: Elasticsearch uses string type for this value

So you should set proper mapping when you use non-default date format.

This e-mail and any attachments are confidential and intended solely for the use of the addressee only. If you have received this message in error, you must not copy, distribute or disclose the contents; please notify the sender immediately and delete the message.

This message is attributed to the sender and may not necessarily reflect the view of Travis Perkins Trading Company Limited, 733503, Lodge Way House, Lodge Way, Harlestone Road, Northampton, NN5 7UG or its parent company Travis Perkins plc (Registered in England No. 824821; Lodge Way House, Lodge Way, Harlestone Road, Northampton, NN5 7UG; VAT number 408556737) and any of its subsidiaries. Agreements binding Travis Perkins Trading Company Limited may not be concluded by means of e-mail communication.

E-mail transmissions are not secure and Travis Perkins Trading accepts no responsibility for changes made to this message after it was sent. Whilst steps have been taken to ensure that this message is virus free, Travis Perkins Trading accepts no liability for infection and recommends that you scan this e-mail and any attachments.

[Yasin Amadmia]

@repeatedly thank. Indeed my tame value was not correct. I now have configured my app to dump out time in format "2016-04-20 12:37:48,165" and have created mappings within elasticsearch so that I have

curl -XPUT 'http://10.6.98.232:9200/_template/custom_time' -d '

{

  "template" : "logstash*",

  "mappings": {

    "fluentd": {

      "properties": {

        "time": {

          "type": "date",

          "format": "yyyy-MM-dd HH:mm:ss,SSS"

        },

        "@timestamp": {

          "type": "date",

          "format": "yyyy-MM-dd HH:mm:ss,SSS"

        }

      }  

    }

  }

}'

I can see that elasticsearch now detects them both as 'date'

However, When I have the above mappings, nothing is shown (Discovered) in Kibana. It does create the index and let me choose 'time' OR '@timestamp' as Time-Fields but nothing is getting discovered.

When I have template to set type for 'time' only (and not for '@timestamp'), then I can still choose 'time' OR ''@timestamp' as Time-Fields but only 'Discover' when time-filed is '@timestamp'

 My configs are:

<source>

  type tail

  tag some_tag

  format multiline

  format_firstline /(?:INFO|WARNING|WARN|ERROR|CRITICAL|CRIT)/

  format1 /(?<level>INFO|WARN|WARNING|CRITICAL|CRIT|ERROR)  (?<time>\S+ \S+) (?<message>.*)/

  keep_time_key true

  time_format %Y-%m-%d %H:%M:%S,%L

  pos_file /var/log/td-agent/hybris.console.pos

  path /tank/hybris/tomcat/logs/console.log

</source>

<match some_tag>

    type elasticsearch

    host 10.6.98.232

    port 9200

    index_name fluentd

    type_name fluentd

    logstash_format true

    logstash_prefix logstash

    #time_key time

</match>

When I uncomment 'time_key time', I can't discover anything. 

On Tuesday, 29 March 2016 14:34:23 UTC+1, Cvijan Uros wrote:

[Mr. Fiber]

> However, When I have the above mappings, nothing is shown (Discovered) in Kibana. It does create the index and let me choose 'time' OR '@timestamp' as Time-Fields but nothing is getting discovered.

If you store events correctly with proper date mapping,

fluent-plugin-elasticsearch works fine.

So the problem seems the elasticsearch side or kibana side.

Maybe, post the question on elastic forum is better.

From your configuration, I can see the data in Discovered: https://gyazo.com/c3feb0be9922f371ab2085930cf57019.png

This e-mail and any attachments are confidential and intended solely for the use of the addressee only. If you have received this message in error, you must not copy, distribute or disclose the contents; please notify the sender immediately and delete the message.

This message is attributed to the sender and may not necessarily reflect the view of Travis Perkins Trading Company Limited, 733503, Lodge Way House, Lodge Way, Harlestone Road, Northampton, NN5 7UG or its parent company Travis Perkins plc (Registered in England No. 824821; Lodge Way House, Lodge Way, Harlestone Road, Northampton, NN5 7UG; VAT number 408556737) and any of its subsidiaries. Agreements binding Travis Perkins Trading Company Limited may not be concluded by means of e-mail communication.

E-mail transmissions are not secure and Travis Perkins Trading accepts no responsibility for changes made to this message after it was sent. Whilst steps have been taken to ensure that this message is virus free, Travis Perkins Trading accepts no liability for infection and recommends that you scan this e-mail and any attachments.

[Yasin Amadmia]

" From your configuration, I can see the data in Discovered: https://gyazo.com/c3feb0be9922f371ab2085930cf57019.png" -> Can I ask if you did custom mapping for timestamp or time or both in ElasticSearch ?

On Tuesday, 29 March 2016 14:34:23 UTC+1, Cvijan Uros wrote:

[Mr. Fiber]

> Can I ask if you did custom mapping for timestamp or time or both in ElasticSearch ?

I use your mapping and fluentd configuration.

This e-mail and any attachments are confidential and intended solely for the use of the addressee only. If you have received this message in error, you must not copy, distribute or disclose the contents; please notify the sender immediately and delete the message.

This message is attributed to the sender and may not necessarily reflect the view of Travis Perkins Trading Company Limited, 733503, Lodge Way House, Lodge Way, Harlestone Road, Northampton, NN5 7UG or its parent company Travis Perkins plc (Registered in England No. 824821; Lodge Way House, Lodge Way, Harlestone Road, Northampton, NN5 7UG; VAT number 408556737) and any of its subsidiaries. Agreements binding Travis Perkins Trading Company Limited may not be concluded by means of e-mail communication.

E-mail transmissions are not secure and Travis Perkins Trading accepts no responsibility for changes made to this message after it was sent. Whilst steps have been taken to ensure that this message is virus free, Travis Perkins Trading accepts no liability for infection and recommends that you scan this e-mail and any attachments.

[Mr. Fiber]

And I changed time value from 2016-04-19 19:53:16,086000 to 2016-04-19 19:53:16,086 in the log.

[Yasin Amadmia]

For anyone who is having issues with this, I wanted to point out that there is a plugin https://github.com/shivaken/fluent-plugin-better-timestamp that does this job very easily and neatly.

On Tuesday, 29 March 2016 14:34:23 UTC+1, Cvijan Uros wrote: