Timestamp processing issue

[George Piskorsky]

Hi,

I have the following pipeline:

SaaS ->HTTP POST JSON array -> Fluentd -> ElasticSearch

Every incomming message in array has field "timestamp" in unix epoch time format. Example:

{"email":"lkmlkm...@gmail.com","timestamp":1496051232,"ip":"66.102.9.31","sg_event_id":"NjAyZTFjODktNDYzYi00MGU2LWE4YzItYWQxOWRkMzU0NzBk","sg_message_id":"MHAD6Q66TbS2PhQCkdpEgw.filter0069p1las1-8821-59240799-33.0","category":"sdfefvsdc","useragent":"Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko Firefox/11.0 (via ggpht.com GoogleImageProxy)","event":"open"}

I use this configuration for processing:

<source>
  @type http
  bind 176.W.Z.X
  port 8887
  format json
</source>
<filter send.grid>
  @type split_array
</filter>
<match sk.**>
  @type copy
  <store>
    @type file
    path /var/log/sglog/sk/current/sk
    time_slice_format %Y%m%d%H
    time_slice_wait 1m
    compress gzip
    utc
  </store>
  <store>
    @type elasticsearch


    logstash_format true
    logstash_prefix sg-sk
    logstash_dateformat %Y-%m
    time_key timestamp
    time_key_format %s
   # time_parse_error_tag es_plugin.output.time.error


    id_key sg_event_id


    host localhost
    port 9200


    buffer_type memory
    buffer_chunk_limit 64M
    buffer_queue_limit 32
    flush_interval 10s
    retry_limit 17
    retry_wait 1.0
    num_threads 1
  </store>
</match>

But have errors in td-agent.log

2017-05-29 12:41:59 +0300 [warn]: dump an error event: error_class=TypeError error="no implicit conversion of Fixnum into String" tag="Fluent::ElasticsearchOutput::TimeParser.error" time=1496050919 record={"tag"=>"sk", "time"=>1496050913, "format"=>"%s", "value"=>1496050879}
2017-05-29 12:43:29 +0300 [warn]: dump an error event: error_class=TypeError error="no implicit conversion of Fixnum into String" tag="Fluent::ElasticsearchOutput::TimeParser.error" time=1496051009 record={"tag"=>"sk", "time"=>1496051004, "format"=>"%s", "value"=>1496050972}
2017-05-29 12:43:29 +0300 [warn]: dump an error event: error_class=TypeError error="no implicit conversion of Fixnum into String" tag="Fluent::ElasticsearchOutput::TimeParser.error" time=1496051009 record={"tag"=>"sk", "time"=>1496051004, "format"=>"%s", "value"=>1496050973}
2017-05-29 12:43:29 +0300 [warn]: dump an error event: error_class=TypeError error="no implicit conversion of Fixnum into String" tag="Fluent::ElasticsearchOutput::TimeParser.error" time=1496051009 record={"tag"=>"sk", "time"=>1496051005, "format"=>"%s", "value"=>1496050972}
2017-05-29 12:43:29 +0300 [warn]: dump an error event: error_class=TypeError error="no implicit conversion of Fixnum into String" tag="Fluent::ElasticsearchOutput::TimeParser.error" time=1496051009 record={"tag"=>"sk", "time"=>1496051005, "format"=>"%s", "value"=>1496050972}

It seems that timestamp has correct value and format, but could you please point me what is wrong in my configuration? How to set timestamp from incoming JSON field in epoch format, but not HTTP recieve timestamp as without time_key/time_key_format options?

[Mr. Fiber]

The log is "warn", so it seems not critical logs.

I checked the code and the problem is parse_time method doesn't consider non-string timestamp.

https://github.com/uken/fluent-plugin-elasticsearch/blob/e0460a1d0e27ee0d03b15032b60fb8331d0ec034/lib/fluent/plugin/out_elasticsearch.rb#L136

This is why fluentd log has these warnings.

Masahiro

On Mon, May 29, 2017 at 7:08 PM, George Piskorsky <pig...@gmail.com> wrote:

Hi,

I have the following pipeline:

SaaS ->HTTP POST JSON array -> Fluentd -> ElasticSearch

Every incomming message in array has field "timestamp" in unix epoch time format. Example:

{"email":"lkmlkmlkmlkm@gmail.com","timestamp":1496051232,"ip":"66.102.9.31","sg_event_id":"NjAyZTFjODktNDYzYi00MGU2LWE4YzItYWQxOWRkMzU0NzBk","sg_message_id":"MHAD6Q66TbS2PhQCkdpEgw.filter0069p1las1-8821-59240799-33.0","category":"sdfefvsdc","useragent":"Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko Firefox/11.0 (via ggpht.com GoogleImageProxy)","event":"open"}

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[George Piskorsky]

But it is may be critical.I double checked right now - incoming messages are written into file, but does not exists in elasticsearch when options

    time_key timestamp
    time_key_format %s

are set and warn message apears in log. Without this options message writes into both places, but without needed timestamp in ES. May be I need to get debug log anywhere else?


понедельник, 29 мая 2017 г., 13:54:11 UTC+3 пользователь repeatedly написал:

The log is "warn", so it seems not critical logs.

I checked the code and the problem is parse_time method doesn't consider non-string timestamp.

https://github.com/uken/fluent-plugin-elasticsearch/blob/e0460a1d0e27ee0d03b15032b60fb8331d0ec034/lib/fluent/plugin/out_elasticsearch.rb#L136

This is why fluentd log has these warnings.

Masahiro

On Mon, May 29, 2017 at 7:08 PM, George Piskorsky <pig...@gmail.com> wrote:

Hi,

I have the following pipeline:

SaaS ->HTTP POST JSON array -> Fluentd -> ElasticSearch

Every incomming message in array has field "timestamp" in unix epoch time format. Example:

{"email":"lkmlkm...@gmail.com","timestamp":1496051232,"ip":"66.102.9.31","sg_event_id":"NjAyZTFjODktNDYzYi00MGU2LWE4YzItYWQxOWRkMzU0NzBk","sg_message_id":"MHAD6Q66TbS2PhQCkdpEgw.filter0069p1las1-8821-59240799-33.0","category":"sdfefvsdc","useragent":"Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko Firefox/11.0 (via ggpht.com GoogleImageProxy)","event":"open"}

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.

[Mr. Fiber]

> May be I need to get debug log anywhere else?

I'm not sure. With your config, logs are inserted into ES in my env.

- Post to fluentd

% curl -v -H "Accept: application/json" -H "Content-type: application/json" -X POST -d '{"email":"lkmlkm...@gmail.com","timestamp":1496051232,"ip":"66.102.9.31","sg_event_id":"NjAyZTFjODktNDYzYi00MGU2LWE4YzItYWQxOWRkMzU0NzBk","sg_message_id":"MHAD6Q66TbS2PhQCkdpEgw.filter0069p1las1-8821-59240799-33.0","category":"sdfefvsdc","useragent":"Mozilla/5.0","event":"open"}' http://localhost:8888/sk.test;


- fluentd has same warning

2017-06-02 11:48:02 +0900 [warn]: #0 dump an error event: error_class=TypeError error="no implicit conversion of Integer into String" tag="Fluent::ElasticsearchOutput::TimeParser.error" time=#<Fluent::EventTime:0x007fed09123178 @sec=1496371682, @nsec=560070000> record={"tag"=>"sk.test", "time"=>1496371671, "format"=>"%s", "value"=>1496051232}
2017-06-02 11:48:02 +0900 [info]: #0 Connection opened to Elasticsearch cluster => {:host=>"localhost", :port=>9200, :scheme=>"http"}

- ES has event

%  curl -XPOST 'localhost:9200/_search' -d '{"query":{"match_all":{}}}'

{"took":27,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":1,"max_score":1.0,"hits":[{"_index":"sg-sk-2017-06","_type":"fluentd","_id":"NjAyZTFjODktNDYzYi00MGU2LWE4YzItYWQxOWRkMzU0NzBk","_score":1.0,"_source":{"email":"lkmlkm...@gmail.com","timestamp":1496051232,"ip":"66.102.9.31","sg_event_id":"NjAyZTFjODktNDYzYi00MGU2LWE4YzItYWQxOWRkMzU0NzBk","sg_message_id":"MHAD6Q66TbS2PhQCkdpEgw.filter0069p1las1-8821-59240799-33.0","category":"sdfefvsdc","useragent":"Mozilla/5.0","event":"open","@timestamp":1496051232}}]}}

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.