Multi Destination with same log

246 views
Skip to first unread message

gun...@gmail.com

unread,
Apr 10, 2013, 5:58:25 AM4/10/13
to flu...@googlegroups.com
Hello,

Im testing Fluentd with Apache Access Log

I want to send an Apache Access Log to web_hdfs and stdout.

I mean 1 log source multi destinatioin.

However, I didnt find out how to do it because it seems like a log pattern match the first match_output_plugin in the configuration file(td-agent.conf), the log doesnt go down anymore.

is there anyway to send a log to many dst?

Thanks

Eduardo Aceituno

unread,
Apr 10, 2013, 6:02:47 AM4/10/13
to flu...@googlegroups.com

Copy output plugin. Examples in fluentd docs: http://docs.fluentd.org/articles/plugin-development

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

pe...@diverta.co.jp

unread,
May 2, 2013, 10:01:51 AM5/2/13
to flu...@googlegroups.com
Hello Mr Eduardo Aceituno

I use the copy out plugin, and send logs to stdout s3 and mongo, but I got the following message when forward the log. I think it may be a bug
of copy out put plugin. 

it seems that the parameter "time_key" is a "" string in /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/mixin.rb line 176.

How to fixed it?

2013-05-02 22:53:46 +0900 fluent.warn: {"message":"emit transaction failed ","error":"can't convert String into Integer","time":1367502826}
2013-05-02 22:53:46 +0900 [warn]: emit transaction failed  error="can't convert String into Integer" time=1367502826
  2013-05-02 22:53:46 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/mixin.rb:176:in `[]='
  2013-05-02 22:53:46 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/mixin.rb:176:in `filter_record'
  2013-05-02 22:53:46 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/mixin.rb:96:in `block in format_stream'
  2013-05-02 22:53:46 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/event.rb:127:in `feed_each'
  2013-05-02 22:53:46 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/event.rb:127:in `each'
  2013-05-02 22:53:46 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/mixin.rb:95:in `format_stream'
  2013-05-02 22:53:46 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/output.rb:224:in `emit'
  2013-05-02 22:53:46 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-mongo-0.7.0/lib/fluent/plugin/out_mongo.rb:100:in `emit'
  2013-05-02 22:53:46 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/output.rb:35:in `next'
  2013-05-02 22:53:46 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/plugin/out_copy.rb:67:in `emit'
  2013-05-02 22:53:46 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/match.rb:38:in `emit'
  2013-05-02 22:53:46 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/engine.rb:120:in `emit_stream'
  2013-05-02 22:53:46 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/plugin/in_forward.rb:118:in `on_message'
  2013-05-02 22:53:46 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/plugin/in_forward.rb:180:in `feed_each'
  2013-05-02 22:53:46 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/plugin/in_forward.rb:180:in `on_read_msgpack'
  2013-05-02 22:53:46 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/plugin/in_forward.rb:168:in `call'
  2013-05-02 22:53:46 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/plugin/in_forward.rb:168:in `on_read'
  2013-05-02 22:53:46 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.1.0/lib/cool.io/io.rb:108:in `on_readable'
  2013-05-02 22:53:46 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.1.0/lib/cool.io/io.rb:170:in `on_readable'
  2013-05-02 22:53:46 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.1.0/lib/cool.io/loop.rb:96:in `run_once'
  2013-05-02 22:53:46 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.1.0/lib/cool.io/loop.rb:96:in `run'
  2013-05-02 22:53:46 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/plugin/in_forward.rb:78:in `run'


2013年4月10日水曜日 19時02分47秒 UTC+9 Eduardo Aceituno:

Eduardo Aceituno

unread,
May 2, 2013, 10:22:14 AM5/2/13
to flu...@googlegroups.com
I think the problem is time format. Mongo plugin in emit method is warning about this:

fluent.warn: {"message":"emit transaction failed ","error":"can't convert String into Integer","time":1367502826}
[warn]: emit transaction failed  error="can't convert String into Integer" time=1367502826
...
 /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-mongo-0.7.0/lib/fluent/plugin/out_mongo.rb:100:in `emit'
...


It's possible you need to parse time properly. See example in test of mongo plugin: 


...
time = Time.parse("2011-01-02 13:14:15 UTC").to_i
...




2013/5/2 <pe...@diverta.co.jp>

pe...@diverta.co.jp

unread,
May 2, 2013, 10:31:56 AM5/2/13
to flu...@googlegroups.com
Thank you for your quickly reply

I will try replacing the UTC with JSZ to see if it works.

2013年5月2日木曜日 23時22分14秒 UTC+9 Eduardo Aceituno:

pe...@diverta.co.jp

unread,
May 4, 2013, 1:50:41 AM5/4/13
to flu...@googlegroups.com
Dear Eduardo Aceituno

I am sorry to disturb you in the golden-week holiday.I have cheeked in the source code, but I do not know how to fix it. 

My fluent sever looks like this:
                                                                                                                                             +--->S3 server
PHP -> fluentd server(in Japan) -> forward to -> fluentd server( aws server) ->copy output plugin |
                                                                                                                                             +--->MongoDB (localhost : aws server)

The time zone of this two fluent server is different, and maybe this is the cause. Can you give me some clue to fix this. 
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
The source code of "/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-mongo-0.7.0/lib/fluent/plugin/out_mongo.rb:100:in `emit' "
  def emit(tag, es, chain)
    # TODO: Should replacement using eval in configure?
    if @tag_mapped
      super(tag, es, chain, tag)
    else
      super(tag, es, chain)
    end
  end

/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/mixin.rb:176:in `[]='

def filter_record(tag, time, record)
    super
    if @include_time_key
      record[@time_key] = @timef.format(time)
    end
  end


2013年4月10日水曜日 19時02分47秒 UTC+9 Eduardo Aceituno:

Copy output plugin. Examples in fluentd docs: http://docs.fluentd.org/articles/plugin-development

Eduardo Aceituno

unread,
May 4, 2013, 5:13:31 AM5/4/13
to flu...@googlegroups.com

Can you paste configurations and log samples, please?

Eduardo Aceituno

unread,
May 4, 2013, 6:45:49 PM5/4/13
to flu...@googlegroups.com
If you don't use out_copy plugin, can you write to stdout or mongo in a single match configuration? Try to isolate what plugin is producing warning, mongo or stdout. It would help to have some log examples to try to reproduce the error/warning.


2013/5/4 Eduardo Aceituno <ach...@gmail.com>

pe...@diverta.co.jp

unread,
May 7, 2013, 3:13:22 AM5/7/13
to flu...@googlegroups.com

Thanks for your suggestion, and I am sorry for my delayed reply.

The config file and log looks like bellowing. When I commented the mongo block in the configure, I got no error or warning message.
But if open it, there will be  ["error":"can't convert String into Integer"]. 

The log message has been transfer to MongoDB although there are error messages. 

----------------------------------------------------------------------------------------configurations-------------------------------------------------------------------------------------------------------------

<source>
  type forward
  port 24224
  bind 0.0.0.0
</source>
<match **>
  type copy
  <store **>
    type stdout
  </store>
  <store **>
    type s3
    aws_key_id **********************************
    aws_sec_key **********************************
    s3_bucket **********
    path test/
    buffer_path /var/log/fluent/s3

    time_slice_format %Y%m%d%H
    time_slice_wait 10m

    buffer_chunk_limit 256m
  </store>
  <store **>
    # plugin type
    type mongo

    # mongodb db + collection
    database xxxx
    collection all_log

    # mongodb host + port
    host localhost
    port 27017

    # interval
    flush_interval 10s
  </store>

</match>

----------------------------------------------------------------------------------------td-agent log message---------------------------------------------------------------------------------------------------
2013-05-07 15:55:07 +0900 fluent.warn: {"message":"emit transaction failed ","error":"can't convert String into Integer"}
2013-05-07 15:55:07 +0900 [warn]: emit transaction failed  error="can't convert String into Integer" time=1367909707
  2013-05-07 15:55:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/mixin.rb:176:in `[]='
  2013-05-07 15:55:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/mixin.rb:176:in `filter_record'
  2013-05-07 15:55:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/mixin.rb:96:in `block in format_stream'
  2013-05-07 15:55:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/event.rb:127:in `feed_each'
  2013-05-07 15:55:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/event.rb:127:in `each'
  2013-05-07 15:55:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/mixin.rb:95:in `format_stream'
  2013-05-07 15:55:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/output.rb:224:in `emit'
  2013-05-07 15:55:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-mongo-0.7.0/lib/fluent/plugin/out_mongo.rb:100:in `emit'
  2013-05-07 15:55:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/output.rb:35:in `next'
  2013-05-07 15:55:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/buffer.rb:170:in `block in emit'
  2013-05-07 15:55:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
  2013-05-07 15:55:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/buffer.rb:166:in `emit'
  2013-05-07 15:55:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/output.rb:509:in `block in emit'
  2013-05-07 15:55:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/event.rb:127:in `feed_each'
  2013-05-07 15:55:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/event.rb:127:in `each'
  2013-05-07 15:55:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/output.rb:499:in `emit'
  2013-05-07 15:55:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/output.rb:35:in `next'
  2013-05-07 15:55:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/plugin/out_stdout.rb:30:in `emit'
  2013-05-07 15:55:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/output.rb:35:in `next'
  2013-05-07 15:55:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/plugin/out_copy.rb:67:in `emit'
  2013-05-07 15:55:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/match.rb:38:in `emit'
  2013-05-07 15:55:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/engine.rb:120:in `emit_stream'
  2013-05-07 15:55:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/plugin/in_forward.rb:118:in `on_message'
  2013-05-07 15:55:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/plugin/in_forward.rb:180:in `feed_each'
  2013-05-07 15:55:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/plugin/in_forward.rb:180:in `on_read_msgpack'
  2013-05-07 15:55:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/plugin/in_forward.rb:168:in `call'
  2013-05-07 15:55:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/plugin/in_forward.rb:168:in `on_read'
  2013-05-07 15:55:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.1.0/lib/cool.io/io.rb:108:in `on_readable'
  2013-05-07 15:55:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.1.0/lib/cool.io/io.rb:170:in `on_readable'
  2013-05-07 15:55:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.1.0/lib/cool.io/loop.rb:96:in `run_once'
  2013-05-07 15:55:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.1.0/lib/cool.io/loop.rb:96:in `run'
  2013-05-07 15:55:07 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/plugin/in_forward.rb:78:in `run'
2013-05-07 15:55:07 +0900 fluent.error: {"message":"forward error: can't convert String into Integer"}
2013-05-07 15:55:07 +0900 [error]: forward error: can't convert String into Integer time=1367909707
  2013-05-07 15:55:07 +0900 [error]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/mixin.rb:176:in `[]='
  2013-05-07 15:55:07 +0900 [error]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/mixin.rb:176:in `filter_record'
  2013-05-07 15:55:07 +0900 [error]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/mixin.rb:96:in `block in format_stream'
  2013-05-07 15:55:07 +0900 [error]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/event.rb:127:in `feed_each'
  2013-05-07 15:55:07 +0900 [error]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/event.rb:127:in `each'
  2013-05-07 15:55:07 +0900 [error]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/mixin.rb:95:in `format_stream'
  2013-05-07 15:55:07 +0900 [error]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/output.rb:224:in `emit'
  2013-05-07 15:55:07 +0900 [error]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-mongo-0.7.0/lib/fluent/plugin/out_mongo.rb:100:in `emit'
  2013-05-07 15:55:07 +0900 [error]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/output.rb:35:in `next'
  2013-05-07 15:55:07 +0900 [error]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/buffer.rb:170:in `block in emit'
  2013-05-07 15:55:07 +0900 [error]: /usr/lib64/fluent/ruby/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
  2013-05-07 15:55:07 +0900 [error]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/buffer.rb:166:in `emit'
  2013-05-07 15:55:07 +0900 [error]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/output.rb:509:in `block in emit'
  2013-05-07 15:55:07 +0900 [error]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/event.rb:127:in `feed_each'
  2013-05-07 15:55:07 +0900 [error]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/event.rb:127:in `each'
  2013-05-07 15:55:07 +0900 [error]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/output.rb:499:in `emit'
  2013-05-07 15:55:07 +0900 [error]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/output.rb:35:in `next'
  2013-05-07 15:55:07 +0900 [error]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/plugin/out_stdout.rb:30:in `emit'
  2013-05-07 15:55:07 +0900 [error]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/output.rb:35:in `next'
  2013-05-07 15:55:07 +0900 [error]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/plugin/out_copy.rb:67:in `emit'
  2013-05-07 15:55:07 +0900 [error]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/match.rb:38:in `emit'
  2013-05-07 15:55:07 +0900 [error]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/engine.rb:120:in `emit_stream'
  2013-05-07 15:55:07 +0900 [error]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/plugin/in_forward.rb:118:in `on_message'
  2013-05-07 15:55:07 +0900 [error]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/plugin/in_forward.rb:180:in `feed_each'
  2013-05-07 15:55:07 +0900 [error]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/plugin/in_forward.rb:180:in `on_read_msgpack'
  2013-05-07 15:55:07 +0900 [error]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/plugin/in_forward.rb:168:in `call'
  2013-05-07 15:55:07 +0900 [error]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/plugin/in_forward.rb:168:in `on_read'
  2013-05-07 15:55:07 +0900 [error]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.1.0/lib/cool.io/io.rb:108:in `on_readable'
  2013-05-07 15:55:07 +0900 [error]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.1.0/lib/cool.io/io.rb:170:in `on_readable'
  2013-05-07 15:55:07 +0900 [error]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.1.0/lib/cool.io/loop.rb:96:in `run_once'
  2013-05-07 15:55:07 +0900 [error]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.1.0/lib/cool.io/loop.rb:96:in `run'
  2013-05-07 15:55:07 +0900 [error]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/plugin/in_forward.rb:78:in `run'

2013年5月5日日曜日 7時45分49秒 UTC+9 Eduardo Aceituno:

pe...@diverta.co.jp

unread,
May 7, 2013, 4:42:42 AM5/7/13
to flu...@googlegroups.com
The further test shows that, when s3 output plugin and mongo output plugin configured in the outcopy block together, there will be error. It seems the two plugin process the time format in a different way.


2013年5月5日日曜日 7時45分49秒 UTC+9 Eduardo Aceituno:
If you don't use out_copy plugin, can you write to stdout or mongo in a single match configuration? Try to isolate what plugin is producing warning, mongo or stdout. It would help to have some log examples to try to reproduce the error/warning.

Eduardo Aceituno

unread,
May 7, 2013, 4:49:56 AM5/7/13
to flu...@googlegroups.com
I think that ** in store element is not needed although this is not the error cause . By the way, could you try to exclude the "time key" in mongo store configuration for testing purposes?

<store>
    # plugin type
    type mongo

    # mongodb db + collection
    database xxxx
    collection all_log

    # mongodb host + port
    host localhost
    port 27017

    # interval
    flush_interval 10s
   
   include_time_key false
  </store>


2013/5/7 <pe...@diverta.co.jp>

xiaohui peng

unread,
May 7, 2013, 5:30:40 AM5/7/13
to flu...@googlegroups.com
If include_time_key false  is used, it seems OK. No warning and no error, but I can not get the time information of the log in my mongoDB.




2013/5/7 Eduardo Aceituno <ach...@gmail.com>

--
You received this message because you are subscribed to a topic in the Google Groups "Fluentd Google Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/fluentd/IuL68fVy3AA/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to fluentd+u...@googlegroups.com.

Eduardo Aceituno

unread,
May 7, 2013, 5:45:14 AM5/7/13
to flu...@googlegroups.com
Ok I know it. Could you send/gist/paste the original message logs produced by PHP application (I think it is the producer) that you want to log? Please, the input log to the fluent system.


2013/5/7 xiaohui peng <pe...@diverta.co.jp>

pe...@diverta.co.jp

unread,
May 7, 2013, 5:53:21 AM5/7/13
to flu...@googlegroups.com
Actually the PHP application does not  continuously produce log, so I test with http logs (execute it several times):
curl -X POST -d 'json={"action":"login","user":201305071840}' http://localhost:8888/test.tag.here

The red characters are date and time string and the log in MongoDB:
{ "_id" : ObjectId("5188cc19bbd51b4df8000009"), "action" : "login", "user" : NumberLong("201305071840") }
{ "_id" : ObjectId("5188cc19bbd51b4df8000008"), "action" : "login", "user" : NumberLong("201305071840") }
{ "_id" : ObjectId("5188cc19bbd51b4df8000007"), "action" : "login", "user" : NumberLong("201305071840") }
{ "_id" : ObjectId("5188cc19bbd51b4df8000006"), "action" : "login", "user" : NumberLong("201305071840") }
{ "_id" : ObjectId("5188cc19bbd51b4df8000005"), "action" : "login", "user" : NumberLong("201305071840") }
{ "_id" : ObjectId("5188cc19bbd51b4df8000004"), "action" : "login", "user" : NumberLong("201305071840") }
{ "_id" : ObjectId("5188cc19bbd51b4df8000003"), "action" : "login", "user" : NumberLong("201305071840") }
{ "_id" : ObjectId("5188cc19bbd51b4df8000002"), "action" : "login", "user" : NumberLong("201305071840") }
{ "_id" : ObjectId("5188cc19bbd51b4df8000001"), "action" : "login", "user" : NumberLong("201305071840") }
{ "_id" : ObjectId("5188c93dbbd51b4db7000005"), "action" : "login", "user" : NumberLong("201305071828") }
{ "_id" : ObjectId("5188c932bbd51b4db7000004"), "action" : "login", "user" : NumberLong("201305071828") }
{ "_id" : ObjectId("5188c932bbd51b4db7000003"), "action" : "login", "user" : NumberLong("201305071828") }
{ "_id" : ObjectId("5188c932bbd51b4db7000002"), "action" : "login", "user" : NumberLong("201305071828") }
{ "_id" : ObjectId("5188c932bbd51b4db7000001"), "action" : "login", "user" : NumberLong("201305071828") }

2013年5月7日火曜日 18時45分14秒 UTC+9 Eduardo Aceituno:

pe...@diverta.co.jp

unread,
May 7, 2013, 6:00:36 AM5/7/13
to flu...@googlegroups.com
Wait a minute, I got error message about 10 minutes after the following command and the transfer to MongoDB is also failed.

curl -X POST -d 'json={"action":"login","user":201305071840}' http://localhost:8888/test.tag.here

-----------------------------------------------------------error message---------------------------------------------------------------------------------------
2013-05-07 18:51:08 +0900 [warn]: temporarily failed to flush the buffer, next retry will be at 2013-05-07 18:51:09 +0900. error="undefined method `has_key?' for #<Array:0x007f35776d1b78>" instance=69933860183560
  2013-05-07 18:51:08 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/bson-1.8.4/lib/bson/types/object_id.rb:75:in `create_pk'
  2013-05-07 18:51:08 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/mongo-1.8.4/lib/mongo/collection.rb:373:in `block in insert'
  2013-05-07 18:51:08 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/mongo-1.8.4/lib/mongo/collection.rb:373:in `collect!'
  2013-05-07 18:51:08 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/mongo-1.8.4/lib/mongo/collection.rb:373:in `insert'
  2013-05-07 18:51:08 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-mongo-0.7.0/lib/fluent/plugin/out_mongo.rb:117:in `operate'
  2013-05-07 18:51:08 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-mongo-0.7.0/lib/fluent/plugin/out_mongo.rb:107:in `write'
  2013-05-07 18:51:08 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/buffer.rb:290:in `write_chunk'
  2013-05-07 18:51:08 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/buffer.rb:274:in `pop'
  2013-05-07 18:51:08 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/output.rb:294:in `try_flush'
  2013-05-07 18:51:08 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/output.rb:121:in `run'
2013-05-07 18:51:10 +0900 [warn]: temporarily failed to flush the buffer, next retry will be at 2013-05-07 18:51:10 +0900. error="undefined method `has_key?' for #<Array:0x007f3578afe628>" instance=69933860183560
  2013-05-07 18:51:10 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/bson-1.8.4/lib/bson/types/object_id.rb:75:in `create_pk'
  2013-05-07 18:51:10 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/mongo-1.8.4/lib/mongo/collection.rb:373:in `block in insert'
  2013-05-07 18:51:10 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/mongo-1.8.4/lib/mongo/collection.rb:373:in `collect!'
  2013-05-07 18:51:10 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/mongo-1.8.4/lib/mongo/collection.rb:373:in `insert'
  2013-05-07 18:51:10 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-mongo-0.7.0/lib/fluent/plugin/out_mongo.rb:117:in `operate'
  2013-05-07 18:51:10 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-mongo-0.7.0/lib/fluent/plugin/out_mongo.rb:107:in `write'
  2013-05-07 18:51:10 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/buffer.rb:290:in `write_chunk'
  2013-05-07 18:51:10 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/buffer.rb:274:in `pop'
  2013-05-07 18:51:10 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/output.rb:294:in `try_flush'
  2013-05-07 18:51:10 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/output.rb:121:in `run'
2013-05-07 18:51:11 +0900 [warn]: temporarily failed to flush the buffer, next retry will be at 2013-05-07 18:51:15 +0900. error="undefined method `has_key?' for #<Array:0x007f3578b111b0>" instance=69933860183560
  2013-05-07 18:51:11 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/bson-1.8.4/lib/bson/types/object_id.rb:75:in `create_pk'
  2013-05-07 18:51:11 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/mongo-1.8.4/lib/mongo/collection.rb:373:in `block in insert'
  2013-05-07 18:51:11 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/mongo-1.8.4/lib/mongo/collection.rb:373:in `collect!'
  2013-05-07 18:51:11 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/mongo-1.8.4/lib/mongo/collection.rb:373:in `insert'
  2013-05-07 18:51:11 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-mongo-0.7.0/lib/fluent/plugin/out_mongo.rb:117:in `operate'
  2013-05-07 18:51:11 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-mongo-0.7.0/lib/fluent/plugin/out_mongo.rb:107:in `write'
  2013-05-07 18:51:11 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/buffer.rb:290:in `write_chunk'
  2013-05-07 18:51:11 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/buffer.rb:274:in `pop'
  2013-05-07 18:51:11 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/output.rb:294:in `try_flush'
  2013-05-07 18:51:11 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/output.rb:121:in `run'
2013-05-07 18:51:16 +0900 [warn]: temporarily failed to flush the buffer, next retry will be at 2013-05-07 18:51:22 +0900. error="undefined method `has_key?' for #<Array:0x007f3578b17f38>" instance=69933860183560
  2013-05-07 18:51:16 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/bson-1.8.4/lib/bson/types/object_id.rb:75:in `create_pk'
  2013-05-07 18:51:16 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/mongo-1.8.4/lib/mongo/collection.rb:373:in `block in insert'
  2013-05-07 18:51:16 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/mongo-1.8.4/lib/mongo/collection.rb:373:in `collect!'
  2013-05-07 18:51:16 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/mongo-1.8.4/lib/mongo/collection.rb:373:in `insert'
  2013-05-07 18:51:16 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-mongo-0.7.0/lib/fluent/plugin/out_mongo.rb:117:in `operate'
  2013-05-07 18:51:16 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-mongo-0.7.0/lib/fluent/plugin/out_mongo.rb:107:in `write'
  2013-05-07 18:51:16 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/buffer.rb:290:in `write_chunk'
  2013-05-07 18:51:16 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/buffer.rb:274:in `pop'
  2013-05-07 18:51:16 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/output.rb:294:in `try_flush'
  2013-05-07 18:51:16 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/output.rb:121:in `run'
2013-05-07 18:51:23 +0900 [warn]: temporarily failed to flush the buffer, next retry will be at 2013-05-07 18:51:37 +0900. error="undefined method `has_key?' for #<Array:0x007f35792e6b88>" instance=69933860183560
  2013-05-07 18:51:23 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/bson-1.8.4/lib/bson/types/object_id.rb:75:in `create_pk'
  2013-05-07 18:51:23 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/mongo-1.8.4/lib/mongo/collection.rb:373:in `block in insert'
  2013-05-07 18:51:23 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/mongo-1.8.4/lib/mongo/collection.rb:373:in `collect!'
  2013-05-07 18:51:23 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/mongo-1.8.4/lib/mongo/collection.rb:373:in `insert'
  2013-05-07 18:51:23 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-mongo-0.7.0/lib/fluent/plugin/out_mongo.rb:117:in `operate'
  2013-05-07 18:51:23 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-mongo-0.7.0/lib/fluent/plugin/out_mongo.rb:107:in `write'
  2013-05-07 18:51:23 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/buffer.rb:290:in `write_chunk'
  2013-05-07 18:51:23 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/buffer.rb:274:in `pop'
  2013-05-07 18:51:23 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/output.rb:294:in `try_flush'
  2013-05-07 18:51:23 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/output.rb:121:in `run'
2013-05-07 18:51:38 +0900 [warn]: temporarily failed to flush the buffer, next retry will be at 2013-05-07 18:52:12 +0900. error="undefined method `has_key?' for #<Array:0x007f3578b10990>" instance=69933860183560
  2013-05-07 18:51:38 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/bson-1.8.4/lib/bson/types/object_id.rb:75:in `create_pk'
  2013-05-07 18:51:38 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/mongo-1.8.4/lib/mongo/collection.rb:373:in `block in insert'
  2013-05-07 18:51:38 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/mongo-1.8.4/lib/mongo/collection.rb:373:in `collect!'
  2013-05-07 18:51:38 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/mongo-1.8.4/lib/mongo/collection.rb:373:in `insert'
  2013-05-07 18:51:38 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-mongo-0.7.0/lib/fluent/plugin/out_mongo.rb:117:in `operate'
  2013-05-07 18:51:38 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-mongo-0.7.0/lib/fluent/plugin/out_mongo.rb:107:in `write'
  2013-05-07 18:51:38 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/buffer.rb:290:in `write_chunk'
  2013-05-07 18:51:38 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/buffer.rb:274:in `pop'
  2013-05-07 18:51:38 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/output.rb:294:in `try_flush'
  2013-05-07 18:51:38 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/output.rb:121:in `run'
2013-05-07 18:52:13 +0900 [warn]: temporarily failed to flush the buffer, next retry will be at 2013-05-07 18:53:08 +0900. error="undefined method `has_key?' for #<Array:0x007f35774ceec0>" instance=69933860183560
  2013-05-07 18:52:13 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/bson-1.8.4/lib/bson/types/object_id.rb:75:in `create_pk'
  2013-05-07 18:52:13 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/mongo-1.8.4/lib/mongo/collection.rb:373:in `block in insert'
  2013-05-07 18:52:13 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/mongo-1.8.4/lib/mongo/collection.rb:373:in `collect!'
  2013-05-07 18:52:13 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/mongo-1.8.4/lib/mongo/collection.rb:373:in `insert'
  2013-05-07 18:52:13 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-mongo-0.7.0/lib/fluent/plugin/out_mongo.rb:117:in `operate'
  2013-05-07 18:52:13 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-mongo-0.7.0/lib/fluent/plugin/out_mongo.rb:107:in `write'
  2013-05-07 18:52:13 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/buffer.rb:290:in `write_chunk'
  2013-05-07 18:52:13 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/buffer.rb:274:in `pop'
  2013-05-07 18:52:13 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/output.rb:294:in `try_flush'
  2013-05-07 18:52:13 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/output.rb:121:in `run'
2013-05-07 18:53:09 +0900 [warn]: temporarily failed to flush the buffer, next retry will be at 2013-05-07 18:55:10 +0900. error="undefined method `has_key?' for #<Array:0x007f3578a2ee28>" instance=69933860183560
  2013-05-07 18:53:09 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/bson-1.8.4/lib/bson/types/object_id.rb:75:in `create_pk'
  2013-05-07 18:53:09 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/mongo-1.8.4/lib/mongo/collection.rb:373:in `block in insert'
  2013-05-07 18:53:09 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/mongo-1.8.4/lib/mongo/collection.rb:373:in `collect!'
  2013-05-07 18:53:09 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/mongo-1.8.4/lib/mongo/collection.rb:373:in `insert'
  2013-05-07 18:53:09 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-mongo-0.7.0/lib/fluent/plugin/out_mongo.rb:117:in `operate'
  2013-05-07 18:53:09 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-mongo-0.7.0/lib/fluent/plugin/out_mongo.rb:107:in `write'
  2013-05-07 18:53:09 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/buffer.rb:290:in `write_chunk'
  2013-05-07 18:53:09 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/buffer.rb:274:in `pop'
  2013-05-07 18:53:09 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/output.rb:294:in `try_flush'
  2013-05-07 18:53:09 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/output.rb:121:in `run'
2013-05-07 18:55:11 +0900 [warn]: temporarily failed to flush the buffer, next retry will be at 2013-05-07 18:59:36 +0900. error="undefined method `has_key?' for #<Array:0x007f35789a5218>" instance=69933860183560
  2013-05-07 18:55:11 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/bson-1.8.4/lib/bson/types/object_id.rb:75:in `create_pk'
  2013-05-07 18:55:11 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/mongo-1.8.4/lib/mongo/collection.rb:373:in `block in insert'
  2013-05-07 18:55:11 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/mongo-1.8.4/lib/mongo/collection.rb:373:in `collect!'
  2013-05-07 18:55:11 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/mongo-1.8.4/lib/mongo/collection.rb:373:in `insert'
  2013-05-07 18:55:11 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-mongo-0.7.0/lib/fluent/plugin/out_mongo.rb:117:in `operate'
  2013-05-07 18:55:11 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-mongo-0.7.0/lib/fluent/plugin/out_mongo.rb:107:in `write'
  2013-05-07 18:55:11 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/buffer.rb:290:in `write_chunk'
  2013-05-07 18:55:11 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/buffer.rb:274:in `pop'
  2013-05-07 18:55:11 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/output.rb:294:in `try_flush'
  2013-05-07 18:55:11 +0900 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.33/lib/fluent/output.rb:121:in `run'

2013年5月7日火曜日 18時45分14秒 UTC+9 Eduardo Aceituno:
Ok I know it. Could you send/gist/paste the original message logs produced by PHP application (I think it is the producer) that you want to log? Please, the input log to the fluent system.

pe...@diverta.co.jp

unread,
May 7, 2013, 6:30:44 AM5/7/13
to flu...@googlegroups.com
Is there any parameter to set the time formate, Mongo output plugin use ["time" : ISODate("2013-05-07T09:17:57Z")] as default.

I  guess that UTC is used for time zone, but I want change it to JSZ. 
Message has been deleted

Eduardo Aceituno

unread,
May 10, 2013, 4:00:11 AM5/10/13
to flu...@googlegroups.com
(Sorry for my english, I'm not native)

Perhaps Masahiro (author of the plugin) can shed further light on the issue. From my knowledge, I think the time is treated internally as UNIX time. Time attribute is not parsed with timezone value, and records are inserted in a collection with mongo operations of ruby driver (mongo gem). The only values I have seen in fluent code, relative to time expressions, are "utc" and "localtime".

Additional information in mongodb documentation:
"BSON Date is a 64-bit integer that represents the number of milliseconds since the Unix epoch. The official BSON specification refers to the BSON Date type as the UTC datetime"



2013/5/7 <pe...@diverta.co.jp>
I think I fixed it, just put localtime in the configuration block of Mongo output plugin.  There is no error message until now.

but log in MongoDB :

{ "_id" : ObjectId("5188e352bbd51b4f00000018"), "action" : "login", "user" : NumberLong("201305072019"), "time" : ISODate("2013-05-07T11:19:39Z") }

how can I change the ISODate to local Date format?


2013年5月7日火曜日 18時45分14秒 UTC+9 Eduardo Aceituno:
Ok I know it. Could you send/gist/paste the original message logs produced by PHP application (I think it is the producer) that you want to log? Please, the input log to the fluent system.

pe...@diverta.co.jp

unread,
May 12, 2013, 12:55:02 PM5/12/13
to flu...@googlegroups.com
Eduardo Aceituno

Thank you very much. I am sorry for replying so late, it is just a little busy these days.

[I think I fixed it, just put localtime in the configuration block of Mongo output plugin.  There is no error message until now.]
 but I found the problem remain after 10 minutes that day.

Never mind, it just woks, the log message does transmit to MongoDB, although there are some warning or error messages.

Thank you again for your kindly help.


2013年5月10日金曜日 17時00分11秒 UTC+9 Eduardo Aceituno:
Reply all
Reply to author
Forward
0 new messages