td-agent and fluent-plugin-amqp2 problem

[luisnrkl]

Hi,

I'm using td-agent to send some messages to rabbitmq and i'm having some troubles when the rabbit queue begins to get large.

It's somewhat random, but sooner or later i get the following error on td-agent and it stops sending the messages until i restart it:

2015-10-27 18:02:15 -0200 [error]: unexpected error error_class=Bunny::NetworkFailure error=#<Bunny::NetworkFailure: detected a network failure: IO timeout when writing to socket>
  2015-10-27 18:02:15 -0200 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/loop.rb:88:in `run_once'

....

2015-10-27 18:02:15 -0200 [info]: shutting down fluentd
2015-10-27 18:02:19 -0200 [warn]: unexpected error while shutting down output plugins plugin=Fluent::AmqpOutput plugin_id="object:3f90812914e4" error_class=Bunny::NetworkErrorWrapper error=#<Bunny::NetworkErrorWrapper: Connection-level error: CHANNEL_ERROR - expected 'channel.open'>
  2015-10-27 18:02:19 -0200 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/bunny-2.2.0/lib/bunny/session.rb:627:in `handle_network_failure'
  2015-10-27 18:02:19 -0200 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/bunny-2.2.0/lib/bunny/session.rb:560:in `handle_frame'
...

On rabbitmq log:

=ERROR REPORT==== 27-Oct-2015::18:02:16 ===
Error on AMQP connection <0.3612.0> (192.168.48.93:47768 -> 192.168.49.29:5672, vhost: '/', user: 'guest', state: running), channel 1:
{amqp_error,channel_error,"expected 'channel.open'",'basic.publish'}

I only have this error if i use fluent-plugin-amqp2. With fluent-plugin-amqp this doesn't happen.

Does anyone know what might be causing this?

My config on td-agent:

 <match eventlogger.**>
    type copy
    <store>
      type amqp
      host RABBITMQ_HOST
      port 5672
      user guest
      password xxxxxx
      vhost /
      exchange events_exchange
      exchange_type topic
      exchange_durable true
      payload_only false
      content_type application/octet-stream
      flush_interval 10s
      buffer_type file
      buffer_path /opt/app/fluentd/buffer/rabbitmq_events/${tag}.buffer
      buffer_chunk_limit 5m
      buffer_queue_limit 1000000
    </store>
    <store>
      type file
      path /logs/fluentd/fluent
      time_slice_format %Y%m%d
      time_slice_wait 10m
      time_format %Y%m%dT%H%M%S%z
      compress gzip
      utc
      buffer_path /logs/fluentd/fluent.*
    </store>
  </match>

Thanks,

[Mr. Fiber]

They use same client called bunny but passing options are different.

Maybe, it causes a different behaviour

amqp: https://github.com/restorando/fluent-plugin-amqp/blob/master/lib/fluent/plugin/out_amqp.rb

amqp2: https://github.com/giraffi/fluent-plugin-amqp/blob/master/lib/fluent/plugin/out_amqp.rb

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[luisnrkl]

Thanks,

I found that the problem only ocurrs if the rabbitMq version is superior to v3.4.3

So we will just use this version.

Regards,