RabbitMQ plugin with tls enable error

[Manuel Eveno]

Hi,

I'm setting up AMQP/TLS on my rabbitmq server but trying to connect fluentd to rabbitmq using TLS is not working.

Using fluentd docker image v1.6.2-1.0
with fluentd-rabbitmq-plugin: v0.0.7 (with bunny v2.14.3)
And rabbitmq docker image rabbitmq:3.7.15-management

fluentd-rabbit-vdev_1  | 2019-10-03 12:16:01 +0000 [info]: adding source type="rabbitmq"
fluentd-rabbit-vdev_1  | 2019-10-03 12:16:01 +0000 [warn]: #0 Using TLS but no client certificate is provided. If RabbitMQ is configured to require & verify peer
fluentd-rabbit-vdev_1  | certificate, connection will be rejected. Learn more at https://www.rabbitmq.com/ssl.html
fluentd-rabbit-vdev_1  |
fluentd-rabbit-vdev_1  | 2019-10-03 12:16:01 +0000 [info]: #0 starting fluentd worker pid=34 ppid=6 worker=0
fluentd-rabbit-vdev_1  | 2019-10-03 12:16:01 +0000 [error]: #0 unexpected error error_class=NoMethodError error="undefined method `add' for #<Fluent::Log:0x0000564ea68845f0>"
fluentd-rabbit-vdev_1  |   2019-10-03 12:16:01 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/bunny-2.14.3/lib/bunny/transport.rb:374:in `log_peer_certificate_info'
fluentd-rabbit-vdev_1  |   2019-10-03 12:16:01 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/bunny-2.14.3/lib/bunny/transport.rb:93:in `connect'
fluentd-rabbit-vdev_1  |   2019-10-03 12:16:01 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/bunny-2.14.3/lib/bunny/session.rb:315:in `start'
fluentd-rabbit-vdev_1  |   2019-10-03 12:16:01 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluent-plugin-rabbitmq-0.0.7/lib/fluent/plugin/in_rabbitmq.rb:113:in `start'
fluentd-rabbit-vdev_1  |   2019-10-03 12:16:01 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.6.2/lib/fluent/root_agent.rb:203:in `block in start'
fluentd-rabbit-vdev_1  |   2019-10-03 12:16:01 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.6.2/lib/fluent/root_agent.rb:192:in `block (2 levels) in lifecycle'
fluentd-rabbit-vdev_1  |   2019-10-03 12:16:01 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.6.2/lib/fluent/root_agent.rb:191:in `each'
fluentd-rabbit-vdev_1  |   2019-10-03 12:16:01 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.6.2/lib/fluent/root_agent.rb:191:in `block in lifecycle'
fluentd-rabbit-vdev_1  |   2019-10-03 12:16:01 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.6.2/lib/fluent/root_agent.rb:178:in `each'
fluentd-rabbit-vdev_1  |   2019-10-03 12:16:01 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.6.2/lib/fluent/root_agent.rb:178:in `lifecycle'
fluentd-rabbit-vdev_1  |   2019-10-03 12:16:01 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.6.2/lib/fluent/root_agent.rb:202:in `start'
fluentd-rabbit-vdev_1  |   2019-10-03 12:16:01 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.6.2/lib/fluent/engine.rb:274:in `start'
fluentd-rabbit-vdev_1  |   2019-10-03 12:16:01 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.6.2/lib/fluent/engine.rb:219:in `run'
fluentd-rabbit-vdev_1  |   2019-10-03 12:16:01 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.6.2/lib/fluent/supervisor.rb:808:in `run_engine'
fluentd-rabbit-vdev_1  |   2019-10-03 12:16:01 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.6.2/lib/fluent/supervisor.rb:551:in `block in run_worker'
fluentd-rabbit-vdev_1  |   2019-10-03 12:16:01 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.6.2/lib/fluent/supervisor.rb:733:in `main_process'
fluentd-rabbit-vdev_1  |   2019-10-03 12:16:01 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.6.2/lib/fluent/supervisor.rb:546:in `run_worker'
fluentd-rabbit-vdev_1  |   2019-10-03 12:16:01 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.6.2/lib/fluent/command/fluentd.rb:30:in `<top (required)>'
fluentd-rabbit-vdev_1  |   2019-10-03 12:16:01 +0000 [error]: #0 /usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
fluentd-rabbit-vdev_1  |   2019-10-03 12:16:01 +0000 [error]: #0 /usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
fluentd-rabbit-vdev_1  |   2019-10-03 12:16:01 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.6.2/bin/fluentd:8:in `<top (require d)>'
fluentd-rabbit-vdev_1  |   2019-10-03 12:16:01 +0000 [error]: #0 /usr/bin/fluentd:23:in `load'
fluentd-rabbit-vdev_1  |   2019-10-03 12:16:01 +0000 [error]: #0 /usr/bin/fluentd:23:in `<main>'
fluentd-rabbit-vdev_1  | 2019-10-03 12:16:01 +0000 [error]: #0 unexpected error error_class=NoMethodError error="undefined method `add' for #<Fluent::Log:0x0000564ea68845f0>"
fluentd-rabbit-vdev_1  |   2019-10-03 12:16:01 +0000 [error]: #0 suppressed same stacktrace
fluentd-rabbit-vdev_1  | 2019-10-03 12:16:02 +0000 [info]: Worker 0 finished unexpectedly with status 1

Here is my configuration file :

  @type rabbitmq
  tag msg.business
  host "#{ENV['RABBITMQ_HOST']}"
  port "#{ENV['RABBITMQ_PORT']}"
  user "#{ENV['RABBITMQ_USER']}"
  pass "#{ENV['RABBITMQ_PWD']}"
  vhost /
  tls "#{ENV['RABBITMQ_ENABLE_TLS']}"
  verify_peer false
  exchange Business # not required. if specified, the queue will be bound to the exchange
  exchange_type topic # required: type of exchange e.g. topic, direct
  exchange_durable true
  create_exchange true
  queue business-fluentd-queue
  include_headers true
  routing_key "#" # if not specified, the tag is used
  heartbeat 10 # integer as seconds or :server (interval specified by server)
  <parse>
    @type json # or msgpack, ltsv, none
  </parse>
  <buffer> # to use in buffered mode
  </buffer>
</source>

<filter msg.**>
  @type record_transformer
  <record>
    contexte "#{ENV['CONTEXTE']}"
    eventType "${tag_parts[1]}"
    env "#{ENV['ENV_NAME']}"
  </record>
</filter>

<match msg.**>
  @type elasticsearch
  host elasticsearch
  include_tag_key true
  logstash_format true
  logstash_prefix "#{ENV['ENV_NAME']}-rabbitmq"
  flush_interval 10s # for testing
</match>

Any clue ?

[Michael Klishin]

Bunny's logger is not set up [1]. It is taken from the connection object [2] which Bunny

will initialize by default [3]. I don't know how the Fluentd client you use sets up logging

but extremely likely that it passes a nil for :logger.

1. https://github.com/ruby-amqp/bunny/blob/master/lib/bunny/transport.rb#L374

2. https://github.com/ruby-amqp/bunny/blob/master/lib/bunny/transport.rb#L43

3. https://github.com/ruby-amqp/bunny/blob/master/lib/bunny/session.rb#L1397

[Manuel Eveno]

I've looked at bunny code (I'm not a ruby pro).

So the logger provided in session logger is not compatible with the one expected by bunny ?

Am I wrong ?

Which logger and version is expecting bunny ?

[Michael Klishin]

No logger is provided at all. Any logger that implements that standard logger
interface would do.

> fluentd-rabbit-vdev_1 | 2019-10-03 12:16:01 +0000 [error]: #0 /usr/lib/ruby/gems/2.5.0/gems/

> --
> Bunny: http://rubybunny.info
> March Hare: http://rubymarchhare.info
>
> IRC: #rabbitmq on irc.freenode.net
>
> Post to the group: ruby...@googlegroups.com | unsubscribe: ruby-amqp+...@googlegroups.com
> ---
> You received this message because you are subscribed to the Google Groups "Ruby RabbitMQ libraries" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ruby-amqp+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/ruby-amqp/4d5ac2e4-df63-4941-afa9-707c1fa0ae42%40googlegroups.com.