Fluentd amqp output format issue

[Anamika Pandey]

Hi,

I am using Fluentd to read log files and route messages to RabbitMQ also I am modifying the event using Fluentd "fluent-plugin-record-modifier" plugin , my configuration looks like:

<source>
  type tail_ex
  format none
  path /home/fluentd/log/*.log
  pos_file /var/log/td-agent/famqp.pos
  tag  input_data
  read_all true
  refresh_interval 5
</source>

<match input_data>
  type record_modifier
  tag  froute
  gen_host ${hostname}
</match>

## RMQ output
<match froute>
  type amqp
  # Set broker host and port
  host 192.168.x.x
  port 5672
  # Set user and password for authentication
  user guest
  password guest
  vhost /
  exchange FLUENTD_EXCHANGE
  exchange_type topic
  exchange_durable true
  payload_only true
  content_type application/octet-stream # optional - default is application/octet-stream. some amqp consumers will expect application/json.
  buffer_type file
  buffer_path /var/log/td-agent/buffer/mylogs.buffer
</match>


* The input data looks like:

{"eS":"BSE","eN":"Bombay Stock Exchange","oV":20000.0,"cV":20017.199802517272,"hV":20017.199802517272,"lV":20000.0,"t":1395222534077}


* After routing to RabbitMQ, the modified output on RabbitMQ queue looks like:

{"message":"{\"eS\":\"BSE\",\"eN\":\"Bombay Stock Exchange\",\"oV\":20000.0,\"cV\":20085.84198064541,\"hV\":20085.84198064541,\"lV\":20000.0,\"t\":1395227305424}","gen_host":"myhost"}


1. Why the event is appended with key "message"?
NOTE: I followed link http://docs.fluentd.org/articles/filter-modify-apache its not as its described there.
 
2. Why backslashes are present in the output message ("\")?

[Kiyoto Tamura]

>1. Why the event is appended with key "message"?

This is because you are using "format none" in tail_ex. Looking at your data, you probably want "format json"

>2. Why backslashes are present in the output message ("\")?

That's just standard escaping of double quotes inside a string. Should become a non-issue once you change your "format"parameter.

Finally, looking at your configuration, it looks like you are reading the entire content of the logs in 
 /home/fluentd/log/*.log every 5 seconds. Unless you know that the data in /home/fluentd/log/*.log are refreshed inside of every 5 seconds, you will have duplicate data going through Fluentd. Is this really what you intend to do?

Kiyoto

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Check out Fluentd, the open source data collector for high-volume data streams

[Anamika Pandey]

Hi Kiyoto,

Thanks changing format to "json" worked for me.

Yes you are right I am getting duplicate data. I believe, I don't need to use refresh in this use case as the data is not streaming.