Modify output json based on input json

[Frederico Ferreira]

Hello everyone,

this is my case: i'm using FluentD to get messages from Kafka. My message is a json that look like this:

{
    "az": "value1",
    "slave_host": "value2",
    "message": "#JSON# {...}"
}

FluentD must get the message inside message field. I need to get rid of '#JSON#" (can't change it before it arrives to Kafka) to make it a valid json, but i also need the information that az and slave_host fields has. This is my conf file::

<source kafka>
  @type                   kafka_group
  brokers                 ============
  zookeepers              ===========
  consumer_group          fluentd_elasticsearch
  topics                  fluentd_marathon_app
  format                  json

  max_bytes               1048576
  max_wait_ms             10
  min_bytes               1
  socket_timeout_ms       1000
</source>

<filter>
  @type record_transformer
  enable_ruby
  <record>
    message ${message.gsub('#JSON# ', '')}
  </record>
</filter>

<filter>
  @type parser
  key_name message
  format json
</filter>

#<filter>
# @type record_transformer
#  enable_ruby
#  <record>
#    environment atlas
#    az "AZ"
#    slave_host "${slave_host}"
#  </record>
#</filter>

<match>
  @type elasticsearch
  host                  ============================
  port                  =============================
  flush_interval        1s
  logstash_format       true
  logstash_prefix       "performance-kafka"
  logstash_dateformat   "%Y.%m.%d.%H"
  time_key              date
  utc_index             true
</match>

The part that is commented is not working and i don't know if i did something wrong or else. Could someone land me a hand on this?

[Mr. Fiber]

Sorry, I don't understand the problem correctly.

What is the problem?

If you want to keep original attributes,

set "reserve_data yes" in parser filter configuration.

See README document: https://github.com/tagomoris/fluent-plugin-parser

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Frederico Ferreira]

Thanks, Mr. Fiber. That did the trick!

Atenciosamente,

Frederico Ferreira

Tecnologia

frederico...@b2wdigital.com

Tel.: +55 (21) 3504-7167 Ramal: 57167

Rádio: +55 (21) 96473-8614 Id: 12*1025777

Skype: frederico.86

2016-08-03 16:48 GMT-03:00 Mr. Fiber <repea...@gmail.com>:

Sorry, I don't understand the problem correctly.

What is the problem?

If you want to keep original attributes,

set "reserve_data yes" in parser filter configuration.

See README document: https://github.com/tagomoris/fluent-plugin-parser

Masahiro

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

 

---

Esta mensagem pode conter informações confidenciais e somente o indivíduo ou entidade a quem foi destinada pode utilizá-la. A transmissão incorreta da mensagem não acarreta a perda de sua confidencialidade. Caso esta mensagem tenha sido recebida por engano, solicitamos que o fato seja comunicado ao remetente e que a mensagem seja eliminada de seu sistema imediatamente. É vedado a qualquer pessoa que não seja o destinatário usar, revelar, distribuir ou copiar qualquer parte desta mensagem. Ambiente de comunicação sujeito a monitoramento.

This message may include confidential information and only the intended addresses have the right to use it as is, or any part of it. A wrong transmission does not break its confidentiality. If you've received it because of a mistake or erroneous transmission, please notify the sender and delete it from your system immediately. This communication environment is controlled and monitored.

B2W Digital