Fluentd sending to Logstash leads to lot of garbage data

[Zeal Vora]

Hi

We have fluentd which is sending logs from client server to our main ELK Server.

In the Kibana, when i read the logs, there is lot of garbage data that comes along with the message.

This is the message that we get from Kibana

\x92\xACsys.messages\xDB\u0000\u0000\u0003\u001A\x92\xCEU\xC1\x9FY\x84\xA4host\xAFip\xA5ident\xA9freshclam\xA3pid\xA48721\xA7message\xDA\u00009ClamAV update process started at Wed Aug 5 11:00:01 2015\x92\xCEU\xC1\x9FY\x84\xA4host\xAFip-10-20-12-209\xA5ident\xA9freshclam\xA3pid\xA48721\xA7message\xDA\u0000Nmain.cvd is up to date (version: 55, sigs: 2424225, f-level: 60, builder: neo)\x92\xCEU\xC1\x9FZ\x84\xA4host\xAFip-

The message comes but it's like encoded between lot of garbage data.

Our Configuration of Logstash :-
input {
     syslog {
     host => "0.0.0.0"
     port => 5141
}
}

output {
       stdout { }
        elasticsearch {
}
}

If we use rsyslog instead of fluentd, the logs comes in proper manner.

Any help will be appreciated.

[Mr. Fiber]

It seems logstash configuration problem.

AKAIK, fluentd doesn't have out_syslog like plugin so

fluentd can't use syslog protocol for sending logs to other systems.

I'm not sure why do you setup logstash instead of using fluent-plugin-elasticsearch.

https://github.com/uken/fluent-plugin-elasticsearch

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Zeal Vora]

Thanks. We tried fluentd codec in logstash but we were facing similar issue , so for trial and error, we tried syslog ( as incoming messages are syslog based ) , but it gives the same problem.

Whenever we use rsyslog to send logs, the messages are properly parsed, But when we use fluentd to send logs, then lot of grumbled up data comes.

--
You received this message because you are subscribed to a topic in the Google Groups "Fluentd Google Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/fluentd/jZcekZ-VB3w/unsubscribe.
To unsubscribe from this group and all its topics, send an email to fluentd+u...@googlegroups.com.

[Mr. Fiber]

What the plugin do you use in fluentd configuration? out_forward?

If so, logstash receives msgpack encoded chunks.

I'm not sure how logstash's fluentd codec handles this format.

[Zeal Vora]

We are using this. Am i missing something ?

<source>

  type config_expander

  <config>

    type tail

    format none

    path /var/log/secure

    pos_file /var/log/td-agent/tmp/secure.log.pos

    tag sys.secure

  </config>

</source>

[Mr. Fiber]

This is input plugin, <source>.

You use output plugin, <match>, to send logs to logstash.

[Zeal Vora]

This is the output plugin which sends to logs to Logstash :-

<source>

  type config_expander

  <config>

    type tail

    format none

    path /var/log/secure

    pos_file /var/log/td-agent/tmp/secure.log.pos

    tag sys.secure

  </config>

</source>

<match sys.*>

  type forest

  subtype copy

  <template>

    <store>

      type s3

      s3_region ap-southeast-1

      s3_bucket logs.security.audittrail

      path sys/${hostname}/${tag}/%Y/%m/%d/

      buffer_path /var/log/td-agent/buffer/${tag}/

      time_slice_format %Y-%m-%d_%H

      flush_interval 30m

      buffer_chunk_limit 100m

    </store>

    <store>

      type forward

      heartbeat_type tcp

      <server>

        host 192.168.24.50

        port 5141

      </server>

    </store>

  </template>

</match>

[Mr. Fiber]

I see.

I checked logstash document and fluent codec doesn't support out_forward format.

https://www.elastic.co/guide/en/logstash/current/plugins-codecs-fluent.html

So you can't use tcp and fluent codec for out_forward.

There are several approaches.

- Use fluentd on both forwarder and aggregator

- Support out_forward format in fluent codec

- Use another plugin combination in fluentd and logstash

Masahiro

[Kiyoto Tamura]

Zeal,

One option is to use Fluentd on the receiver too and use out_elasticsearch. This approach is proven to scale at many companies.

Unfortunately, the Fluentd core team does not maintain Logstash's Fluentd codec. You should report to the authors of Logstash's Fluentd codec library.

What are the reasons for running Fluentd on edge but Logstash for the aggregator?

Kiyoto

[Zeal Vora]

Thanks.

One of my colleague, helped me out.

I had to include : logstash_format true  for writing data into elasticsearch compatible to what logstash writes