Remote server syslogs reaching fluentd but not seen in elasticsearch

[cody.ro...@gmail.com]

Hi,

I'm new to fluentd and trying to get a new EFK stack setup. I am able to forward local syslogs to elastic search/kibana quite easily but unable to get logs from a remote Debian server. So far I can get the logs into td-agent log but I can't seem to get them seen in Kibana no matter what I try. Below is the td-agent.conf for forwarding local logs that works:

<source>
@type syslog
port 3356
tag syslog
</source>

<source>
@type forward
</source>

<match syslog.**>
 @type elasticsearch
 logstash_format true
 flush_interval 10s # for testing
</match>

 

and below is the recipe I am trying for pushing remote server logs to elasticsearch. I can see the remote logs in the td-agent logfile just fine but don't see them in elasticsearch. Could someone advise what I'm doing wrong?

<source>
 @type syslog
 port 3356
 protocol_type tcp
 tag remote_syslog
</source>

<source>
@type forward
</source>

<match remote_syslog.*.*>
 type copy
 <store>
    type stdout
 </store>
 <store>
    type elasticsearch
    logstash_format true
    flush_interval 10s
    host localhost
    port 9200
 </store>
</match>

Many Thanks
Cody

[Mr. Fiber]

You add "protocol_type tcp" in your syslog configuration.

Is this correct? First example doesn't set this parameter.

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[cody.ro...@gmail.com]

Hi Masahiro,

Thanks for your response. I added in that parameter because without it logs do not get forwarded from the remote host at all, only from the localhost. I got the suggestion from the below thread. The logs do come in to the td-agent.log, I just need to be able to feed them into elasticsearch.

http://serverfault.com/questions/619082/forward-linux-logs-to-fluentd-on-linux

Thanks

Daniel

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.

[Mr. Fiber]

> I just need to be able to feed them into elasticsearch.

You need to check kibana doesn't show logs or elasticsearch doesn't have logs.

Former, the problem is kibana side.

Latter, the problem is fluentd side.

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.