How to send log from td-agent agregator to elasticsearch with SearchGuard as Sercurity

[rd85]

Hi, Im Rudi,

Is it possible to forwarding log from td-agent agregator to elasticsearch with SearchGuard as Sercurity, How to config it. I was success trying to forwarding log from node to EFK server using td-agent without searchguard as security, and the log is indexed on Kibana.

i have the following environment :

(td-agent forwarder on node)

OS : windows 10

Installed Plugin : fluent-plugin-elasticsearch-2.10.1, fluent-plugin-secure-forward-0.4.5

(td-agent aggregator on EFK Server)

OS : RHEL7

Installed Plugin : fluent-plugin-elasticsearch-2.10.1, fluent-plugin-secure-forward-0.4.5

[Mr. Fiber]

I'm not familiar with SearchGuard.

fluent-plugin-elasticsearch depends on official elasticsearch ruby client.

If elasticsearch ruby client has a option for SearchGuard, elasticsearch plugin should support it.

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[rudi atur]

Hi Masahiro,

I have configure the td-agent.conf aggregator on EFK server. I am using existing searchguard user and password,  but the log is not sent yet to ES.

this is my td-agent.conf :

<source>

   @type forward

   port 24224

</source>

<match eaea>

   @type copy

   <store>

     @type elasticsearch

     host localhost

     port 9200

     scheme https

     ssl_verify false

     user logstash

     password logstash

     index_name fluentd

     logstash_format true

     logstash_prefix fluentd

     include_timestamp true

     #time_format %Y-%m-%dT%H:%M:%S

     #timezone +0700

     <buffer>

        flush_interval 1s

     </buffer>

   </store>

</match>

Rudi,

[rudi atur]

Solved, thx masahiro

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.

[Venkata Narayana Yarava]

Hi, I have some issue. when I am trying to configure the efk. td-agent is running but logs are not forwarding to aggregator.

<match td.*.*>
  @type tdlog
  apikey YOUR_API_KEY
  auto_create_table
  buffer_type file
  buffer_path /var/log/td-agent/buffer/td

  <secondary>
    @type file
    path /var/log/td-agent/failed_records
  </secondary>
</match>

## match tag=debug.** and dump to console
<match debug.**>
  @type stdout
</match>

## built-in TCP input
## @see http://docs.fluentd.org/articles/in_forward

<source>
  @type forward
  port 24224
</source>

<source>
  @type http
  port 8888
</source>

## live debugging agent

<source>
  @type debug_agent
  bind 127.0.0.1
  port 24230
</source>

<source>
  @type tail
  path /var/log/myapp.log
  pos_file /var/log/td-agent/myorg.log.pos
  tag myorg.myapp
  format /^(?<level>[^ ]*)[ \t]+\[(?<time>[^\]]*)\] \[(?<thread>[^\]]*)\] \[(?<request>[^\]]*)\] (?<class>[^ ]*): (?<message>.*)$/

  time_format %Y-%m-%d %H:%M:%S,%L %z
  timezone +0530
  time_key time
  keep_time_key true
  types time:time
</source>

<match myorg.**>
   @type copy
   <store>
    @type file
    path /var/log/td-agent/forward.log
  </store>


  <store>
    @type forward
   heartbeat_type tcp

    #aggregator IP

    host 192.168.0.26
   <buffer>
     flush_interval 30s
   </buffer>
 </store>

</match>

Kindly let me know when i missed?