Good recommendation on how to stream log data into Amazon Elastic Search?

[Marco Pas]

Does anyone have a good stable recommendation to stream log data into Amazon Elastic Search? We are using the aws-elasticsearch-service plugin but we keep running into issues. Like required automatic restarts for the aggregator. (https://github.com/atomita/fluent-plugin-aws-elasticsearch-service)

[David Wood]

Have a look through https://groups.google.com/d/topic/fluentd/6y4AvuP7c4U/discussion

[Marco Pas]

Hi David,

do you have an example of a configuration for plugin configuration for AWS Elasticsearch that seems to work? We now need to reboot the aggregators very day to keep them going and this is causing problems.

--
You received this message because you are subscribed to a topic in the Google Groups "Fluentd Google Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/fluentd/uW87VAOqxeE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[David Wood]

Hi,

I don't use the plugin https://github.com/atomita/fluent-plugin-aws-elasticsearch-service.  It seems dead: https://github.com/atomita/fluent-plugin-aws-elasticsearch-service/issues/29.

I instead use https://github.com/uken/fluent-plugin-elasticsearch.  It doesn't support Amazon signing so you'll need an IP access policy.  You'd need to ensure you have these settings:

resurrect_after 5s

reload_connections false

buffer_chunk_limit 8m

The buffer_chunk_limit could be larger depending on the instance size you select.  The link I provided earlier has more information.

David

[David Wood]

https://github.com/uken/fluent-plugin-elasticsearch/issues/261

[Marco Pas]

Hi David,

when using these settings do you still need te restart the Fluented container now and then? We ran into the issues that a restart of Fluent is needed on a scheduled basis. When we did not do this the inflow would stop into ES.

If an IP access policy is used is there still a requirement to do the signing?

- Marco

[David Wood]

when using these settings do you still need te restart the Fluented container now and then?

No - it runs for months without any issue. 

 

If an IP access policy is used is there still a requirement to do the signing?

No - no need for both.

David

[Marco Pas]

I am for sure going to try this out. If i point this plugin to AWS ES what are the settings for the host / port?

[David Wood]

The port is 443.  The host is whatever is listed as the "endpoint" in the AWS console (search-XXXXXXXXX.es.amazonaws.com).

David

[Marco Pas]

Would you be able to share you config when using this plugin and pointing it to AWS ES?

I am trying to implement the same and an example how your succesful config is would be great!

@type elasticsearch

logstash_format true

flush_interval 60s

host <points to??>

port <is the port needed ??>

index_name fluentd

type_name fluentd

resurrect_after 5s

reload_connections false

buffer_chunk_limit 8m

On Tuesday, June 6, 2017 at 2:23:57 PM UTC+2, David Wood wrote:

[David Wood]

Sure:

      @type elasticsearch

      host XXXXX

      port 443

      scheme https

      request_timeout 30s

      resurrect_after 5

      # avoid https://discuss.elastic.co/t/elasitcsearch-ruby-raises-cannot-get

      # -new-connection-from-pool-error/36252/6

      reload_connections false

      logstash_format true

      logstash_prefix syslog

      # @timestamp: use event time, not time of indexing

      time_key time

      include_tag_key true

      # https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/

      # aes-limits.html

      buffer_chunk_limit 8m

      buffer_queue_limit 2048    # 8m * 2048 = 16GB

[David Wood]

 How'd you make out?  David

[Marco Pas]

Hi David, 

apologies for the delay. We have just integrated it into our infrastructure and until now it seems to work great. No more hick ups and data is nicely flowing into AWS ES. Thanks a lot for your help!!!!

On Wed, Jun 21, 2017 at 8:20 PM David Wood <ya...@yankeefrog.com> wrote:

 How'd you make out?  David