Fluentd Performance Numbers with input plugin http & output plugin s3

[shefal...@teamaol.com]

Hello All,

We are trying out Fluentd for one of our projects. This is our 

<source>
 @type http
 port 9999
 log_level warn
 keepalive_timeout 40s
</source>

<match pathname>
 type copy
 log_level warn
 <store>
  @type s3
  s3_bucket 
  path pathname
  use_server_side_encryption AES256
  buffer_type file
  buffer_path /tmp/s3/pathname
  flush_interval 300s
  flush_at_shutdown true
  time_slice_format %Y/%m/%d/%H/pathname_%H%M
  time_format %H%M
  time_slice_wait 10m
  s3_object_key_format %{path}%{time_slice}-%{hostname}-%{hex_random}_%{index}.%{file_extension}
  buffer_chunk_limit 64m
  buffer_queue_limit 4096
  num_threads 8
  format json
  store_as json
 </store>
</match>

</source>

I am using AWS and for Performance Testing I am using single instance m4.2x large behind ELB . But the max number of message I can only send are 3000 message/sec . Each message is 2KB. 

Is this the upper limit ?? As per documentation we can get at least 14000 messages/sec so I was wondering whether I am missing anything in configuration?

Can anyone help?

[Mr. Fiber]

The performance depends on plugins.

I'm not sure maximum in_http performance but it seems slower than in_tail or in_forward like popular case by HTTP overhead.

Do you use batch mode? > http://docs.fluentd.org/v0.12/articles/in_http#batch-mode

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Shefal...@teamaol.com]

Thanks!! Batching did help. I was able to get to 5000 messages / sec.  Is their a way we can also use compression with batching with in_http plugin ?

Regards,

Shefali.

[Mr. Fiber]

We added simple benchmark to in_http article: http://docs.fluentd.org/v0.12/articles/in_http#batch-mode

>  Is their a way we can also use compression with batching with in_http plugin ?

Currently, no compression support in in_http.

Maybe, we can support gzip compression but gain is less with msgpack.

Masahiro

--

[Shefal...@teamaol.com]

Yes Thanks for the numbers . 

Currently we cannot do batching  for our project so we are trying out http://docs.fluentd.org/v0.12/articles/in_multiprocess to see whether we can bump up the performance.

We need to set up in an autoscaling group depending on the number of messages per second . We have the datadog monitoring in place Is their a way to add number of message per second metrics with in_http plugin ?

Regards,

Shefali.

On Wednesday, March 15, 2017 at 2:19:14 PM UTC-4, shefal...@teamaol.com wrote: