td-agent 2.1.4

141 views
Skip to first unread message

repeatedly

unread,
Feb 9, 2015, 10:50:29 PM2/9/15
to flu...@googlegroups.com
Hi all,

We released td-agent v2.1.4.


- fluentd v0.10.60
- fluent-plugin-s3 v0.5.3
- fluent-plugin-mongo v0.7.5
- fluent-plugin-td v0.10.25
- fluentd-ui v0.3.14
- msgpack 0.5.11

This version uses fluentd v0.10.60 and msgpack 0.5.11.
If you saw incompatible encoding error in previous td-agent 2,
please upgrade to v2.1.4. See more details in v0.10.60 release note:


We have a plan to release td-agent 2.2.0 with fluentd v0.12 soon.
Please wait :)

Enjoy fluentd!


Thanks,
Masahiro

Lance N.

unread,
Feb 10, 2015, 6:36:45 PM2/10/15
to flu...@googlegroups.com
Cool!

Have you decided when you will upgrade to Ruby 2.2? My aggregator is regularly at 100% CPU and it is probably caused by garbage collection.

Lance

Kazuki Ohta

unread,
Feb 11, 2015, 3:02:10 AM2/11/15
to flu...@googlegroups.com
Lance,

What output plugins are you using for aggregators? S3?

Also in_multiprocess (http://docs.fluentd.org/articles/in_multiprocess) plugin might be interesting for you.



==
Kazuki Ohta | Founder and CTO, Treasure Data
Follow us:    Watch our latest intro video

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Lance N.

unread,
Feb 12, 2015, 5:52:46 PM2/12/15
to flu...@googlegroups.com
Hi-

Yes, I'm using multiprocessor. I just now have a finished design. The 'pen' tcp socket load-balancer is on the instance (amazon) with fluentd. The clients talk the pen, which talks to 4 fluentd processes run by in_multiprocessor.

These four processes only do intake and statistics. They send metrics to another set of fluentd processes which save gzipped files to S3. 

If Fluentd ran in JRuby this could all be one process that used multiple CPUs :)

Lance N.

unread,
Feb 12, 2015, 5:54:20 PM2/12/15
to flu...@googlegroups.com
Originally I was saving to HDFS but found the WebHDFS output plugin would lose the connection and then be unable to re-open it. In a previous job we used several storage systems: S3, HDFS, local disk, Cassandra, Redis, Mysql. S3 was the most reliable. 

Kazuki Ohta

unread,
Feb 14, 2015, 1:18:04 AM2/14/15
to flu...@googlegroups.com
Lance,

From fluent-plugin-s3 v0.5.4 which is included in td-agent2 v2.1.4, it has a nice feature: Use external gzip command for compression. Please just add these lines.

> store_as gzip_command
> num_threads 8

By combining num_threads parameter, you can offload the CPU usage to the external CPU and execute gzip commands in parallel. Hope this helps your deployment...




==
Kazuki Ohta | Founder and CTO, Treasure Data
Follow us:    Watch our latest intro video

Reply all
Reply to author
Forward
0 new messages