UTF-8 Support

80 views
Skip to first unread message

Kirk Bloomquist

unread,
Apr 17, 2015, 6:31:07 PM4/17/15
to flu...@googlegroups.com
Hello, we are doing a POC with fluentd to aggregate our JSON events, some of these events come from external entities and the content contains UTF-8 characters. These events are buffering to a file and then to S3 using the S3 plugin. 

I noticed in the buffer files that some UTF characters are not encoded correctly. Is there a way to configure UTF-8 encoding, or is ASCII the only supported encoding? I've yet to find much about this scouring the web. Everything was working awesome and we were extremely happy with fluentd up to this point. I'm not afraid to get into the plugins and change code if necessary, just need to know where to go if that's the route we need to take.

Thanks in advance for any help!

Satoshi Tagomori

unread,
Apr 17, 2015, 6:43:35 PM4/17/15
to flu...@googlegroups.com
Hi,

At first, never mind. Fluentd supports UTF-8 characters :)

How do you input your data into Fluentd? And how do you output your data into file and S3?
Could you show me your configuration?

tagomoris.


--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--
田籠 聡 (TAGOMORI Satoshi)
Mail: tago...@gmail.com

Kirk Bloomquist

unread,
Apr 17, 2015, 6:52:36 PM4/17/15
to flu...@googlegroups.com
Hey thanks for the quick reply, that's good to know.

We are using the HTTP input plugin and the encodings are set correctly in our HTTP POST, but I will double check that. The config is very simple and is attached. We are using the forest plugin for templating, hopefully that isn't causing troubles.
td-agent.conf

Kirk Bloomquist

unread,
Apr 17, 2015, 6:53:33 PM4/17/15
to flu...@googlegroups.com
Our additional config is at the very end of the file....

Kirk Bloomquist

unread,
Apr 18, 2015, 5:05:18 PM4/18/15
to flu...@googlegroups.com
Nevermind, our fault, we figured out the problem. We thought the input encoding was set correctly but it appears it wasn't. Once that was changed it's working like a charm.

--
You received this message because you are subscribed to a topic in the Google Groups "Fluentd Google Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/fluentd/1T4Baa5nvaQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to fluentd+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
Kirk Bloomquist

Satoshi Tagomori

unread,
Apr 19, 2015, 9:37:58 PM4/19/15
to flu...@googlegroups.com
Good to hear.
Enjoy Fluentd!

tagomoris.
Reply all
Reply to author
Forward
0 new messages