Amazon Kinesis Input

[Daryl Robbins]

Is there any way to use Amazon Kinesis as an input to fluentd? My plan is to use fluentd on each node as a log shipper to Kinesis and then I'd like to run the log aggregator using the Kinesis client. Unfortunately, it's written in Java, so it doesn't seem obvious on how/if the two can be connected. Thanks very much!

[Masahiro Nakagawa]

HI Daryl,

Is your system like below?

app -> fluentd -> kinesis queue -> kinesis client -> fluentd -> other destinations?

If so, you can use fluent-logger-java to send events to fluentd aggregator.

https://github.com/fluent/fluent-logger-java

or if you want to consume events from Fluentd directly,

requesting in_kinesis to fluent-plugin-kinesis is better.

https://github.com/awslabs/aws-fluent-plugin-kinesis

Thanks,

Masahiro

On Wed, Oct 8, 2014 at 10:36 PM, Daryl Robbins <darylr...@gmail.com> wrote:

Is there any way to use Amazon Kinesis as an input to fluentd? My plan is to use fluentd on each node as a log shipper to Kinesis and then I'd like to run the log aggregator using the Kinesis client. Unfortunately, it's written in Java, so it doesn't seem obvious on how/if the two can be connected. Thanks very much!

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Daryl Robbins]

Thanks for the quick response!

Yes, that's exactly the pipeline that I had in mind. The Java logger is a good idea, but I'm guessing it's asynchronous, which isn't quite ideal since I'd lose the guaranteed delivery offered by Kinesis. There would be the potential for message loss between when the Kinesis client checkpointed and when fluentd was able to successfully deliver to an output.

So, unless you see another transactional option, I guess I'll have to pass along my enhancement request to AWS. Thanks!

[Masahiro Nakagawa]

> I'm guessing it's asynchronous

https://github.com/fluent/fluent-logger-java/blob/master/src/main/java/org/fluentd/logger/sender/RawSocketSender.java

I'm not a Java logger maintainer but the implementation seems to use synchronous approach.

I also want Kinesis and Fluentd pair information...

[Daryl Robbins]

Sorry, I wasn't entirely clear. When I said that I'm guessing it's asynchronous, I was referring to the complete transaction from receive in Kinesis to successful processing in fluentd.

The connection is buffered, so the log message returning doesn't necessarily mean it has been transmitted yet. And even if you called flush after each log, you would still get no feedback on when the 'transaction committed' so to speak on the fluentd side, which is fine for application logging but not for building a reliable data stream.

I know the Kinesis output plugin was built by the AWS Kinesis team, so I spoke to our AWS Solution Architect to pass along this suggestion directly to them. I'll let you know if I hear anything back. They have been working hard to make Kinesis play nicely with more and more systems lately, so here's to hoping (the fluentd plugin is quite recent, and they released an ElasticSearch connector recently too).

Amazon reINVENT is coming up mid-November, so there are bound to be plenty of announcements about new capabilities. Hopefully, that includes some good news about Kinesis and Fluentd as well. 

[Tsuyoshi OZAWA]

Hi Daryl,

Yeah, fluend-logger-java buffers all data currently. We can add synchronous mode for your use case. 

Let's discuss on issue board on github of fluent-logger-java. Your pull request is also welcome :-)

https://github.com/fluent/fluent-logger-java/issues

Thanks,

- Tsuyoshi