Warp 10 on EC2 instances with data from Kinesis Data Stream

[Adam Makhlouf]

Hi everyone,

I am considering using Warp 10 (standalone) in the AWS ecosystem on 2 EC2 instances (with datalog replication).

I am getting sensor data from Kinesis Data Stream that I plan on ingesting in Warp 10. I have found no reference on how that would work.

I can see that Amazon Kinesis Data Firehose can now stream data to an HTTP endpoint. Would that be a correct approach?

Thanks for your help!

[Adam Makhlouf]

I have continued my research and unfortunately it appears Kinesis Data Firehose will not do the work as it is not process data in real time. It processes data every 60s which is not enough for my use case. 

Still looking for a solution, if you have an idea please share! :)

[mathias....@gmail.com]

Warp 10 has plugins for consuming data from an MQTT or Kafka message broker, if such protocol is not an option for the data flowing through Kinesis then maybe you can create a Kinesis plugin using the Java SDK (https://docs.aws.amazon.com/streams/latest/dev/developing-consumers-with-sdk.html), it would probably be rather simple to do following the same logic present in the MQTT or Kafka plugins.

Kafka Plugin: https://github.com/senx/warp10-plugin-kafka

MQTT Plugin: https://github.com/senx/warp10-plugin-mqtt

[Adam Makhlouf]

Thank you Mathias for this. 

While looking around, I've stumbled upon Telegraf (an agent for collecting, processing, aggregating, and writing metrics) that has a Kinesis input plugin and a Warp 10 outplut plugin. If you have any comment on this solution, I'll happily hear it!

Thanks again!

Adam

[mathias....@gmail.com]

I do not know how Telegraf handles the consumption of messages from Kinesis and notably how it would decode the payloads, but I guess if it can be configured correctly then it would work.