Apache Storm spout implemented with NATS?

218 views
Skip to first unread message

Tim Molter

unread,
Aug 24, 2015, 10:13:03 AM8/24/15
to nats
Hi All,

Do you know of any open source Apache Storm spouts using NATS?

I would like to create a spout that pulls from a queue and was drawn to NATS' low latency. I've come across example code that pulls messages from a Kafka queue into a Storm spout. Do you know of any example code out there using NATS for this? Do you think it would be advantageous as far as latency and throughput?

Cheers, Tim

Derek Collison

unread,
Aug 24, 2015, 2:42:09 PM8/24/15
to nat...@googlegroups.com
I think it would be straightforward to have a NATS queue subscriber (or regular subscriber) push to the Storm spout..


--
You received this message because you are subscribed to the Google Groups "nats" group.
To unsubscribe from this group and stop receiving emails from it, send an email to natsio+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Larry McQueary

unread,
Aug 24, 2015, 5:49:41 PM8/24/15
to nat...@googlegroups.com
Greetings Tim, 

As Derek mentions, it should be pretty trivial to create a NATS subscriber that will push to Storm. Can you point out which example code you looked at showing pull of messages from Kafka into a Storm spout? It's not important, but depending on the simplicity of the example, I might be able to show the analogous NATS integration (using java_nats).

-Larry

Tim Molter

unread,
Aug 25, 2015, 6:06:42 AM8/25/15
to nats
Hello Larry,

Here are two Apache spout Kafka implementations I was looking at:



Here's another one. The nice thing about this one is the integration with Zookeeper as well so that the queue servers' IP addresses don't need to be hardcoded and passed to the spout. 

I'd be interested to hear what your opinion is on the idea of a NATS Storm spout and if there'd be any throughput/latency advantages over Kafka. I haven't tried a Kafka setup at all yet.

~Tim

Matthew Ring

unread,
Oct 25, 2015, 10:30:08 AM10/25/15
to nats
I have a NATS source for Apache Flink. It's probably quite easy to turn it into a Storm spout.


-Matt

Larry McQueary

unread,
Oct 25, 2015, 10:31:59 AM10/25/15
to nat...@googlegroups.com
Cool -- thanks Matt! We will have to compile a list of integrations for the NATS.io website at some point.
--
You received this message because you are subscribed to the Google Groups "nats" group.
To unsubscribe from this group and stop receiving emails from it, send an email to natsio+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
Larry McQueary | Director, Messaging Technology

Matthew Ring

unread,
Oct 25, 2015, 5:33:21 PM10/25/15
to nats
Hi, Larry and others,

I also has similar NATS messaging in/out processors for Apache NiFi: https://github.com/mring33621/nats-messaging-for-nifi

-Matt

Larry McQueary

unread,
Oct 25, 2015, 6:42:56 PM10/25/15
to nat...@googlegroups.com
Matt -- that's awesome! I don't know if we have any left, but just in case: what size t-shirt do you wear? 

On Sunday, October 25, 2015, Matthew Ring <angrydu...@gmail.com> wrote:
--
You received this message because you are subscribed to the Google Groups "nats" group.
To unsubscribe from this group and stop receiving emails from it, send an email to natsio+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply all
Reply to author
Forward
0 new messages