Is vert.x a good fit for Real Time Stream Analytics Use Case?

[KeepSimple]

Hi, 

Has anyone used vert.x for Real Time Stream Analytics? I am thinking more like Apache Storm. I am wondering whether Vert.x would be a good fit for such a use case.

[jordan.h...@gmail.com]

It really depends on your availability/fault tolerance/scaling requirements and the complexity of your streams and processing/aggregations. Storm has a memory efficient algorithm for tracking complex tuples of streams. But Storm's algorithm isn't well suited for doing aggregations, which is why they had to build Trident on top of it, and Trident tends to rely significantly on external databases like Cassandra and ZooKeeper to ensure stream consistency which can often mean more work for the user. So, IMO Storm isn't great for doing anything even remotely math heavy - you should really look at Spark or Flink for that. 

If your streams and processing within them are not too complex you can get away with Vert.x for its simplicity and flexibility. You can track message completion with request-reply, or you can implement Storm's algorithm fairly easily (it's just taking the XOR of random message IDs to track completion of related messages once a 64-bit tracker becomes 0).

But considering you used the word "analytics," I would strongly discourage going outside of the well established open source Apache projects like Spark, Storm, Samza, or Flink. Stream aggregations are a delicate process if you expect to run your processes on an unreliable system (which, presumably, you do) and still attain accurate results. Flink in particular seems to be the popular topic of conversation these days, but I can't really vouch for its usefulness as my experience is with Spark and Storm.

On Sep 29, 2015, at 1:22 PM, KeepSimple <chira...@gmail.com> wrote:

Hi, 

Has anyone used vert.x for Real Time Stream Analytics? I am thinking more like Apache Storm. I am wondering whether Vert.x would be a good fit for such a use case.

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at http://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/de94a83a-7b72-4468-8f6b-5ce8138d46ce%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tim Fox]

Jordan-

Do you have plans to port Vertigo to Vert.x 3? I thought it was a pretty cool project.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/A5FBDF22-3C7B-4143-B822-91C5F78656B7%40gmail.com.

[Cosmic Interloper]

I think orolle already started?

https://github.com/orolle/vertigo-prototype

[jordan.h...@gmail.com]

Indeed. I have been talking to him a bit, and he more or less took it on himself. I would love to be working on it some more, but it's currently my third favorite project (behind http://github.com/atomix/copycat and now http://github.com/atomix/atomix)

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/9da9a496-56d0-42bf-83c1-c127f91bc8f8%40googlegroups.com.