Realtime vs Long Term Data Analysis with Storm/Hadoop/Cassandra

[Gary Malouf]

I've been experimenting with Storm + Cassandra for our realtime ad serving analytics platform.  While doing research, I came across a 2011 blog post from Nathan: http://nathanmarz.com/blog/how-to-beat-the-cap-theorem.html.  One of the things that stood out to me was the suggestion that everything in Cassandra be transient as the 'correct' computation will take place in Hadoop within a few hours.  Is this how Twitter handles their data analytics - keeping the past few hours almost accurate data in Cassandra but replacing this with Hadoop's batch processing results as time goes on?

Basically, I'm trying to decide whether I should keep things like impression/click counts JUST in Cassandra or if I should be recomputing everything in Hadoop. 

Connection to Storm - as demonstrated in the ETE 2012 presentation,  I am using Storm to validate data and run partial ETL (Translate third party data to internal api).  Once these steps are done, I both append to Hadoop and update the minute buckets for the ad in Cassandra column family. 

[Nathan Marz]

Yes, that's how we're doing much of our data analytics. The realtime computation is still correct, but having the majority of our rollups served via Hadoop computed indices gives us a lot of flexibility with respect to doing recomputes when necessary. It also lets us keep our Cassandra cluster much smaller, which is a huge win.

--
You received this message because you are subscribed to the Google Groups "storm-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Twitter: @nathanmarz
http://nathanmarz.com

[Gary Malouf]

Is it possible?  Yes.  I am setting up storm ( as Twitter does) to validate incoming data and then in append it to Hadoop along with updating my DHT Cassandra.  I think the only difference in what you are describing is to update HBase instead of Cassandra right?

On Tuesday, March 5, 2013 6:34:03 AM UTC-5, SHOBIN JOSEPH wrote:

Can we Integrate Hadoop and Storm together? Is it possible to fetch data relatime from Haoop's Hbase through Storm and can we update tables in Hbase using Storm?

[Viral Bajaria]

Yes it is possible to integrate HBase and Storm. You can use HBase as your final database while Storm as your processing engine. I am not sure what you meant by integrating Hadoop, if all you meant was to ask about HBase then yes that is possible but if you meant whether you can integrate MapReduce then No (or atleast I don't know about it) but you can definitely write to HDFS.

On Tue, Mar 5, 2013 at 3:34 AM, SHOBIN JOSEPH <shobinjo...@gmail.com> wrote:

Can we Integrate Hadoop and Storm together? Is it possible to fetch data relatime from Haoop's Hbase through Storm and can we update tables in Hbase using Storm?

On Tuesday, 26 February 2013 10:49:02 UTC+5:30, Nathan Marz wrote:

[Ted Dunning]

Funny you should use just those words.

http://www.slideshare.net/tdunning/buzz-words-2012-realtime-learning

You don't even necessarily need something like HBase or Cassandra.

On Tue, Mar 5, 2013 at 6:34 AM, SHOBIN JOSEPH <shobinjo...@gmail.com> wrote:

Can we Integrate Hadoop and Storm together? Is it possible to fetch data relatime from Haoop's Hbase through Storm and can we update tables in Hbase using Storm?


On Tuesday, 26 February 2013 10:49:02 UTC+5:30, Nathan Marz wrote:

[SHOBIN JOSEPH]

Hi,


Whether Storm and Kafka can co-exist with Apache Hadoop (in the same cluster or different)?
whether HBASE can support both MapReduce as well as Storm based data processing?

[Ted Dunning]

Your questions are hard to understand.

What is it that you are actually asking?

--

[SHOBIN JOSEPH]

Hi, Currently I had a hadoop cluster avialable, whether I can configure storm on the same cluster? I am planning to use Apache Kafka and Storm for big data real time analytics in the same hadoop cluster, is this architecture is possible? whether i can pass the data in hbase(in the above mentioned cluster) to storm and/or the output of storm to hbase?

[manoj jannu]

hi,i am very new to storm -cassandra.can anyone plz send me a proper working steps for storm-cassandra integration,

[storm dev]

pls help me add this group, thanks