[project-voldemort] Voldemort - Hadoop integration

88 views
Skip to first unread message

Roshan Sumbaly

unread,
May 21, 2010, 5:42:11 PM5/21/10
to project-...@googlegroups.com
Hi all,
We've also been adding some cool new Hadoop based features to Voldemort. We have implemented VoldemortInputFormat ( http://github.com/rsumbaly/voldemort/blob/hadoop/contrib/hadoop/src/java/voldemort/hadoop/VoldemortInputFormat.java ) which would now allow users to read data directly off Voldemort and run Map Reduce jobs on them. A simple Word Count example can be run using bin/hadoop-run-job.sh .

We also have a Pig LoadFunc implementation ( http://github.com/rsumbaly/voldemort/blob/hadoop/contrib/hadoop/src/java/voldemort/hadoop/pig/VoldemortStore.java ) which would now allow users to run Pig scripts on your Voldemort data. Pig should be started using bin/voldemort-pig after which you can load data as follows :
grunt> LOAD '<admin_client_url>/<store_
name>' USING VoldemortStore() AS (key, value);
For example, grunt> LOAD 'tcp://deatheater.hogwards.edu:6667/test_store' USING VoldemortStore() AS (key, value);

This is still v0.1. Feedback is welcome!

Thanks,
Roshan

--
You received this message because you are subscribed to the Google Groups "project-voldemort" group.
To post to this group, send email to project-...@googlegroups.com.
To unsubscribe from this group, send email to project-voldem...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/project-voldemort?hl=en.
Reply all
Reply to author
Forward
0 new messages