How to setup Hive to run on Alluxio ?

425 views
Skip to first unread message

Guy Shilo

unread,
Jul 11, 2016, 7:34:02 AM7/11/16
to Alluxio Users

Hello

I am just learning Alluxio and playing around with it. I tried working on Alluxio files with Spark and it worked (although it wasn't as fast as I expected).
Now I want to configure Hive to work on Alluxio instead of directly on HDFS (I am using latest Alluxio 1.1.0 with Hadoop 2.7.1).
There is no clear documentation about this setup, and only few question posted here. From the answers people got I can say that it is possible, but the answers aren't very clear and, again, there is no documentation.
They just say "point hive to read and write from alluxio instead of HDFS". I searched through my hive-site.xml file and was not sure which parameters I should change. Can someone give me the exact parameters I should change in order for the Alluxio-hive combination to work ?

Thank you

Guy

Yupeng Fu

unread,
Jul 11, 2016, 1:52:54 PM7/11/16
to Guy Shilo, Alluxio Users
Hi Guy,

Hive can talk to Alluxio via MR, so first you need to configure Alluxio with MR as documented here.
To point hive to Alluxio File System, you can try bin/hive --hiveconf fs.default.name=alluxio://localhost:19998

Hope this helps,


--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Yupeng Fu

unread,
Jul 19, 2016, 1:14:02 PM7/19/16
to Alluxio Users, ni4...@gmail.com
HI Guy,

Were you able to run Hive with Alluxio?
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-users+unsubscribe@googlegroups.com.

Chanh Le

unread,
Jul 20, 2016, 3:23:30 AM7/20/16
to Alluxio Users
It can run with Alluxio.
In my case I use Spark Thrift Server and Alluxio.

Yupeng Fu

unread,
Jul 20, 2016, 11:55:55 AM7/20/16
to Chanh Le, Alluxio Users
Thanks for the information!


--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-user...@googlegroups.com.

ashish kumar

unread,
Jul 21, 2016, 8:29:39 AM7/21/16
to Alluxio Users, ni4...@gmail.com
Hi,

I've tried many ways to run Hive with Alluxio but NOT able to make it work. Here is the detail about steps I followed: 


Anybody can help?

Thanks,
Ashish
HI Guy,

To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-user...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages