SparkSQL Configuration

Julian

unread,

May 31, 2016, 5:16:30 AM5/31/16

to hue-...@cloudera.org

Hi All,

We found that that the defaul SparkSQL did not work, and we could not find any configure for the Spark SQL server. is it offically supported in the Hue 3.10? This doc just mation it but do not have more detail description. Could anyone shed some light on this?

Thanks,

Julian

This email may contain or reference confidential information and is intended only for the individual to whom it is addressed. Please refrain from distributing, disclosing or copying this email and the information contained within unless you are the intended recipient. If you received this email in error, please notify us at le...@appannie.com immediately and remove it from your system.

hyunwoo cho

unread,

May 31, 2016, 10:24:58 PM5/31/16

to Hue-Users

hi Julian

you can use SparkSQL with pyspark notebook or spark notebook. also R notebook.

there isn't way direct connect SparkSQL Server.

thank.

hyunwoo

Julian

unread,

May 31, 2016, 10:44:51 PM5/31/16

to hyunwoo cho, Hue-Users, Haitao Wu, Ruojia Zhang

Thank you hyunwoo for replay.

Is there a plan to support such function. As i know, we can connect to SparkSQL server through beeline which just like connect to HiveServer2. And they can share same table scherma.

SparkSQL Server

In spark, start up thriftserver

./start-thriftserver.sh
beeline> !connect jdbc:hive2://localhost:10001

Hive Server

beeline> !connect jdbc:hive2://localhost:10000

Thanks,

Julian

--
You received this message because you are subscribed to the Google Groups "Hue-Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hue-user+u...@cloudera.org.

Romain Rigaux

unread,

Jun 1, 2016, 2:57:32 AM6/1/16

to Julian, hyunwoo cho, Hue-Users, Haitao Wu, Ruojia Zhang

We tried the Spark SQL thrift server, it kind of works but it pretty buggy as it does not follow completely the HiveServer2 interface https://issues.cloudera.org/browse/HUE-2985

Julian

unread,

Jun 1, 2016, 3:13:15 AM6/1/16

to Romain Rigaux, hyunwoo cho, Hue-Users, Haitao Wu, Ruojia Zhang

Gotcha! Hope we can support in near future. This should be really useful tool which we can replace hive by it.

BTW, why don't we support export to a file function in the query output? is there a limitation, we just start to use it. we are going to look at the source code later.

Romain Rigaux

unread,

Jun 1, 2016, 10:59:04 AM6/1/16

to Julian, hyunwoo cho, Hue-Users, Haitao Wu, Ruojia Zhang

In the next release we might add a SparkSQL connector with Livy.

About the download, this is not implemented yet:
https://issues.cloudera.org/browse/HUE-3996

Christian Grassi

unread,

Jun 1, 2016, 1:01:35 PM6/1/16

to Romain Rigaux, Julian, hyunwoo cho, Hue-Users, Haitao Wu, Ruojia Zhang

Is there a reason not to use hive on spark?
Just curious.

Julian

unread,

Jun 1, 2016, 10:17:16 PM6/1/16

to Christian Grassi, Romain Rigaux, hyunwoo cho, Hue-Users, Haitao Wu, Ruojia Zhang

Hive on spark is not stable, it is a beta version which only static resource allocation can be used. Wait for the GA version release.

Christian Grassi

unread,

Jun 2, 2016, 4:33:59 AM6/2/16

to Julian, Haitao Wu, Hue-Users, Romain Rigaux, Ruojia Zhang, hyunwoo cho

Hive on spark on cdh 5.7 is declared stable at least in the release notes and you have dynamic resource allocation. You should give it a try.

Reply all

Reply to author

Forward