Spark SQL Thrift JDBC/ODBC server + Cassandra

529 views
Skip to first unread message

Mohammed Guller

unread,
Apr 7, 2015, 4:23:33 PM4/7/15
to spark-conn...@lists.datastax.com

Hi –

 

Is anybody using Cassandra with the Spark SQL Thrift JDBC/ODBC server? I can programmatically (within our app) use Spark SQL with C* using the Spark-Cassandra-Connector, but can’t find any documentation on how to query C* through the Spark SQL Thrift JDBC/ODBC server. Would appreciate if you can point me to where I can find documentation on this topic.

 

Thanks.

 

Mohammed

 

Alex Liu

unread,
Apr 7, 2015, 4:26:45 PM4/7/15
to spark-conn...@lists.datastax.com
It's current only available in DSE which is a commercial product on top of C*.

Alex

Mohammed Guller

unread,
Apr 7, 2015, 4:54:31 PM4/7/15
to <spark-connector-user@lists.datastax.com>
Alex,
Thanks for the quick response.

Did Datastax create a custom version of the Spark Sql thrift server?

Mohammed
> To unsubscribe from this group and stop receiving emails from it, send an email to spark-connector-...@lists.datastax.com.

Alex Liu

unread,
Apr 7, 2015, 6:17:58 PM4/7/15
to spark-conn...@lists.datastax.com
Spark Sql thrift server is same as OSS, but we add Hive Cassandra Storage Handler/connector and some DSE lib jars.

The thrift server only works for Spark SQL Hive.

Alex

Mohammed Guller

unread,
Apr 7, 2015, 6:32:01 PM4/7/15
to spark-conn...@lists.datastax.com
Isn't the Hive Cassandra Storage Handler open source? Are the DSE lib jars that you added closed source?

Mohammed

Alex Liu

unread,
Apr 7, 2015, 6:47:35 PM4/7/15
to spark-conn...@lists.datastax.com
They are closed code. But you can try DSE product for free development.

Alex

Santosh B

unread,
May 13, 2015, 3:33:51 AM5/13/15
to spark-conn...@lists.datastax.com
Hi,
am using DSE environment and trying to connect to SPARK SQl via ODBC driver..
am getting the below error..
"SQLSTATE: HY000[Simba][SparkODBC] (68) Attempting to connect to an incorrect server type: ORG.APACHE.HADOOP.UTIL.RUNJAR /USR/SHARE/DSE/RESOURCES/HIVE/LIB/HIVE-SERVICE-0.12.0.6.JAR ORG.APACHE.HIVE.SERVICE.SERVER.HIVESERVER2. Please check your configuration."

Is there any way in DSE to start sparkthrift server like in Spark.."./sbin/start-thriftserver.sh"..?

DSE supports queries with HiveContext..does it also support CassandraContext queries....

Thanks,
Santosh B

Alex Liu

unread,
May 13, 2015, 3:59:16 PM5/13/15
to spark-conn...@lists.datastax.com
You can start thrift server by bin/dse start-spark-sql-thriftserver which launches server through start-thriftserver.sh

DSE supports both HiveContext and CassandraSQLContext. HiveContext mostly uses Cassandra Hive Storage and Spark execution engine. HiveContext uses Spark execution engine directly access Cassandra tables. Performance wise, CassandraSQLContext is faster, but there is no JDBC thrift server for it

Alex

Alex Liu

unread,
May 13, 2015, 4:01:17 PM5/13/15
to spark-conn...@lists.datastax.com
A typo, CassandraSQLContext uses Spark execution engine directly access Cassandra tables.

Santosh.B

unread,
May 18, 2015, 5:45:34 AM5/18/15
to spark-conn...@lists.datastax.com
Hi,
Was not able to find this script  "bin/dse start-spark-sql-thriftserver" in DSE 4.6.
searched in default installed directory /usr/share/dse/bin*
 
Please let me know if its a same name to start sparksqlthrift server or is it at some other location.

Thanks,
Santosh B

Thanks,
Santosh B
 

Serega Sheypak

unread,
Jun 17, 2015, 11:32:10 AM6/17/15
to spark-conn...@lists.datastax.com
среда, 8 апреля 2015 г., 0:17:58 UTC+2 пользователь Alex Liu написал:
Are you sure it works on DSE 4.7? We met weird bug: if we run query through that thrift server, spark generates hundreds of tasks ansd estimates imput as 800000 TB.
Looks like a bug, we have 3 lines in table...
Ifwe run the same query from spark console using scala api, it works without any problems.
Reply all
Reply to author
Forward
0 new messages