Presto + Spark: JDBC Driver issues

[Michael Moss]

Hello,

I'm attempting to use Spark JDBC to query to Presto and running into issues.

Initially, I tried to use the facebook presto jdbc driver and ran into the issue documented here (http://theckang.com/2016/spark-with-presto/): "com.facebook.presto.jdbc.NotImplementedException: Method Connection.prepareStatement is not yet implemented"

So, I tried the teradata presto jdbc driver which has prepared statement support, but run into a new issue:

[Teradata][Presto](100200) Connection string is invalid: dbtable is not recognized.

java.sql.SQLException: [Teradata][Presto](100200) Connection string is invalid: dbtable is not recognized.

at com.teradata.presto.core.PRConnection.connect(Unknown Source)

at com.teradata.jdbc.common.BaseConnectionFactory.doConnect(Unknown Source)

at com.teradata.jdbc.common.AbstractDriver.connect(Unknown Source)

at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$createConnectionFactory$2.apply(JdbcUtils.scala:61)

at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$createConnectionFactory$2.apply(JdbcUtils.scala:52)

at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$.resolveTable(JDBCRDD.scala:120)

at org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation.<init>(JDBCRelation.scala:91)

at org.apache.spark.sql.execution.datasources.jdbc.DefaultSource.createRelation(DefaultSource.scala:57)

at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:158)

at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:119)

It appears the 'dbtable' property which is required for Spark gets passed down into the teradata driver and throws an exception during a property validation check. I'd remove this validation logic, but can't seem to track down the source code for this driver. Is it open source?

Any other ideas?

Best,

-Mike

[Rickman, Brian T]

The Teradata JDBC driver is free but not open source.  Are you not able to configure the properties that are passed to the JDBC driver from your application?

--
You received this message because you are subscribed to the Google Groups "Presto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to presto-users...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michael Moss]

Thanks, Brian.

In my original example, the following was the syntax that did not work and threw the teradata validation error:

https://github.com/apache/spark/blob/6ee28423ad1b2e6089b82af64a31d77d3552bb38/examples/src/main/scala/org/apache/spark/examples/sql/SQLDataSourceExample.scala#L155

However, I was able to get this alternative syntax working (and reading from presto):

https://github.com/apache/spark/blob/6ee28423ad1b2e6089b82af64a31d77d3552bb38/examples/src/main/scala/org/apache/spark/examples/sql/SQLDataSourceExample.scala#L166

[roberto.f...@corp.badoo.com]

Hi Michael, would you be able to share the code referencing where to specify the JDBC driver? I suffer from the same issue (in python) and I am not able to solve it using your syntax.

Thanks!