CassandraSQL Context

[bobby.ch...@gmail.com]

I am working on adding a custom interpreter for CassandraSQLContext (using datastax cassadnra spark connector https://github.com/datastax/spark-cassandra-connector) basically its extension of SparkSQLContext. I was able to get it compile and working but facing issues when table has numbers, i tested with Int, Long , Double. For int it fails with below error

Java.util.NoSuchElementException: key not found: scala.Int
 at scala.collection.MapLike$class.default(MapLike.scala:228)
 at scala.collection.AbstractMap.default(Map.scala:58)
 at scala.collection.MapLike$class.apply(MapLike.scala:141)
 at scala.collection.AbstractMap.apply(Map.scala:58)
 at org.apache.spark.sql.cassandra.ColumnDataType$.scalaDataType(CassandraRelation.scala:66)
 at org.apache.spark.sql.cassandra.CassandraRelation.columnToAttribute(CassandraRelation.scala:25)
 at org.apache.spark.sql.cassandra.CassandraRelation$$anonfun$1.apply(CassandraRelation.scala:15)
 at org.apache.spark.sql.cassandra.CassandraRelation$$anonfun$1.apply(CassandraRelation.scala:15)
 at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
 at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
 at scala.collection.immutable.List.foreach(List.scala:318)
 at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
 at scala.collection.AbstractTraversable.map(Traversable.scala:105)
 at org.apache.spark.sql.cassandra.CassandraRelation.<init>(CassandraRelation.scala:15)
 at org.apache.spark.sql.cassandra.CassandraCatalog.lookupRelation(CassandraCatalog.scala:42)
 at org.apache.spark.sql.cassandra.CassandraSQLContext$$anon$2.org$apache$spark$sql$catalyst$analysis$OverrideCatalog$$super$lookupRelation(CassandraSQLContext.scala:60)
 at org.apache.spark.sql.catalyst.analysis.OverrideCatalog$$anonfun$lookupRelation$3.apply(Catalog.scala:123)
 at org.apache.spark.sql.catalyst.analysis.OverrideCatalog$$anonfun$lookupRelation$3.apply(Catalog.scala:123)
 at scala.Option.getOrElse(Option.scala:120)
 at org.apache.spark.sql.catalyst.analysis.OverrideCatalog$class.lookupRelation(Catalog.scala:123)
 at org.apache.spark.sql.cassandra.CassandraSQLContext$$anon$2.lookupRelation(CassandraSQLContext.scala:65)
 at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$2.applyOrElse(Analyzer.scala:84)
 at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$2.applyOrElse(Analyzer.scala:82)
 at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:165)
 at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:183)
 at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
 at scala.collection.Iterator$class.foreach(Iterator.scala:727)
 at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
 at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
 at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
 at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
 at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
 at scala.collection.AbstractIterator.to(Iterator.scala:1157)
 at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
 at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
 at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
 at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
 at org.apache.spark.sql.catalyst.trees.TreeNode.transformChildrenDown(TreeNode.scala:212)
 at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:168)
 at org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:156)
 at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:82)
 at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:81)
 at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$apply$1$$anonfun$apply$2.apply(RuleExecutor.scala:61)
 at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$apply$1$$anonfun$apply$2.apply(RuleExecutor.scala:59)
 at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:111)
 at scala.collection.immutable.List.foldLeft(List.scala:84)
 at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$apply$1.apply(RuleExecutor.scala:59)
 at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$apply$1.apply(RuleExecutor.scala:51)
 at scala.collection.immutable.List.foreach(List.scala:318)
 at org.apache.spark.sql.catalyst.rules.RuleExecutor.apply(RuleExecutor.scala:51)
 at org.apache.spark.sql.SQLContext$QueryExecution.analyzed$lzycompute(SQLContext.scala:397)
 at org.apache.spark.sql.SQLContext$QueryExecution.analyzed(SQLContext.scala:397)
 at org.apache.spark.sql.SQLContext$QueryExecution.optimizedPlan$lzycompute(SQLContext.scala:398)
 at org.apache.spark.sql.SQLContext$QueryExecution.optimizedPlan(SQLContext.scala:398)
 at org.apache.spark.sql.SQLContext$QueryExecution.sparkPlan$lzycompute(SQLContext.scala:402)
 at org.apache.spark.sql.SQLContext$QueryExecution.sparkPlan(SQLContext.scala:400)
 at org.apache.spark.sql.SQLContext$QueryExecution.executedPlan$lzycompute(SQLContext.scala:406)
 at org.apache.spark.sql.SQLContext$QueryExecution.executedPlan(SQLContext.scala:406)
 at org.apache.spark.sql.SchemaRDD.collect(SchemaRDD.scala:438)
 at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:23)
 at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:28)
 at $iwC$$iwC$$iwC$$iwC.<init>(<console>:30)
 at $iwC$$iwC$$iwC.<init>(<console>:32)
 at $iwC$$iwC.<init>(<console>:34)
 at $iwC.<init>(<console>:36)
 at <init>(<console>:38)
 at .<init>(<console>:42)
 at .<clinit>(<console>)
 at .<init>(<console>:7)
 at .<clinit>(<console>)
 at $print(<console>)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:606)
 at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:789)
 at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1062)
 at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:615)
 at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:646)
 at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:610)
 at com.nflabs.zeppelin.spark.SparkInterpreter._interpret(SparkInterpreter.java:359)
 at com.nflabs.zeppelin.spark.SparkInterpreter.interpret(SparkInterpreter.java:341)
 at com.nflabs.zeppelin.spark.SparkInterpreter.interpret(SparkInterpreter.java:335)
 at com.nflabs.zeppelin.interpreter.ClassloaderInterpreter.interpret(ClassloaderInterpreter.java:46)
 at com.nflabs.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:50)
 at com.nflabs.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:176)
 at com.nflabs.zeppelin.scheduler.Job.run(Job.java:132)
 at com.nflabs.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:75)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
 at java.util.concurrent.FutureTask.run(FutureTask.java:262)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 at java.lang.Thread.run(Thread.java:745)

I am not sure if this is issue at Interpreter or some where. Interestingly i tested this with table that has only string type and works with no issues....

Can any one point me in the right direction??

Thanks

[moon soo Lee]

Hi bobby,

With your stacktrace, i couldn't get enough idea to help because of i have no experience with spark-cassandra-connector.

Is there anyway i can reproduce the problem?

Sorry for not being helpful.

Thanks,

moon

 at scala.collection.TraversableOnce$class.toBuffer(<span style="color: #606;" class="styled-by-prett

...

[Todd Nist]

Hi Bobby,

I'm getting ready to start down this path and was wondering if by chance you would mind sharing your initial start at an Interpreter for the CassandraSQL Context?  TIA for the assistance.

- Todd

...