creating a cassandra table from dataframe which have StructType columns in schema throws an exception as unsupported type

150 views
Skip to first unread message

Vishvendra Rana

unread,
Jan 4, 2018, 11:36:04 PM1/4/18
to DataStax Spark Connector for Apache Cassandra
My Dataframe Schema is:
root
|-- batters: struct (nullable = true)
| |-- batter: array (nullable = true)
| | |-- element: struct (containsNull = true)
| | | |-- id: string (nullable = true)
| | | |-- type: string (nullable = true)
|-- id: string (nullable = true)
|-- name: string (nullable = true)
|-- ppu: double (nullable = true)
|-- topping: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- id: string (nullable = true)
| | |-- type: string (nullable = true)
|-- type: string (nullable = true)

which in the StructType format in spark looks like mentioned below

StructType(StructField(batters,StructType(StructField(batter,ArrayType(StructType(StructField(id,StringType,true), StructField(type,StringType,true)),true),true)),true), StructField(id,StringType,true), StructField(name,StringType,true), StructField(ppu,DoubleType,true), StructField(topping,ArrayType(StructType(StructField(id,StringType,true), StructField(type,StringType,true)),true),true), StructField(type,StringType,true))


I am using createCassandraTable function on the dataframe to create a cassandra table:

import com.datastax.spark.connector._
df.createCassandraTable("spark_example", "nested_columns_test")


it throws an exception as:
Exception in thread "main" java.lang.IllegalArgumentException: Unsupported type: StructType(StructField(batter,ArrayType(StructType(StructField(id,StringType,true), StructField(type,StringType,true)),true),true))
at com.datastax.spark.connector.types.ColumnType$.unsupportedType$1(ColumnType.scala:132)
at com.datastax.spark.connector.types.ColumnType$.fromSparkSqlType(ColumnType.scala:155)
at com.datastax.spark.connector.mapper.DataFrameColumnMapper$$anonfun$1.apply(DataFrameColumnMapper.scala:23)
at com.datastax.spark.connector.mapper.DataFrameColumnMapper$$anonfun$1.apply(DataFrameColumnMapper.scala:21)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.immutable.List.foreach(List.scala:381)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.immutable.List.map(List.scala:285)
at com.datastax.spark.connector.mapper.DataFrameColumnMapper.newTable(DataFrameColumnMapper.scala:21)
at com.datastax.spark.connector.cql.TableDef$.fromDataFrame(Schema.scala:223)
at com.datastax.spark.connector.DataFrameFunctions.createCassandraTable(DataFrameFunctions.scala:29)
...


Although, it seems to me that StructType is not supported in the cassandra, but I just want to confirm. Could someone here confirm the same?

Reply all
Reply to author
Forward
0 new messages