geotrellis preferred Spark and Cassandra versions

63 views
Skip to first unread message

Alexander Kmoch

unread,
May 10, 2015, 9:10:01 PM5/10/15
to geotrel...@googlegroups.com
Hi,

are there distinct preferred (required?) version of Apache Spark (1.2.x vs 1.3.x) and Cassandra (2.0.x 2.1.x) for the GeoTrellis master and feature/cassandra branches?

Cheers,
Alex

Kevin Yeh

unread,
May 10, 2015, 9:35:06 PM5/10/15
to geotrel...@googlegroups.com
Hey Alex!

I worked on the cassandra features and run Spark 1.2.0 and Cassandra 2.1.2. The feature/cassandra branch uses the Datastax Spark-Cassandra connector, which is currently compatible with Cassandra 2.0+ and Spark 1.2. They currently have a release in the works for 1.3.x, so that will hopefully be out soon.

Kevin


--
You received this message because you are subscribed to the Google Groups "geotrellis-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to geotrellis-us...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Kevin Yeh
The University of Texas at Austin

Alexander Kmoch

unread,
May 11, 2015, 5:24:55 AM5/11/15
to geotrel...@googlegroups.com
Hi Kevin,

thanks. I also have a Spark 1.2.0 with Cassandra 2.1.2 and 2.1.5 working with the Cassandra Spark Connector. So I'll stick with that until the Spark Cassandra Connector got its update.

Cheers,
Alex

Alexander Kmoch

unread,
May 12, 2015, 4:29:00 AM5/12/15
to geotrel...@googlegroups.com
Hey Kevin,

did you do a GeoTIFF ingest into Cassandra? Would you mind post your Spark submit?

Cheers,
Alex

Kevin Yeh

unread,
May 12, 2015, 2:51:10 PM5/12/15
to geotrel...@googlegroups.com
Here's the script I used for a spatial ingest: https://github.com/kyeah/gt-admin/blob/master/ingest-cassandra.sh

You'll need to have the keyspace already created. I haven't checked it with the latest refactors, so let me know if you run into any problems with it.

Best,
Kevin

--

Alex K

unread,
May 12, 2015, 8:01:29 PM5/12/15
to geotrel...@googlegroups.com
Thanks, Kevin. I did a ./sbt "project spark" assembly and referenced the
resulting jar in your script. I have the Cassandra running, KEYSPACE
created and SPARK_HOME set/exported.

I get a stack trace:

[akmoch@acer1 geotrellis-cassandra]$ ./casingest.sh
zip warning: name not matched: META-INF/ECLIPSEF.RSA

zip error: Nothing to do!
(/home/akmoch/dev/build/geotrellis-cassandra/spark/target/scala-2.10/geotrellis-spark-assembly-0.10.0-SNAPSHOT.jar)
zip warning: name not matched: META-INF/ECLIPSEF.SF

zip error: Nothing to do!
(/home/akmoch/dev/build/geotrellis-cassandra/spark/target/scala-2.10/geotrellis-spark-assembly-0.10.0-SNAPSHOT.jar)
11:55:26 Slf4jLogger: Slf4jLogger started
11:55:26 Remoting: Starting remoting
11:55:26 Remoting: Remoting started; listening on addresses
:[akka.tcp://sparkDriver@acer1:45240]
11:55:29 Cluster: New Cassandra host /127.0.0.1:9042 added
11:55:29 CassandraConnector: Connected to Cassandra cluster: Test Cluster
Exception in thread "main" java.lang.NoSuchMethodError:
org.apache.hadoop.mapreduce.Job.getInstance(Lorg/apache/hadoop/conf/Configuration;)Lorg/apache/hadoop/mapreduce/Job;
at
geotrellis.spark.io.hadoop.HdfsUtils$.putFilesInConf(HdfsUtils.scala:57)
at
geotrellis.spark.io.hadoop.package$HadoopConfigurationWrapper.withInputDirectory(package.scala:55)
at
geotrellis.spark.io.hadoop.HadoopSparkContextMethods$class.hadoopGeoTiffRDD(HadoopSparkContextMethods.scala:29)
at
geotrellis.spark.io.hadoop.package$HadoopSparkContextMethodsWrapper.hadoopGeoTiffRDD(package.scala:43)
at
geotrellis.spark.ingest.CassandraIngestCommand$$anonfun$main$1.apply(CassandraIngestCommand.scala:41)
at
geotrellis.spark.ingest.CassandraIngestCommand$$anonfun$main$1.apply(CassandraIngestCommand.scala:40)
at
geotrellis.spark.io.cassandra.Cassandra$.withSession(Cassandra.scala:19)
at
geotrellis.spark.ingest.CassandraIngestCommand$.main(CassandraIngestCommand.scala:40)
at
geotrellis.spark.ingest.CassandraIngestCommand$.main(CassandraIngestCommand.scala:29)
at com.quantifind.sumac.ArgMain$class.mainHelper(ArgApp.scala:45)
at com.quantifind.sumac.ArgMain$class.main(ArgApp.scala:34)
at
geotrellis.spark.ingest.CassandraIngestCommand$.main(CassandraIngestCommand.scala:29)
at
geotrellis.spark.ingest.CassandraIngestCommand.main(CassandraIngestCommand.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
11:55:29 CassandraConnector: Disconnected from Cassandra cluster: Test
Cluster

Ideas?

Cheers,
Alex
> <mailto:geotrellis-us...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "geotrellis-user" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/geotrellis-user/OPRJy8FBzxU/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> geotrellis-us...@googlegroups.com
> <mailto:geotrellis-us...@googlegroups.com>.
Reply all
Reply to author
Forward
0 new messages