Compatiblity with Spark 2.3

[polaco...@gmail.com]

Hi,

is it possible to run olap queries using SparkGraphComputer via YARN, with Apache Spark 2.3.2 ?

https://docs.janusgraph.org/latest/version-compat.html states that only 2.2.x is supported. Has anyone tried compatibility with new version of Apache Spark?

Thanks.

[Evgeniy Ignatiev]

Hello.

As far as I remember, Spark compatibility is dictated solely by the corresponding spark-gremlin module from Tinkerpop and 3.3.x release train officially supports only 2.2.
Maybe it is worth duplicating this question to the gremlin-users list? The change that lifted Spark support to 2.3 wasn't large though - https://github.com/apache/tinkerpop/pull/886
it will probably work out of the box with properly fixed Netty dependencies versions.

Best regards,
Evgeniy Ignatiev.

--
You received this message because you are subscribed to the Google Groups "JanusGraph users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgraph-use...@googlegroups.com.
To post to this group, send email to janusgra...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/janusgraph-users/1a4414e8-abde-4432-8a18-94d9a8ed65c4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[HadoopMarc]

Hi,

JanusGraph/TinkerPop have all necessary Spark dependencies included in their distributions, so Spark compatibility on Spark/Yarn is not an issue as long as you keep the Spark jars of the Yarn cluster from the various CLASSPATHS involved. You can get the general idea by combining:

https://tinkerpop.apache.org/docs/current/recipes/#olap-spark-yarn

and 

https://docs.janusgraph.org/latest/hadoop-tp3.html

What storage backend do you use for JanusGraph? SparkGraphComputer is known to be painfully slow on JanusGraph/HBase (although I am not sure whether reading from the HBase Snaphot feature was ever implemented in JanusGraph, have not ever tried that).

Cheers,    Marc

Op woensdag 24 april 2019 12:42:17 UTC+2 schreef Evgeniy Ignatiev:

Hello.

As far as I remember, Spark compatibility is dictated solely by the corresponding spark-gremlin module from Tinkerpop and 3.3.x release train officially supports only 2.2.
Maybe it is worth duplicating this question to the gremlin-users list? The change that lifted Spark support to 2.3 wasn't large though - https://github.com/apache/tinkerpop/pull/886
it will probably work out of the box with properly fixed Netty dependencies versions.

Best regards,
Evgeniy Ignatiev.

On 4/24/2019 2:25 PM, polaco...@gmail.com wrote:

Hi,

is it possible to run olap queries using SparkGraphComputer via YARN, with Apache Spark 2.3.2 ?

https://docs.janusgraph.org/latest/version-compat.html states that only 2.2.x is supported. Has anyone tried compatibility with new version of Apache Spark?

Thanks.

--
You received this message because you are subscribed to the Google Groups "JanusGraph users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to janusgra...@googlegroups.com.

[Evgeniy Ignatiev]

By the way, some time ago we used HBase Snapshot InputFormat for JanusGraph - https://github.com/JanusGraph/janusgraph/blob/master/janusgraph-hadoop-parent/janusgraph-hadoop-core/src/main/java/org/janusgraph/hadoop/formats/hbase/HBaseSnapshotBinaryInputFormat.java - it is actually faster, in one of our tests it outpaced Cassandra by an order of magnitude, but after that we faced with some extremely tricky issues with data distribution across cluster (horrible data skew in comparison with the identical code used for Cassandra backend) and were not able to truly track the root cause in our timeframe of HBase use.

Best regards,
Evgeniy Ignatiev.

To unsubscribe from this group and stop receiving emails from it, send an email to janusgraph-use...@googlegroups.com.

To post to this group, send email to janusgra...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/janusgraph-users/9d854d76-7147-48dc-9628-3c235abe8da7%40googlegroups.com.

[Juraj Polačok]

Hi, 

Currently, I am getting this error: 

java.lang.ClassCastException: org.apache.hadoop.yarn.proto.YarnServiceProtos$GetNewApplicationRequestProto cannot be cast to org.apache.hadoop.hbase.shaded.com.google.protobuf.Message

TinkerPop 3.4+ should support Apache Spark 2.3, so I think it should work. Have you tried running some queries using Spark 2.3?