MongoDB Spark Connector py4j.protocol.Py4JJavaError: An error occurred while calling o50.load

[Micah Shanks]

I have been able to load this MongoDB database before, but am now receiving an error I haven't been able to figure out.

Here is how I start my Spark session:

spark = SparkSession.builder \
        .master("local[*]") \
        .appName("collab_rec") \
        .config("spark.mongodb.input.uri", "mongodb://127.0.0.1/bgg.game_comments") \
        .config("spark.mongodb.output.uri", "mongodb://127.0.0.1/bgg.game_comments") \
        .getOrCreate()

I run this script so that I can interact with spark through ipython wich loads the mongo spark connector package:

#!/bin/bash

export PYSPARK_DRIVER_PYTHON=ipython

${SPARK_HOME}/bin/pyspark \
--master local[4] \
--executor-memory 1G \
--driver-memory 1G \
--conf spark.sql.warehouse.dir="file:///tmp/spark-warehouse" \
--packages com.databricks:spark-csv_2.11:1.5.0 \
--packages com.amazonaws:aws-java-sdk-pom:1.10.34 \
--packages org.apache.hadoop:hadoop-aws:2.7.3 \
--packages org.mongodb.spark:mongo-spark-connector_2.11:2.0.0\

Spark loads fine and it appears the package is loading correctly as well.

Here is how I attempt to load that database into a dataframe:

df = spark.read.format("com.mongodb.spark.sql.DefaultSource").load()

However, on that line, I receive the following error:

Py4JJavaError: An error occurred while calling o46.load.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost, executor driver): java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.analysis.TypeCoercion$.findTightestCommonTypeOfTwo()Lscala/Function2;
 at com.mongodb.spark.sql.MongoInferSchema$.com$mongodb$spark$sql$MongoInferSchema$$compatibleType(MongoInferSchema.scala:132)
 at com.mongodb.spark.sql.MongoInferSchema$$anonfun$3.apply(MongoInferSchema.scala:76)
 at com.mongodb.spark.sql.MongoInferSchema$$anonfun$3.apply(MongoInferSchema.scala:76)

From what I can see through the following documentation/tutorial I am attempting to load the dataframe correctly:

https://docs.mongodb.com/spark-connector/master/python-api/

I am using Spark 2.2.0 Note that I have been able to replicate this error on both my mac and linux through AWS.

[Wan Bachtiar]

I am using Spark 2.2.0 Note that I have been able to replicate this error on both my mac and linux through AWS.

Hi Micah,

Since you’re using Apache Spark version 2.2.0, please also use MongoDB Connector for Spark version 2.2.0 for compatibility. You could also see the compatibility chart on the mongo-spark-connector manual.

Try to use : --packages org.mongodb.spark:mongo-spark-connector_2.11:2.2.0

Best regards,

Wan.

​

[Micah Shanks]

That was it. I changed that line in my bash script and it loaded correctly. I found out the reason is that the findTightestCommonTypeOfTwo value was renamed in the PR: https://github.com/apache/spark/pull/16786/files

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to a topic in the Google Groups "mongodb-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mongodb-user/azTODEranGg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mongodb-user+unsubscribe@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/23bb31e9-294f-4eec-9b4e-d434bb9a324f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.