Mongo Spark Connector issue

Eduardo Curonisy

unread,

Jan 17, 2017, 8:41:09 AM1/17/17

to mongodb-user

Hi, im new in MongoDB. The currently version i use is 3.4.

Mi Spark version is 1.6.0 and my Scala version is 2.10.5 (Using Scala version 2.10.5 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_67))

java -version is openjdk version "1.8.0_91"

Im trying to test the "MongoDB Spark Connector" following the basics steps documented on url https://docs.mongodb.com/spark-connector/v1.1/getting-started/:

spark-shell --conf "spark.mongodb.input.uri=mongodb://oclmongc002.isbcloud.isban.corp/adn360.personas?readPreference=primaryPreferred" \

--conf "spark.mongodb.output.uri=mongodb://oclmongc002.isbcloud.isban.corp/adn360.personas" \

--packages org.mongodb.spark:mongo-spark-connector_2.10:1.1.0 \

--jars /home/rtp/bson-3.2.2.jar

import com.mongodb.spark._

import scala.collection.JavaConverters._

import com.mongodb.spark.config._

import org.bson.Document

val rdd = MongoSpark.load(sc)

I got this error when i execute the MongoSpark.load:

scala> val rdd = MongoSpark.load(sc)

error: bad symbolic reference. A signature in MongoRDD.class refers to term client

in package com.mongodb which is not available.

It may be completely missing from the current classpath, or the version on

the classpath might be incompatible with the version used when compiling MongoRDD.class.

error: bad symbolic reference. A signature in MongoRDD.class refers to term client

in package com.mongodb which is not available.

It may be completely missing from the current classpath, or the version on

the classpath might be incompatible with the version used when compiling MongoRDD.class.

Can anyone know if i need some additional configuration?

kind regards,

Wan Bachtiar

unread,

Jan 25, 2017, 12:47:27 AM1/25/17

to mongodb-user

Can anyone know if i need some additional configuration?

Hi Eduardo,

Could you provide the full trace output from your spark-shell ? Is there any error when resolving dependencies ? You should be seeing output similar to below:

:: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
    confs: [default]
    found org.mongodb.spark#mongo-spark-connector_2.10;1.1.0 in central
    found org.mongodb#mongo-java-driver;3.2.2 in central
downloading https://repo1.maven.org/maven2/org/mongodb/spark/mongo-spark-connector_2.10/1.1.0/mongo-spark-connector_2.10-1.1.0.jar ...
    [SUCCESSFUL ] org.mongodb.spark#mongo-spark-connector_2.10;1.1.0!mongo-spark-connector_2.10.jar (2310ms)
downloading https://repo1.maven.org/maven2/org/mongodb/mongo-java-driver/3.2.2/mongo-java-driver-3.2.2.jar ...
    [SUCCESSFUL ] org.mongodb#mongo-java-driver;3.2.2!mongo-java-driver.jar (3585ms)
    :: resolution report :: resolve 14468ms :: artifacts dl 5909ms

    --jars /home/rtp/bson-3.2.2.jar

Could you also try to remove this bson-3.2.2.jar ?

The mongo-java-driver jar (dependency of the mongo-spark-connector) is an all-in-one jar that includes the core driver and the BSON. It is likely that the namespace collides which then failed to resolve.

Regards,

Wan

Eduardo Curonisy

unread,

Feb 10, 2017, 6:29:21 AM2/10/17

to mongodb-user

Hi Wan,

Here is my trace output.

Ivy Default Cache set to: /home/rtp/.ivy2/cache

The jars for the packages stored in: /home/rtp/.ivy2/jars

:: loading settings :: url = jar:file:/data/1/cloudera/parcels/CDH-5.8.3-1.cdh5.8.3.p0.2/jars/spark-assembly-1.6.0-cdh5.8.3-hadoop2.6.0-cdh5.8.3.jar!/org/apache/ivy/core/settings/ivysettings.xml

org.mongodb.spark#mongo-spark-connector_2.10 added as a dependency

:: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0

confs: [default]

found org.mongodb.spark#mongo-spark-connector_2.10;1.1.0 in local-m2-cache

:: resolution report :: resolve 289ms :: artifacts dl 3ms

:: modules in use:

org.mongodb.spark#mongo-spark-connector_2.10;1.1.0 from local-m2-cache in [default]

---------------------------------------------------------------------

| | modules || artifacts |

| conf | number| search|dwnlded|evicted|| number|dwnlded|

---------------------------------------------------------------------

| default | 1 | 0 | 0 | 0 || 1 | 0 |

---------------------------------------------------------------------

:: retrieving :: org.apache.spark#spark-submit-parent

confs: [default]

0 artifacts copied, 1 already retrieved (0kB/7ms)

Setting default log level to "WARN".

To adjust logging level use sc.setLogLevel(newLevel).

Welcome to

____ __

/ __/__ ___ _____/ /__

_\ \/ _ \/ _ `/ __/ '_/

/___/ .__/\_,_/_/ /_/\_\ version 1.6.0

/_/

Using Scala version 2.10.5 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_67)

Type in expressions to have them evaluated.

Type :help for more information.

Spark context available as sc (master = yarn-client, app id = application_1486455252346_0102).