Unable to connect Hive with MongoDB using mongo-hadoop connector

366 views
Skip to first unread message

Prathamesh joshi

unread,
Aug 8, 2017, 9:53:42 AM8/8/17
to mongodb-user
Hello,

I am begI am trying to install and configure hive with mongo-hadoop-core 2.0.2, for the first time. I have installed hadoop 2.8.0, Hive 2.1.1 and MongoDB 3.4.6. and everything works fine when running individually.

My problem is, I am not able to connect MongoDB with Hive. I am using mongo-Hadoop connector for this as mentioned here https://github.com/mongodb/mongo-hadoop/wiki/Hive-Usage The required jars are added to Hadoop and Hive lib. Even I add them in hive.sh or runtime from hive console. I am getting error while executing Create table query

My Query is 

CREATE EXTERNAL TABLE testHive ( id STRING, name STRING )
 STORED BY 'com.mongodb.hadoop.hive.MongoStorageHandler' WITH 
 SERDEPROPERTIES('mongo.columns.mapping'='{"id":"_id","name":"name"}') 
 TBLPROPERTIES('mongo.uri'='mongodb://localhost:27017/hiveDb.testHive');


And I get the following error 

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. com/mongodb/hadoop/io/BSONWritable
hive> ERROR hive.ql.exec.DDLTask - java.lang.NoClassDefFoundError: com/mongodb/hadoop/io/BSONWritable 
at com.mongodb.hadoop.hive.BSONSerDe.initialize(BSONSerDe.java:132) 
at org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:537) 
at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:424) 
at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:411) 
at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:279) 
at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:261)



It shows that com/mongodb/hadoop/io/BSONWritable class is not in classpath but I have added the required(mongo-hadoop-core.jar) jar and class are present in the jar.

The version of jars I am using


mongo-hadoop-core 2.0.2,

mongo-hadoop-hive 2.0.2,

mongo-java-driver 3.0.2

Thanks

Prathamesh joshi

unread,
Aug 8, 2017, 9:54:27 AM8/8/17
to mongodb-user
Hello 

I am trying to install and configure hive with mongo-hadoop-core 2.0.2, for the first time. I have installed hadoop 2.8.0, Hive 2.1.1 and MongoDB 3.4.6. and everything works fine when running individually.

   My problem is, I am not able to connect MongoDB with Hive. I am using mongo-Hadoop connector for this as mentioned here https://github.com/mongodb/mongo-hadoop/wiki/Hive-Usage The required jars are added to Hadoop and Hive lib. Even I add them in hive.sh and runtime from hive console (with ADD JAR .....). I am getting error while executing Create table query

Wan Bachtiar

unread,
Aug 17, 2017, 10:00:49 PM8/17/17
to mongodb-user

It shows that com/mongodb/hadoop/io/BSONWritable class is not in classpath

H Prathamesh,

Based on the error message that you’re getting it is likely that mongo-hadoop-core is not registered properly, as the class com.mongodb.hadoop.io.BSONWritable is in mongo-hadoop-core-2.0.2.jar.

Note that with Apache Hive, you need to specify full path to the jar, for example: 

ADD JAR /full/path/to/mongo-hadoop-core-2.0.2.jar;

Unrelated to your question, I would also recommend to use mongo-java-driver version 3.4.x if you are using MongoDB server v3.4.6.
See also MongoDB Java driver compatibility

Regards,
Wan.

​
Reply all
Reply to author
Forward
0 new messages