java.lang.RuntimeException: can't convert: org.bson.types.ObjectId to BSON

[yadid ayzenberg]

Hi all,

Im using spark + mongo-hadoop cdh3u3 -1.1.0 to read/write from mongodb.
The reads are working fine, however when I try to write, I get the following error:

java.lang.RuntimeException: can't convert: org.bson.types.ObjectId to BSON
    at com.mongodb.hadoop.io.BSONWritable.toBSON(BSONWritable.java:212)
    at com.mongodb.hadoop.mapred.output.MongoRecordWriter.write(MongoRecordWriter.java:57)
    at org.apache.hadoop.mapred.SparkHadoopWriter.write(SparkHadoopWriter.scala:93)
    at org.apache.spark.rdd.PairRDDFunctions.writeToFile$1(PairRDDFunctions.scala:673)
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$2.apply(PairRDDFunctions.scala:680)
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$2.apply(PairRDDFunctions.scala:680)
    at org.apache.spark.scheduler.ResultTask.run(ResultTask.scala:99)
    at org.apache.spark.scheduler.local.LocalScheduler.runTask(LocalScheduler.scala:198)
    at org.apache.spark.scheduler.local.LocalActor$$anonfun$launchTask$1$$anon$1.run(LocalScheduler.scala:68)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)

Im using com.mongodb.hadoop.mapred.MongoOutputFormat
I havent modified the data after reading it from the DB, so the key is an ObjectId and the Value is a BSONObject

Any ideas on what Im doing wrong ?

Best,

Yadid

[Nasir Rasul]

Hi Yadid,

There are a few potential problems which could be the source. Your stacktrace does not provide much information about what is going on.

- the objectId -> are you setting it yourself? or driver is generating it for you?

 --> if you are setting it yourself, there could be an error in your generation mechanism

 --> if driver is generating it, there could be a bug in there.

I would first eliminate it as a driver bug. Are you able to write using the official mongodb drivers?

- do you have any fields which are not "serializable" and I use the term serializable loosely to mean that it can't be serialized to mongodb (not the java specific semantics). An example would be JodaTime has to be manually converted into Date otherwise mongodb does not how to save it. Unless your drivers provides this functionality for you.

you can try official drivers, or other wrappers morphia, jongo. I have used all 3 at various points and basic reads/writes work. I can't say the same about mongo-hadoop cdh3u3 -1.1.0 but I expect it works.

One more note: mongo-hadoop cdh3u3 -1.1.0 seems to require driver version 2.7.3. This is an older version of mongodb driver. Is it possible you are having issues with recent mongodb version and older driver? or a combination thereof?

Regards
Nasir

 

- Nasir

--
--
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com
To unsubscribe from this group, send email to
mongodb-user...@googlegroups.com
See also the IRC channel -- freenode.net#mongodb
 
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[yadid ayzenberg]

Hi Nasir,

Im not setting the objectId myself. The driver is generating it for me.
Regarding writing - when I use the java driver directly everything works as expected. Its only when I try to use the hadoop driver that things start to fail.
I have no fields which are not serializable. Everything is either string or double.

regarding the driver version, im using 2.11.3 as other parts of my code rely on the new functionality.

From what I can tell, the problem doesnt seem related to the content of the objectId, but rather to the fact the BSONWritable class does not accept org.bson.types.ObjectId as a possible type for conversion to BSON.

Regards,
Yadid

[Mike O'Brien]

Yadid,

Can you share your map/reduce source?

It looks like the problem is that your Map or Reduce phase is returning an instance of ObjectId, but it needs to be returning an instance of BSONWritable.

[yadid ayzenberg]

 Mike,

In spark there is no Map reduce phase per say - but I can describe exactly what's happening.
When reading from Mongo - the Input format produces a dataset of Keys of type ObjectId and Values of Type BSONObject.
I take the exact same data set and try to write it to Mongo using the mongo outputFormat. Of course the keys are still ObjectId's as they have not been manipulated.
Even after doing some operations on the data set - the keys will remain of type ObjectID. The only way around this is actively exchanging the keys with a different type that is BSONWritable.
My question is: why does the hadoop-driver not support ObjectID as keys ?

Yadid

[Siyuan Zhou]

Hi Yadid,

So far, ObjectId is not supported when writing data to MongoDB. I just filed a ticket HADOOP-113 for this issue. It will be fixed in a future release.

A workaround is to convert the ObjectId type to a String type before writing the data into MongoDB.

Thanks for pointing out this issue.

Best,
Siyuan