Re: Mongo and Hadoop 1.1

Todd McGrath

unread,

Mar 15, 2013, 5:38:53 PM3/15/13

to mongod...@googlegroups.com

Yikes. I downgraded Hadoop to 1.0.* and still getting the same IO Exception.

Now, I'm stumped.

Todd McGrath

unread,

Mar 18, 2013, 11:57:55 AM3/18/13

to mongod...@googlegroups.com

Could this be a difference between 1.0.3 and 1.0.4? I downgraded to 1.0.4 and just noticed project/MongoHadoopBuild.scala is 1.0.3

Thoughts? Anything I can do to help?

mi...@10gen.com

unread,

Mar 19, 2013, 2:54:26 PM3/19/13

to info-...@10gen.com, mongod...@googlegroups.com, todd.m...@gmail.com

Todd,

Can you paste in here the full command and config you're using to execute the job? It looks like most likely a mismatch between the key/value that is expected based on the conf versus what is present in the code. 1.0.4 should be able to work fine.

On Monday, March 18, 2013 11:57:55 AM UTC-4, Todd McGrath wrote:

Could this be a difference between 1.0.3 and 1.0.4? I downgraded to 1.0.4 and just noticed project/MongoHadoopBuild.scala is 1.0.3

Thoughts? Anything I can do to help?

--
--
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com
To unsubscribe from this group, send email to
mongodb-user+unsubscribe@googlegroups.com
See also the IRC channel -- freenode.net#mongodb

---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Todd McGrath

unread,

Mar 22, 2013, 8:12:30 AM3/22/13

to mongod...@googlegroups.com, info-...@10gen.com, todd.m...@gmail.com

I'm runnin

hadoop jar examples/treasury_yield/target/treasury-example-1.1.0-SNAPSHOT.jar com.mongodb.hadoop.examples.treasury.TreasuryYieldXMLConfig

Am I using the default conf with this?

--
--
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com
To unsubscribe from this group, send email to

mongodb-user...@googlegroups.com

See also the IRC channel -- freenode.net#mongodb

---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.

FAQUser

unread,

Jun 17, 2013, 7:36:22 AM6/17/13

to mongod...@googlegroups.com, info-...@10gen.com, todd.m...@gmail.com

Having the same issue here can't resolve what the actual problem is. If anyone could shed some light I would be very great full.

mpobrien

unread,

Jun 17, 2013, 4:02:08 PM6/17/13

to mongod...@googlegroups.com, info-...@10gen.com, todd.m...@gmail.com

Hey,

Sorry all, just figured out the issue here. There was an error in the example code that was preventing it from correctly getting the mapper output class when used with 1.1. If you get the latest version of the code here http://github.com/mongodb/mongo-hadoop/ and set your build.sbt file to have:

hadoopRelease in ThisBuild := "1.1.x"

it should work fine again.

Shawn Ritchie

unread,

Jun 18, 2013, 5:56:22 AM6/18/13

to mongod...@googlegroups.com, info-...@10gen.com, todd.m...@gmail.com

Now I am getting the following error;

[hdoopuser@localhost hadoop]$ bin/hadoop jar /opt/mongo-hadoop/examples/treasury_yield/target/treasury-example_1.1.2-1.1.0.jar com.mongodb.hadoop.examples.treasury.TreasuryYieldXMLConfig

13/06/18 11:49:18 INFO util.MongoTool: Created a conf: 'Configuration: core-default.xml, core-site.xml, src/examples/treasury_yield/src/main/resources/mongo-defaults.xml, src/examples/treasury_yield/src/main/resources/mongo-treasury_yield.xml' on {class com.mongodb.hadoop.examples.treasury.TreasuryYieldXMLConfig} as job named '<unnamed MongoTool job>'

13/06/18 11:49:18 INFO util.MongoTool: Mapper Class: null

Exception in thread "main" java.lang.NullPointerException

at java.lang.Class.isAssignableFrom(Native Method)

at org.apache.hadoop.conf.Configuration.setClass(Configuration.java:922)

at org.apache.hadoop.mapreduce.Job.setMapperClass(Job.java:125)

at com.mongodb.hadoop.util.MongoTool.run(MongoTool.java:64)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)

at com.mongodb.hadoop.examples.treasury.TreasuryYieldXMLConfig.main(TreasuryYieldXMLConfig.java:41)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:601)

at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

You received this message because you are subscribed to a topic in the Google Groups "mongodb-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mongodb-user/-vUc4X_yERo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mongodb-user...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
Thanks
Shawn Ritchie

Shawn Ritchie

unread,

Jun 18, 2013, 8:13:37 AM6/18/13

to mongod...@googlegroups.com, info-...@10gen.com, todd.m...@gmail.com

never mind my previous post I had a misconfiguration in my xml files. Would really appretiate any help as I am really stuck.

But I am still getting the previous error we disucessed.

[hdoopuser@localhost hadoop]$ /opt/hadoop/hadoop/bin/hadoop jar /opt/mongo-hadoop/examples/treasury_yield/target/treasury-example_1.1.2-1.1.0.jar com.mongodb.hadoop.examples.treasury.TreasuryYieldXMLConfig

13/06/18 14:09:27 INFO util.MongoTool: Created a conf: 'Configuration: core-default.xml, core-site.xml, mongo-defaults.xml, mongo-treasury_yield.xml' on {class com.mongodb.hadoop.examples.treasury.TreasuryYieldXMLConfig} as job named '<unnamed MongoTool job>'

13/06/18 14:09:27 INFO util.MongoTool: Mapper Class: class com.mongodb.hadoop.examples.treasury.TreasuryYieldMapper

13/06/18 14:09:27 INFO util.MongoTool: Setting up and running MapReduce job in foreground, will wait for results. {Verbose? true}

13/06/18 14:09:28 INFO util.MongoSplitter: MongoSplitter calculating splits

13/06/18 14:09:28 INFO util.MongoSplitter: use shards: false

13/06/18 14:09:28 INFO util.MongoSplitter: use chunks: true

13/06/18 14:09:28 INFO util.MongoSplitter: collection sharded: false

13/06/18 14:09:28 INFO util.MongoSplitter: use range queries: false

13/06/18 14:09:28 INFO util.MongoSplitter: Creation of Input Splits is enabled.

13/06/18 14:09:28 INFO util.MongoSplitter: Using Unsharded Split mode (Calculating multiple splits though)

13/06/18 14:09:28 INFO util.MongoSplitter: Calculating unsharded input splits on namespace 'mongo_hadoop.yield_historical.in' with Split Key '{ "_id" : 1}' and a split size of '8'mb per

13/06/18 14:09:28 WARN util.MongoSplitter: WARNING: No Input Splits were calculated by the split code. Proceeding with a *single* split. Data may be too small, try lowering 'mongo.input.split_size' if this is undesirable.

13/06/18 14:09:28 INFO util.MongoSplitter: MongoSplitter found 1 splits.

13/06/18 14:09:29 INFO mapred.JobClient: Running job: job_201306181409_0001

13/06/18 14:09:30 INFO mapred.JobClient: map 0% reduce 0%

13/06/18 14:09:38 INFO mapred.JobClient: Task Id : attempt_201306181409_0001_m_000000_0, Status : FAILED

java.io.IOException: wrong value class: class com.mongodb.hadoop.io.BSONWritable is not class org.apache.hadoop.io.DoubleWritable

at org.apache.hadoop.mapred.IFile$Writer.append(IFile.java:167)

at org.apache.hadoop.mapred.Task$CombineOutputCollector.collect(Task.java:1168)

at org.apache.hadoop.mapred.Task$NewCombinerRunner$OutputConverter.write(Task.java:1492)

at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)

at com.mongodb.hadoop.examples.treasury.TreasuryYieldReducer.reduce(TreasuryYieldReducer.java:62)

at com.mongodb.hadoop.examples.treasury.TreasuryYieldReducer.reduce(TreasuryYieldReducer.java:39)

at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)

at org.apache.hadoop.mapred.Task$NewCombinerRunner.combine(Task.java:1513)

at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1436)

at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1298)

at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:699)

at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:766)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)

at org.apache.hadoop.mapred.Child$4.run(Child.java:255)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:415)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)

at org.apache.hadoop.mapred.Child.main(Child.java:249)

13/06/18 14:09:44 INFO mapred.JobClient: Task Id : attempt_201306181409_0001_m_000000_1, Status : FAILED