How to execute the sensor logs example of Mongo-Hadoop Connector?

[Eric Lin]

I want to combine MongoDB with Hadoop. What I found is Mongo-Hadoop Connector. However I could not find complete document about this example.

There are four files in mongo-hadoop/examples/sensors, build, run_job.sh, src, testdata_generator.js, respectively. I imported data to MongoDB by using testdata_generator.js, and the dbs is demo. When I try to run run_job.sh, there has an exception:

MongoDB shell version: 2.6.1
connecting to: demo
false
Exception in thread "main" java.lang.ClassNotFoundException: -D
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:249)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:205)

run_job.sh

#!/bin/sh


mongo demo --eval "db.logs_aggregate.drop()"
#Set your HADOOP_HOME directory here.
#export HADOOP_HOME="/Users/mike/hadoop/hadoop-2.0.0-cdh4.3.0" 
export HADOOP_HOME="/home/hduser/hadoop"

#FIRST PASS - map all the devices into an output collection
declare -a job1_args
job1_args=("jar" "`pwd`/build/libs/sensors-1.2.1-SNAPSHOT-hadoop_2.2.jar")
#job1_args=(${job1_args[@]} "com.mongodb.hadoop.examples.sensors.Devices")
job1_args=(${job1_args[@]} "-D" "mongo.job.input.format=com.mongodb.hadoop.MongoInputFormat")
job1_args=(${job1_args[@]} "-D" "mongo.input.uri=mongodb://localhost:27017/demo.devices")
job1_args=(${job1_args[@]} "-D" "mongo.job.mapper=com.mongodb.hadoop.examples.sensors.DeviceMapper")
job1_args=(${job1_args[@]} "-D" "mongo.job.reducer=com.mongodb.hadoop.examples.sensors.DeviceReducer")

job1_args=(${job1_args[@]} "-D" "mongo.job.output.key=org.apache.hadoop.io.Text")
job1_args=(${job1_args[@]} "-D" "mongo.job.output.value=org.apache.hadoop.io.Text")

job1_args=(${job1_args[@]} "-D" "mongo.output.uri=mongodb://localhost:27017/demo.logs_aggregate")
job1_args=(${job1_args[@]} "-D" "mongo.job.output.format=com.mongodb.hadoop.MongoOutputFormat")
$HADOOP_HOME/bin/hadoop "${job1_args[@]}" "$1"

I can run basic Map/Reduce example in my computer, but this problem bothers me many days...

I also posted this questions on stackoverflow: http://stackoverflow.com/questions/24131808/how-to-execute-the-sensor-logs-example-of-mongo-hadoop-connector

[Justin Lee]

run it using gradle.  those bash scripts are a bit outdated and should be removed:

./gradlew sensorData

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/4856158a-cc22-4551-817a-8535d503ff74%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.