Re: [mongodb-user] MapReduce Issue using the JAVA Driver

[Scott Hernandez]

Can you call cmd.toString() before you do your mapReduce call to make
sure the command looks correct?

How are you executing the single one that works?

What is the query used for your mapReduce commands, what do your
property files look like?

On Mon, Aug 6, 2012 at 2:33 PM, Karel Lahmy <karel...@yahoo.co.uk> wrote:
> I have a strange issue with map reduce, I have a collection which get updated daily, once it does, I am running 13 different map reduce function on that collection. For some reason when it runs in a loop, then those map reduced collection do not get updated HOWEVER when I run just on of them, it works properly. I don't see any error on the Mongo log nor on the JAVA side. Here's the code for the map/reduce part:
>
> MongoConnection con = MongoConnection.getInstance();
> con.connect( "", "", dbprop.getProperty("MongoServerURL"),dbprop.getProperty("MongoDBName"),Integer.parseInt(dbprop.getProperty("MongoPort")));
> DBCollection col = con.getDatabase().getCollection(dbprop.getProperty("MongoDBColMain"));
> String[] cols = mapredprop.getProperty("collections").split(",");
> String reduceFunc = mapredprop.getProperty("Reduce");//same for every collection
> BasicDBObject query = null;
> for (int t=0;t<cols.length;t++)
> {
> log.info("Start map reduce for collection " + cols[t]);
> if (mapredprop.getProperty(cols[t] + "Query") != null)
> {
> query = new BasicDBObject((Map)JSON.parse(mapredprop.getProperty(cols[t] + "Query")));
> }
> MapReduceCommand cmd = new MapReduceCommand( col, mapredprop.getProperty(cols[t] + "Map"),
> reduceFunc, cols[t],MapReduceCommand.OutputType.REPLACE, query);
> cmd.setFinalize(mapredprop.getProperty(cols[t] + "Finalize"));
> col.mapReduce(cmd);
> log.info("Finiah map reduce for collection " + cols[t]);
> }
>
> I've noticed in the mongo command line output that the first MapReduce iteration scans all of the collection, but the second iteration scans less than that, so I assume this is the issue, not sure how to solve this... From the command line output:
>
> Mon Aug 06 14:14:04 [conn30] build index done 0 records 0.001 secs
> 13400/51215 26%
> 28900/51215 56%
> 43500/51215 84%
> Mon Aug 06 14:14:15 [conn30] CMD: drop bsd.bsdRawChart3A
> Mon Aug 06 14:14:15 [conn30] info DFM::findAll(): extent 0:a000 was e
> stem.indexes
> Mon Aug 06 14:14:15 [conn30] CMD: drop bsd.tmp.mr.bsdRaw_26
> Mon Aug 06 14:14:15 [conn30] CMD: drop bsd.tmp.mr.bsdRaw_26
> Mon Aug 06 14:14:15 [conn30] CMD: drop bsd.tmp.mr.bsdRaw_26_inc
> Mon Aug 06 14:14:15 [conn30] command bsd.$cmd command: { mapreduce: "
> ar dateKey=this.CreateDate;dateKey=new Date(dateKey);var key=(dateKey
> onth():dateKey.getMonth())+"-"+dat...", reduce: "function Reduce(key,
> ;for(var i in values){var inter=values[i];for(var j in inter.data){re
> ..", verbose: true, out: { replace: "bsdRawChart3A" }, finalize: "fun
> c1;var ArticleCount=0;var TextbookCount=0;var OnlineCount=0;var NonJ
> 0;var NJMCount=0..." } ntoreturn:1 reslen:228 11211ms
> Mon Aug 06 14:14:15 [conn30] CMD: drop bsd.tmp.mr.bsdRaw_27_inc
> Mon Aug 06 14:14:15 [conn30] build index bsd.tmp.mr.bsdRaw_27_inc { 0
> Mon Aug 06 14:14:15 [conn30] build index done 0 records 0 secs
> Mon Aug 06 14:14:15 [conn30] CMD: drop bsd.tmp.mr.bsdRaw_27
> Mon Aug 06 14:14:15 [conn30] build index bsd.tmp.mr.bsdRaw_27 { _id:
> Mon Aug 06 14:14:15 [conn30] build index done 0 records 0.001 secs
> 11500/40410 28%
> 26500/40410 65%
> Mon Aug 06 14:14:25 [conn30] CMD: drop bsd.bsdRawChart2
> Mon Aug 06 14:14:25 [conn30] CMD: drop bsd.tmp.mr.bsdRaw_27
>
>
> Thanks.
>
> --
> You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com
> To unsubscribe from this group, send email to
> mongodb-user...@googlegroups.com
> See also the IRC channel -- freenode.net#mongodb