MapReduce when field doesn't exist in every document

[EKTW]

Is it possible to do a mapreduce when a field doesn't exist in every
document? I'm getting a "map invoke failed" error saying my field has
no properties.

[Scott Hernandez]

Yes, you need to check for the fields first.

if(obj.field) {
// do something with field

> --
> You received this message because you are subscribed to the Google Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
>

[EKTW]

Thank you, I was able to get that part figured out, but am now facing
another problem. I'm trying to use mapreduce to get the number of
embedded documents created per day, and any document in the collection
may have several of these embedded documents. So the fields in my
document look like:

blah.0.DateCreated
blah.1.DateCreated
blah.2.DateCreated

and so on. I tried using blah.DateCreated and it worked, but I got no
results. Is there a way to use mapreduce for this situation?

thanks!

[Scott Hernandez]

Can you provide a document and the reduce you want to use?

[EKTW]

The document has a lot of fields, but basically the relevant structure
looks something like this:

_id: 12345
SubDoc[2]
0
_id: 234
DateCreated: 5/7/2012 3:30 pm
1
_id: 456
DateCreated: 5/8/2012 1: 15 pm

_id: 56789
SubDoc[1]
0
_id: 632
DateCreated: 5/6/2012 2:15 am

and this is the mapreduce I'm trying to use:

map = function() {
if (!this.SubDoc.DateCreated) {
return;
}
day = this.SubDoc.DateCreated.getFullYear() + “-” +
this.SubDoc.DateCreated.getMonth() + “-“ +
this.SubDoc.DateCreated.getDate();
emit({day: day} , {count: 1});
}

reduce = function(key, values) {
var count = 0;
values.forEach(function(mr) {count += mr.count;});
return {count: count};
}

two_weeks_ago = new Date(Date.now() – 60*60*24*14*1000);
results = db.DocCollection.mapReduce(map, reduce, {out: ‘test’,
query: {“SubDoc.DateCreated”: {$gt: two_weeks_ago}}});

[Sam Millman]

The way you are accessing your arrays is the problem:

if (!this.SubDoc.DateCreated) {
return;
}

That actually means get the property of DateCreated that exists in SubDoc (only once) that exists in this doc (only once again).

What you actually wanna do is loop across SubDoc testing each row with what you have written for your MR since the map will only loop the root doc not subdocuments by default.

[EKTW]

That definitely sounds like what I want to do. Do you have an example
of how I would write that?

On May 7, 2:12 pm, Sam Millman <sam.mill...@gmail.com> wrote:
> The way you are accessing your arrays is the problem:
>
> if (!this.SubDoc.DateCreated) {
> return;
>
> }
>
> That actually means get the property of DateCreated that exists in SubDoc
> (only once) that exists in this doc (only once again).
>
> What you actually wanna do is loop across SubDoc testing each row with what
> you have written for your MR since the map will only loop the root doc not
> subdocuments by default.
>

[EKTW]

I tried writing the map like this:

map = function() {
for (var q in this.SubDoc.DateCreated) {
day = this.SubDoc.DateCreated[q].getFullYear() + “-” +
this.SubDoc.DateCreated[q].getMonth() + “-“ +
this.SubDoc.DateCreated[q].getDate();

emit({day: day} , {count: 1});
}
}

and still got no results.

Thanks so much for your help!

[Sam Millman]

Try something closer to:

for(var row in this.SubDoc){

row.DateCreated.getFullYear() + “-” +
row.DateCreated.getMonth() + “-“ +
row.DateCreated.getDate();

[EKTW]

Thanks, Sam.

I tried that, and got my original "map invoke failed: JS Error:
TypeError: row.DateCreated has no properties" error message that I
believe has to deal with not every document having something in
SubDoc. Is there a way to work in a fix for that as well?

[Sam Millman]

Yea cos it required some basic JS to make the day variable equal the date.

You can fix this by doing:

var day;
for(var row in this.SubDoc){

day = row.DateCreated.getFullYear() + “-” +

row.DateCreated.getMonth() + “-“ +
row.DateCreated.getDate();

emit({day:  day} , {count:  1});

}

But MongoDB MR runs on  a JS engine known as spider monkey (it's what Firefox uses). I would recommend you read some JavaScript basics over at: http://www.w3schools.com/js/ . You will really need to know your JS to write further MRs.

[Sam Millman]

But looking closer at your error I just realised I made a mistake myself :\ so ignore that message a little.

What is happening is that the ISODate does not actually work like that.

I believe in a MR you can do:

date = Date(row.DateCreate);

Add that to your loop and then try again using date to get the parts (http://www.w3schools.com/js/js_obj_date.asp).

[Sam Millman]

And yet again looking at your code I realise the "no properties" thing. That could denote a null field so make sure every field is filled before you try and test on them.

[EKTW]

I tried it that way, and still got the same error message. I do need
to learn more about JavaScript. Thank you for the link and for all of
your help.

On May 7, 3:33 pm, Sam Millman <sam.mill...@gmail.com> wrote:
> Yea cos it required some basic JS to make the day variable equal the date.
>
> You can fix this by doing:
>
> var day;
> for(var row in this.SubDoc){
>
> > day = row.DateCreated.getFullYear() + “-” +
> > row.DateCreated.getMonth() + “-“ +
> > row.DateCreated.getDate();
>

> > emit({day:  day} , {count:  1});
>
> }
>
> But MongoDB MR runs on  a JS engine known as spider monkey (it's what

> Firefox uses). I would recommend you read some JavaScript basics over at:http://www.w3schools.com/js/. You will really need to know your JS to
> write further MRs.

[Sam Millman]

Can you show the exact code your running?

Also: DateCreated: 5/8/2012 1: 15 pm

Are those like that in the DB? Or are they ISODATEs in the DB?

[EKTW]

Yes, I think that is the problem. There are documents that have no
SubDoc, and thus no DateCreated field. That's why I was originally
using:

if (!this.SubDoc.DateCreated) {
return;

}

Maybe I could try a combination like:

if (!this.SubDoc.DateCreated) {

var day;
for(var row in this.SubDoc){

> day = row.DateCreated.getFullYear() + “-” +
> row.DateCreated.getMonth() + “-“ +
> row.DateCreated.getDate();

> emit({day: day} , {count: 1});
}

}

On May 7, 3:38 pm, Sam Millman <sam.mill...@gmail.com> wrote:
> And yet again looking at your code I realise the "no properties" thing.
> That could denote a null field so make sure every field is filled before
> you try and test on them.
>

> On 7 May 2012 21:36, Sam Millman <sam.mill...@gmail.com> wrote:
>
>
>
>
>
>
>
> > But looking closer at your error I just realised I made a mistake myself
> > :\ so ignore that message a little.
>
> > What is happening is that the ISODate does not actually work like that.
>
> > I believe in a MR you can do:
>
> > date = Date(row.DateCreate);
>
> > Add that to your loop and then try again using date to get the parts (
> >http://www.w3schools.com/js/js_obj_date.asp).
>

> > On 7 May 2012 21:33, Sam Millman <sam.mill...@gmail.com> wrote:
>
> >> Yea cos it required some basic JS to make the day variable equal the date.
>
> >> You can fix this by doing:
>
> >> var day;
> >> for(var row in this.SubDoc){
>
> >>> day = row.DateCreated.getFullYear() + “-” +
>
> >>> row.DateCreated.getMonth() + “-“ +
> >>> row.DateCreated.getDate();
>

> >>> emit({day:  day} , {count:  1});
>
> >> }
>
> >> But MongoDB MR runs on  a JS engine known as spider monkey (it's what
> >> Firefox uses). I would recommend you read some JavaScript basics over at:

> >>http://www.w3schools.com/js/. You will really need to know your JS to
> >> write further MRs.

[Sam Millman]

You can do:

if (this.hasOwnProperty("SubDoc")) { // As seen: http://stackoverflow.com/questions/2281633/javascript-isset-equivalent

var day, date; // Lets globally define our variables
for(var row in this.SubDoc){ // For every row in our SubDoc
   if(row.hasOwnProperty("DateCreated")){ // Have we got such a field?
     
      date = Date(row.DateCreated); // convert the date to the JS version

      day = date.getFullYear() + “-” +
      date.getMonth() + “-“ +
      date.getDate();

      emit({day:  day} , {count:  1});
   }
}

But the error states that the date field itself has no properties. Which means either:

- You can used: "5/8/2012 1: 15 pm" in full string format without any sort of ISO container which the JS Date function cannot translate
- Or the field on the row itself (the date field) is null or non existant (which we are checking for niow).

[EKTW]

Tried that, and didn't get an error, but still got no results.

On May 7, 4:09 pm, Sam Millman <sam.mill...@gmail.com> wrote:
> You can do:
>
> if (this.hasOwnProperty("SubDoc")) { // As seen:http://stackoverflow.com/questions/2281633/javascript-isset-equivalent
> var day, date; // Lets globally define our variables
> for(var row in this.SubDoc){ // For every row in our SubDoc
>    if(row.hasOwnProperty("DateCreated")){ // Have we got such a field?
>
>       date = Date(row.DateCreated); // convert the date to the JS version
>
>       day = date.getFullYear() + “-” +
>       date.getMonth() + “-“ +
>       date.getDate();
>       emit({day:  day} , {count:  1});
>    }
>
> }
>
> But the error states that the date field itself has no properties. Which
> means either:
>
> - You can used: "5/8/2012 1: 15 pm" in full string format without any sort
> of ISO container which the JS Date function cannot translate
> - Or the field on the row itself (the date field) is null or non existant
> (which we are checking for niow).
>

> > > >>> > > > > > --...
>
> read more »

[Sam Millman]

Can you give me an exact dump straight from the Mongo shell of your document?

No formatting on your part or anything, a straight copy and paste..

[EKTW]

{
  "SubDoc": [
    {
      "_id": "4fa05613c8829e6620fd32ed",
      "DateCreated": "Tue, 01 May 2012 16:30:59 GMT -05:00"
    },
    {
      "_id": "4fa056150372b50fd47d8c33",
      "DateCreated": "Tue, 01 May 2012 16:31:01 GMT -05:00"
    },
    {
      "_id": "4fa05615c8829e6620fd34b2",
      "DateCreated": "Tue, 01 May 2012 16:31:01 GMT -05:00"
    },
    {
      "_id": "4fa05617c8829e597855a168",
      "DateCreated": "Tue, 01 May 2012 16:31:03 GMT -05:00"
    }
  ],
  "_id": "4fa05612c8829e6620fd32ac"
}


I removed unrelated fields.

On May 7, 5:20 pm, Sam Millman <sam.mill...@gmail.com> wrote:
> Can you give me an exact dump straight from the Mongo shell of your
> document?
>
> No formatting on your part or anything, a straight copy and paste..
>

> ...
>
> read more »

[Sam Millman]

Ok I think this is your problem:

"DateCreated": "Tue, 01 May 2012 16:30:59 GMT -05:00"

I dont think the JS Date object is capable of converting that syntax. You should use your drivers Date() class to put dates into a ISODate() object and use the localized syntax within ISO to allow you to represent GMT - 5.

Providing there isn't someting wrong with the reduce, which by the looks of it there isn't (would require more testing) since it's a simple count and return hopefully it should just be those dates.

There is one way to check, put a static value into your map for it to emit on like so:

if (this.hasOwnProperty("SubDoc")) { // As seen: http://stackoverflow.com/questions/2281633/javascript-isset-equivalent

var day, date; // Lets globally define our variables
for(var row in this.SubDoc){ // For every row in our SubDoc
   if(row.hasOwnProperty("DateCreated")){ // Have we got such a field?
     
      date = Date(row.DateCreated); // convert the date to the JS version

      day = date.getFullYear() + “-” +
      date.getMonth() + “-“ +
      date.getDate();

      emit({day:  12} , {count:  1});
   }
}

If you get one or more results your know its your dates doing it. hasOwnProperty is fully supported by spider monkey so it should be nothing to do with that, otherwise you'd get an error.

On 7 May 2012 23:39, EKTW <erin.w...@gmail.com> wrote:

{

> ...
>
> read more »

[Sam Millman]

Oh and remember to use "new Date()". I just realised I've not been using the new keyword :\.

[EKTW]

No luck. I tried using day: 12 (and new Date()) and still got no
results. I don't think the date conversion is the problem b/c I've
done the same kind of mapreduce on a date of the same type before
where the field existed in every document and that worked fine.

On May 7, 6:16 pm, Sam Millman <sam.mill...@gmail.com> wrote:
> Oh and remember to use "new Date()". I just realised I've not been using
> the new keyword :\.
>

> On 8 May 2012 00:08, Sam Millman <sam.mill...@gmail.com> wrote:
>
>
>
>
>
>
>
> > Ok I think this is your problem:
>
> > "DateCreated": "Tue, 01 May 2012 16:30:59 GMT -05:00"
>
> > I dont think the JS Date object is capable of converting that syntax. You
> > should use your drivers Date() class to put dates into a ISODate() object
> > and use the localized syntax within ISO to allow you to represent GMT - 5.
>
> > Providing there isn't someting wrong with the reduce, which by the looks
> > of it there isn't (would require more testing) since it's a simple count
> > and return hopefully it should just be those dates.
>
> > There is one way to check, put a static value into your map for it to emit
> > on like so:
>
> > if (this.hasOwnProperty("SubDoc")) { // As seen:
> >http://stackoverflow.com/questions/2281633/javascript-isset-equivalent
> > var day, date; // Lets globally define our variables
> > for(var row in this.SubDoc){ // For every row in our SubDoc
> >    if(row.hasOwnProperty("DateCreated")){ // Have we got such a field?
>
> >       date = Date(row.DateCreated); // convert the date to the JS version
>
> >       day = date.getFullYear() + “-” +
> >       date.getMonth() + “-“ +
> >       date.getDate();
> >       emit({day:  12} , {count:  1});
> >    }
> > }
>
> > If you get one or more results your know its your dates doing it.
> > hasOwnProperty is fully supported by spider monkey so it should be nothing
> > to do with that, otherwise you'd get an error.
>

> ...
>
> read more »

[EKTW]

Finally got it figured out:

map = function() {
if (!this.SubDoc.DateCreated) {

this.SubDoc.forEach(function(e){
day = e.DateCreated.getMonth()+”-“+e.DateCreated.getDate()
+”-“+e.DateCreated.getFullYear();

emit({day: day} , {count: 1});

});
}
}

reduce = function(key, values) {
var count = 0;
values.forEach(function(mr) {count += mr.count;});
return {count: count};
}

two_weeks_ago = new Date(Date.now() – 60*60*24*14*1000);
results = db.DocCollection.mapReduce(map, reduce, {out: ‘test’,
query: {“SubDoc.DateCreated”: {$gt: two_weeks_ago}}});

> > >> > > > > > >>> > > > > subdocuments by default....
>
> read more »

[Sam Millman]

Heh I had just got that solution was about to post it :P

But yes Mongo auto-translates dates, which I had forgotten (gotta admit I try and avoid MRs now-a-days).

Though you should be wary of using the false boolean to check the existance of a field, use typeof != undefined or something.

>
> read more »

[EKTW]

Yeah, for this case it worked fine, but I'll be sure to be careful
about that in the future.

Thanks, again, for all of your help! I really appreciate it!

On May 8, 3:09 pm, Sam Millman <sam.mill...@gmail.com> wrote:
> Heh I had just got that solution was about to post it :P
>
> But yes Mongo auto-translates dates, which I had forgotten (gotta admit I
> try and avoid MRs now-a-days).
>
> Though you should be wary of using the false boolean to check the existance
> of a field, use typeof != undefined or something.
>

> ...
>
> read more »