Map Reduce ignoring query params

[Cwookie]

Hi all,

I'm trying to put together a simple PHP (Lithium) map/reduce function
that aggregates 'likes' per 'author', based on 1 doc per like. Here
the code:

$map = new \MongoCode("function() { emit(this.author,
this.value); }");
$reduce = new \MongoCode("function(k, vals) { ".
"var sum = 0;".
"for (var i in vals) {".
"sum += vals[i];".
"}".
"return sum; }");

$likes = $db->connection->command(array(
"mapreduce" => "boosts",
"map" => $map,
"reduce" => $reduce,
"query" => array('author' => $username, 'type' =>
'like', 'retracted' => 0),
"out" => array("merge" => "likeCount")));

$lines = $db->connection-
>selectCollection($likes['result'])->find();

var_dump($lines);


The function does as it's supposed to, the problem I have is that this
always returns aggregated result for ALL users rather than just those
for $username (the function also prints out $username so I know it's
set). If I remove the "query" array it does exactly the same thing. I
can't figure out why it's not filtering...

Any help would be much appreciated.

[Sam Millman]

It is really hard to read that code on here so ima have to take a bit of a random stab to begin.

This may be Lithium driver related. Are the other params in the query working and just not $username?

What if you use a variable param for something else like 'retracted'?

Does that function correctly?

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com.
To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.

[Cwookie]

Thanks for the quick response.

I just hard coded the 'author' parameter to a value I know if the in
DB and it still returns all entries. I also changed the other values
to nonsense, so it should return nothing, but it still returns
everything. It's like it's being ignored.

On Aug 18, 9:08 pm, Sam Millman <sam.mill...@gmail.com> wrote:
> It is really hard to read that code on here so ima have to take a bit of a
> random stab to begin.
>
> This may be Lithium driver related. Are the other params in the query
> working and just not $username?
>
> What if you use a variable param for something else like 'retracted'?
>
> Does that function correctly?
>

[Sam Millman]

Yea post this on the lith forums and see if they tell you to come back to us but I have a feeling this might be cos lith overrides command() with its own version and the query value is not being seen right.

But one last try, safe cast it to an object first and also use " to encapsulate all params:

"query" => (Object)array("author"=>1)

[Sam Millman]

Also you can try putting no query in:

"query" => array()

on mongo php driver 1.2.2 this actually return no result. So if this returns a result you know some thing is wrong else where.

[Cwookie]

Casting to Object etc. as above makes no difference.

If I remove all query params I get this error:

MongoDB::__construct(): invalid name

Will chuck it over to the lithium guys and see what they say.

On Aug 18, 9:40 pm, Sam Millman <sam.mill...@gmail.com> wrote:
> Also you can try putting no query in:
>
> "query" => array()
>
> on mongo php driver 1.2.2 this actually return no result. So if this returns
> a result you know some thing is wrong else where.
>

> On 18 August 2011 21:37, Sam Millman <sam.mill...@gmail.com> wrote:
>
>
>
>
>
>
>
> > Yea post this on the lith forums and see if they tell you to come back to
> > us but I have a feeling this might be cos lith overrides command() with its
> > own version and the query value is not being seen right.
>
> > But one last try, safe cast it to an object first and also use " to
> > encapsulate all params:
>
> > "query" => (Object)array("author"=>1)
>

[Antoine Girbal]

I just tried doing a map reduce through the java driver with a query,
something simple like:
{ "mapreduce" : "foo" ,
"map" : "function(){ emit(this._id, {count: 1}); }" ,
"reduce" : "function(key, values) { total = 0; for (var i = 0; i <
values.length; ++i) { total += values[i].count; }; return {count:
total}; }" ,
"verbose" : true ,
"out" : { "replace" : "mrtest"} ,
"query" : { "_id" : { "$oid" : "4e4d670744aef4e8d89570df"}}}

The query field worked as expected.
So I would look for an issue with the php driver or the way it's being
passed to driver.
AG

[Cwookie]

Thanks. Who's best placed to help me troubleshoot the PHP driver? Is
there a group or list?

[Sam Millman]

It can't be the PHP driver cos I use it and MR works fine for me using:

$rs = Glue::Db()->command(array(

"mapreduce"=>$this->getCollectionName(),

"map"=>$map,

"reduce"=>new MongoCode("function(key, values) {

var docs = '';

values.forEach ( function(val) { docs = val; })

return docs;

}"),

"scope"=>array('terms'=>null),

"query"=>array(),

"out"=>session_id()."_rsearch",

));

I got no construct error about it and you shouldn't.

Also casting query param as an object effects my results where as it does not effect your results.

Basically Lith has most likely put a layer on top of the command() function and it is broken.

[Sam Millman]

But the author of the PHP driver actually is in this group. So this is the best place to post really.

[Cwookie]

I just rewrote the code using the mongo driver directly:

use Mongo;
$m = new Mongo();
$db = $m->selectDB("mydb");

instead of lithium's

$db = Model::connection();

Which I assume bypasses the lithium layer, and I get exactly the same
results. Which means that either a) i'm doing something stupidly or
obviously wrong, or b) the PHP driver is at fault (which seems a
little unlikely).

Any suggestions on how to debug further?

On Aug 19, 10:32 am, Sam Millman <sam.mill...@gmail.com> wrote:
> But the author of the PHP driver actually is in this group. So this is the
> best place to post really.
>

> On 19 August 2011 10:29, Sam Millman <sam.mill...@gmail.com> wrote:
>
>
>
> > It can't be the PHP driver cos I use it and MR works fine for me using:
>
> > $rs = Glue::Db()->command(array(
> > "mapreduce"=>$this->getCollectionName(),
> >  "map"=>$map,
> > "reduce"=>new MongoCode("function(key, values) {
> >  var docs = '';
> > values.forEach ( function(val) { docs = val; })
> >  return docs;
> > }"),
> > "scope"=>array('terms'=>null),
> >  "query"=>array(),
> > "out"=>session_id()."_rsearch",
> >  ));
>
> > I got no construct error about it and you shouldn't.
>
> > Also casting query param as an object effects my results where as it does
> > not effect your results.
>
> > Basically Lith has most likely put a layer on top of the command() function
> > and it is broken.
>

> >>http://groups.google.com/group/mongodb-user?hl=en.- Hide quoted text -
>
> - Show quoted text -

[Sam Millman]

using the normal php handler and not lith if you was to use your query:

array('author' => $username, 'type' =>
'like', 'retracted' => 0)

Within a normal find what do you get back?

[Cwookie]

Also, if I use an empty array for the query - "query" => array() - i
get the same error as before:

MongoDB::__construct(): invalid name

This should return an unconstrained set should it not?

> > >>http://groups.google.com/group/mongodb-user?hl=en.-Hide quoted text -
>
> > - Show quoted text -- Hide quoted text -

[Cwookie]

using array('author' => $username, 'type' => 'like', 'retracted' => 0)
with the normal php driver yeilds an unconstrained set rather than
just entries where author='username'

On Aug 19, 12:15 pm, Sam Millman <sam.mill...@gmail.com> wrote:
> using the normal php handler and not lith if you was to use your query:
>
> array('author' => $username, 'type' =>
> 'like', 'retracted' => 0)
>
> Within a normal find what do you get back?
>

> > > >>http://groups.google.com/group/mongodb-user?hl=en.-Hide quoted text -

[Sam Millman]

"This should return an unconstrained set should it not?"

Nah in PHP 1.2.2 it actually returns nothing which is incorrect behaviour (bug) but it is a good way of seeing if you got errors before your MR.

"using array('author' => $username, 'type' => 'like', 'retracted' => 0)
with the normal php driver yeilds an unconstrained set rather than
just entries where author='username'"

AHA! So this is happening in everything.

Ok that does narrow it down a little, enough to go onto the next step.

What driver version do you have? Look for it in php_info()

What Mongo version are you using?

if you query directly from the console does it work?

[Cwookie]

Mongo: v1.8.2
PHP Mongo: v1.2.3

I tried this from the command the other day and it seemed to work
fine, but I'd need to double check to be sure (I can't do this until
later as I'm currently working on my ipad and doing mongo queries on
that may be pushing it a bit!)

On Aug 19, 12:25 pm, Sam Millman <sam.mill...@gmail.com> wrote:
> "This should return an unconstrained set should it not?"
>
> Nah in PHP 1.2.2 it actually returns nothing which is incorrect behaviour
> (bug) but it is a good way of seeing if you got errors before your MR.
>
> "using array('author' => $username, 'type' => 'like', 'retracted' => 0)
> with the normal php driver yeilds an unconstrained set rather than
> just entries where author='username'"
>
> AHA! So this is happening in everything.
>
> Ok that does narrow it down a little, enough to go onto the next step.
>
> What driver version do you have? Look for it in php_info()
>
> What Mongo version are you using?
>
> if you query directly from the console does it work?
>

> ...
>
> read more »- Hide quoted text -

[Cwookie]

Just ran the following against mongo which worked as expected:

> map = function() { emit(this.author, this.value); }
function () {
emit(this.author, this.value);
}
> reduce = function(k, vals) { var sum = 0; for (var i in vals) { sum += vals[i]; } return sum; }
function (k, vals) {
var sum = 0;
for (var i in vals) {
sum += vals[i];
}
return sum;
}
> res = db.boosts.mapReduce(map, reduce, { query : {author:"alex"}, out : 'boostres'})
{
"result" : "boostres",
"timeMillis" : 31,
"counts" : {
"input" : 4,
"emit" : 4,
"output" : 1
},
"ok" : 1,
}
> db[res.result].find()
{ "_id" : "alex", "value" : 400 }

> ...
>
> read more »

[Sam Millman]

You can downgrade your driver through PECL:

http://pagebakers.nl/post/5447529259/downgrading-a-pecl-module

So you would do:

pecl install -f mongo-1.2.2

Hopefully that might give you an older dirver, try the same thing on that

To post to this group, send email to mongod...@googlegroups.com.

[Cwookie]

I've gone back to 1.2.2 and 1.2.0 and am still getting the same
result....

On Aug 19, 7:44 pm, Sam Millman <sam.mill...@gmail.com> wrote:
> You can downgrade your driver through PECL:
>
> http://pagebakers.nl/post/5447529259/downgrading-a-pecl-module
>
> So you would do:
>
> pecl install -f mongo-1.2.2
>
> Hopefully that might give you an older dirver, try the same thing on that
>

> ...
>
> read more »

[Sam Millman]

WEIRD!

Hmmm

If you do a query (find()) with explain what does it give?

Cos I am using 1.2.2 and it is working awesome

> ...
>
> read more »

[Cwookie]

I'm convinced I'm going to find something really basic and dumb that
I've done wrong. It's just all a bit too strange.

Anyways, here's the explain. Nothing out of the ordinary here I think.

> db.boosts.find({}).explain()
{
"cursor" : "BasicCursor",
"nscanned" : 5,
"nscannedObjects" : 5,
"n" : 5,
"millis" : 0,
"nYields" : 0,
"nChunkSkips" : 0,
"isMultiKey" : false,
"indexOnly" : false,
"indexBounds" : {

}
}
> db.boosts.find({author:'alex'}).explain()
{
"cursor" : "BasicCursor",
"nscanned" : 5,
"nscannedObjects" : 5,
"n" : 4,
"millis" : 0,
"nYields" : 0,
"nChunkSkips" : 0,
"isMultiKey" : false,
"indexOnly" : false,
"indexBounds" : {

}
}

On Aug 19, 9:08 pm, Sam Millman <sam.mill...@gmail.com> wrote:
> WEIRD!
>
> Hmmm
>
> If you do a query (find()) with explain what does it give?
>
> Cos I am using 1.2.2 and it is working awesome
>

> ...
>
> read more »

[Sam Millman]

Yea 4 out of 5 objects are by alex. That is perfect.

This is really making me scratch my head. Try a php explain on the same query:

http://www.php.net/manual/en/class.mongocursor.php

> ...
>
> read more »

[Cwookie]

OK, so I just put a version of the same code in a file (and thus
totally removed from the the lithium namespace) and it works fine:

# cat test_mr.php
<?php
namespace myapp;

use Mongo;
$m = new Mongo();

$db = $m->myapp;

$map = new \MongoCode("function() { emit(this.author,
this.value); }");
$reduce = new \MongoCode("function(k, vals) { ".
"var sum = 0;".
"for (var i in vals) {".
"sum += vals[i];".
"}".
"return sum; }");

$likes = $db->command(array(
"mapreduce" => "boosts",
"map" => $map,
"reduce" => $reduce,
"query" => array("author" => "alex"),
"out" => array("merge" => "ArticleLikeCount")));

$lines = $db->selectCollection($likes['result'])->find();

foreach ($lines as $line) {
var_dump($line);
}
?>
# php -f test_mr.php
array(2) {
["_id"]=>
string(4) "alex"
["value"]=>
float(400)
}


So I guess this means the problem's with Lithium.

On Aug 19, 10:03 pm, Sam Millman <sam.mill...@gmail.com> wrote:
> Yea 4 out of 5 objects are by alex. That is perfect.
>
> This is really making me scratch my head. Try a php explain on the same
> query:
>
> http://www.php.net/manual/en/class.mongocursor.php
>

> ...
>
> read more »

[Sam Millman]

But that's just crazy, you was using the mongo object directly without lithium at one point.

Hmm I know this seems crazy but did you restart your computer after install of 1.2.2 driver? Or did you just query again straight off. Did you shut down the server after that query, switch it on this morning and then try querying?

Basically did you make sure to refresh the server after the downgrade yesterday?

> ...
>
> read more »

[Sam Millman]

Unless Lithkums active record is somehow getting in the way....."query" does normally relate to active record objects.

[Cwookie]

I suspect lithium overrides some of the db functions, so all the while
I was in that namespace, lithium ruled.

I restarted apache after downgrading.

On Aug 20, 9:15 am, Sam Millman <sam.mill...@gmail.com> wrote:
> But that's just crazy, you was using the mongo object directly without
> lithium at one point.
>
> Hmm I know this seems crazy but did you restart your computer after install
> of 1.2.2 driver? Or did you just query again straight off. Did you shut down
> the server after that query, switch it on this morning and then try
> querying?
>
> Basically did you make sure to refresh the server after the downgrade
> yesterday?
>

> ...
>
> read more »

[Sam Millman]

Yea that must be it.

See what lithium says about this and make srue to tell them it works without their namespace.

> ...
>
> read more »

[Cwookie]

I was using Lithium's mongo driver wrong. See here:

https://github.com/UnionOfRAD/lithium/issues/87

Thanks for you help with this guys.

On Aug 20, 9:23 am, Sam Millman <sam.mill...@gmail.com> wrote:
> Yea that must be it.
>
> See what lithium says about this and make srue to tell them it works without
> their namespace.
>

> ...
>
> read more »

[Sam Millman]

AHA so that explains it very well :).

Heh glad to know you got it to work.

> ...
>
> read more »