How to set cursor timeout for long running collection->group() operation?

[Martin Thierer]

I'm running "group()" on a query using MongoDb 3.0.3 and the PHP driver 1.6.9 under PHP 5.5.

The query takes about two minutes to complete, so with the default settings, it throws an MongoExecutionTimeoutException after 30 seconds.

I did expect that raising the limit using the "maxTimeMS" option to group() as documented at http://be2.php.net/manual/en/mongocollection.group.php would help, but it doesn't. I don't really understand why, but I suspect it's about some server-side timeout, while the timeout I get is a client-side timeout.

What does help is setting MongoCursor::$timeout to e.g. -1, but that gives me a depecation warning, so I wonder what would be the correct way to set the timeout. As collection->group() returns the result directly, I can't see a way to use cursor->timeout() as suggested by the deprecation warning.

Besides, even if it's a server side timeout, shouldn't the group() function set the timeout for its internal cursor set at least to the value of the "maxTimeMS"? The option seems pretty pointless to me otherwise.

Thanks

Martin

[Asya Kamsky]

My strong recommendation would be to not use group() which is a subset
of mapReduce and switch to aggregation framework instead.

This doesn't directly answer your question as to why maxTimeMS isn't
working for you (when it is expected to work) but I can assure you
that aggregation framework would be faster, possibility side-stepping
this issue for you entirely.

Asya

> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user"
> group.
>
> For other MongoDB technical support options, see:
> http://www.mongodb.org/about/support/.
> ---
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mongodb-user...@googlegroups.com.
> To post to this group, send email to mongod...@googlegroups.com.
> Visit this group at http://groups.google.com/group/mongodb-user.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/mongodb-user/3d9c3cf9-60ba-4a66-acdb-26e216b2d4e2%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Hannes Magnusson]

Are you sure you got MongoExecutionTimeoutException using the default settings?
I would have expected MongoCursortimeoutException.

MongoExecutionTimeoutException means we hit maxTimeMS -- while MongoCursorTimeoutException means we hit the client-side socket timeout.

maxTimeMS is applied by the server.
socket timeout is applied by the client.

When a socket timeout is hit, the server will continue processing whatever it was supposed to process -- but the client kills the connection and errors.
Whena maxTimeMS is hit, the server stops the processing and tells the client "hey, this took me longer then you wanted, I'm cancelling it now".


There is currently no way for you to provide the "socketTimeoutMS" option as part of calling $collection->group();
It is generally passed in as part of the connection string, to MongoClient.
On the flipside, its default to 30seconds as normal web apps would never accept waiting 30seconds for something to happen :]

Hope it helps.

-Hannes

[Martin Thierer]

Thanks, Hannes, that helped indeed!

You're right, I mixed up MongoExecutionTimeoutException and MongoCursorTimeoutException, so no wonder that setting maxTimeMS didn't help... I now set socketTimeoutMS when connecting and that works fine. 

[Martin Thierer]

Thanks for your suggestion, but the problem with this particular query is that it's using a slightly complex JS reduce function. The possibility to do that is why I still sometimes prefer group(), because the custom javascript makes some things straightforward that get rather complicated if not impossible with aggregate().

So you're right, that's probably the reason why it's running so slow in the first place, but it's for an analysis I only run every few months, so I don't care much about performace anyway.