Re: Big query results solutions?

[Asya Kamsky]

Can you post some representative document and the type of aggregations that you are trying to do?

$skip and $limit don't really make sense without a sort (otherwise it's not very meaningful since the results aren't ordered)

I assume the 16M limit is the result you are getting - if you post what you are trying to calculate maybe we can see a way to create a smaller result or possibly there may be a way to split up the job into multiple aggregation queries that are more management in size?

Asya

On Monday, October 1, 2012 2:21:57 PM UTC-4, Kamlesh wrote:

I am trying to do some aggregate queries on 5M+ document sharded collections.

And am hitting the 16MB limit very frequently.

First issue is hitting limits between mongod and mongos instances, and cant do $skip/$limit pipelines without $sort.

What possible solutions are there around this?

Second issue is when doing aggregate queries via Java Drivers, the entire document is passed back.

This is killing my memory utilization, is there any way to get a cursor back?

We tried to design bigger documents, but needed to use $unwind and came back to same issue.

[Sam Millman]

There is no out option for the aggregation framework which means it just outputs as a single BSON document.

It sounds like you might need an incremental job doing this once every so often.

--
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com
To unsubscribe from this group, send email to
mongodb-user...@googlegroups.com
See also the IRC channel -- freenode.net#mongodb