Aggregation that fails only in sharded environment

[Mark Hansen]

I have an aggregation that uses limit at the end to keep the results under the 16MB limit.  However, when I run it in a sharded environment, it fails.  The $limit comes after a $sort - and I guess that mongos has to get the entire sorted shard results back and interleave with the other shards in order to produce the full sorted list that gets limited.

Is there anyway around this?  Can I output the results of aggregation to a temporary collection?

Here is the error message:

mongos> db.ga_page_path.aggregate(match, group, sort, limit);
{
    "errmsg" : "exception: sharded pipeline failed on shard megaset3: { errmsg: \"exception: aggregation result exceeds maximum document size (16MB)\", code: 16389, ok: 0.0 }",
    "code" : 16390,
    "ok" : 0
}

[keith kirk]

Is this no way to better limit the amount of documents you're including by having better constraints in a $match - With out seeing your actual query, I believe that would be the first place I would look.

[Stephen Lee]

I also assume that you've tried the aggregation without the sort?    In general, can you try your query w/ and w/o each successive step in your aggregation pipeline to see if the exception when and where that exception is throw?