First - I work on one of those drivers. ;-)
I have spent more hours than I care to count trying to make MongoDB go as fast as possible. I assure you we do lots of really crazy stuff and worry about every bit of latency we can squeeze out of the system. Having to make two round trips to the server to get 1 result is exactly the kind of stuff that blows our performance budget.
The find does return a result that is close to the 16MB limit in the first reply. There is no 4MB limit imposed on the first reply. The aggregate cursor has to based on the format of the reply alone.
What I expected was for the reply to a cursor based aggregation to be a OP_REPLY message where each of the result documents in the message was one of the results from the aggregation.
I have already written the code to adapt the "command" cursor into a standard reply, I am just wondering why not make the command cursors look like a query's cursor from the very first reply. Like I asked in the original message: What am I missing?
Rob
P.S. In writing a test to verify that find would return and the aggregation wouldn't (my attempt at understanding the code) I ran into a bug... The code below (which uses my driver and a pre-release version at that) returns this reply:
{
errmsg : 'exception: Tried to create string longer than 16MB',
code : 16493,
ok : 0.0
}
If I reduce the size of the _id by 1000 bytes I get back a cursor document but with no entries in the "firstBatch".
{
cursor : {
id : NumberLong('89091207075701'),
ns : 'acceptance_test.acceptance_1',
firstBatch : []
},
ok : 1.0
}
I will try to reproduce in the shell and submit a Jira ticket.
@Test
public void testFindReallyBigDocument() {
final DocumentBuilder doc = BuilderFactory.start();
// 4 is the cString for _id.
// 6 is the Binary Element overhead.
// 5 is the Document overhead.
doc.add("_id", new byte[Client.MAX_DOCUMENT_SIZE - (4 + 6 + 5)]);
myCollection.insert(Durability.ACK, doc);
final Find.Builder find = new Find.Builder();
find.setQuery(Find.ALL);
MongoIterator<Document> iter = myCollection.find(find);
try {
assertTrue(iter.hasNext());
assertEquals(doc.build(), iter.next());
assertFalse(iter.hasNext());
}
finally {
iter.close();
}
// Now use aggregation.
final Aggregation aggregation = Aggregation.builder().match(Find.ALL)
.useCursor().build();
iter = myCollection.aggregate(aggregation);
try {
assertTrue(iter.hasNext());
assertEquals(doc.build(), iter.next());
assertFalse(iter.hasNext());
}
finally {
iter.close();
}
}