Aggregation with $out throwing org.bson.BsonSerializationException: Document size of X is larger than maximum of 16793600

[Willian Campos]

Hello guys!

I am facing a problem when aggregating data.

My aggregation pipeline is being outputted to a collection ($out), so as far as I understand the output total size should not have the 16MB limit.

Sometime ago I was not using such output and then I had problems with the max size, so I started using the $out collection and the problem was fixed.

But now, I have more data in the input collection I am aggregating from and this exception is being thrown.

The only limitation I am aware of is that each document inserted on the $out collection should not be over 16MB, but they are not (the documents resulting from the aggregation are only composed by two String fields).

Am I missing something here?

Here is my aggregation code (I am using Spring Data MongoDB):

Aggregation aggregation = newAggregation(
        match(Criteria.where("property1").in(property1List).and("type.$ref").is("MY_TYPE").and("type.$id").in(idList).and("property2").ne(true)),
        sort(Sort.Direction.ASC, "property1", "property2"),
        project("property1", "property2"),
        group("property1", "property2"),
        out(tempCollection)
).withOptions(new AggregationOptions.Builder().allowDiskUse(true).build());

template.aggregate(aggregation, MyCollection.class, TempCollection.class);

And here is the exception being thrown:

Caused by: org.bson.BsonSerializationException: Document size of 19888895 is larger than maximum of 16793600.

at org.bson.BsonBinaryWriter.backpatchSize(BsonBinaryWriter.java:376) ~[mongo-java-driver-3.4.2.jar:?]

at org.bson.BsonBinaryWriter.doWriteEndArray(BsonBinaryWriter.java:143) ~[mongo-java-driver-3.4.2.jar:?]

at org.bson.AbstractBsonWriter.writeEndArray(AbstractBsonWriter.java:347) ~[mongo-java-driver-3.4.2.jar:?]

at com.mongodb.DBObjectCodec.encodeIterable(DBObjectCodec.java:273) ~[mongo-java-driver-3.4.2.jar:?]

at com.mongodb.DBObjectCodec.writeValue(DBObjectCodec.java:200) ~[mongo-java-driver-3.4.2.jar:?]

at com.mongodb.DBObjectCodec.encodeMap(DBObjectCodec.java:222) ~[mongo-java-driver-3.4.2.jar:?]

at com.mongodb.DBObjectCodec.writeValue(DBObjectCodec.java:198) ~[mongo-java-driver-3.4.2.jar:?]

at com.mongodb.DBObjectCodec.encodeMap(DBObjectCodec.java:222) ~[mongo-java-driver-3.4.2.jar:?]

at com.mongodb.DBObjectCodec.writeValue(DBObjectCodec.java:198) ~[mongo-java-driver-3.4.2.jar:?]

at com.mongodb.DBObjectCodec.encodeMap(DBObjectCodec.java:222) ~[mongo-java-driver-3.4.2.jar:?]

at com.mongodb.DBObjectCodec.writeValue(DBObjectCodec.java:198) ~[mongo-java-driver-3.4.2.jar:?]

at com.mongodb.DBObjectCodec.encodeIterable(DBObjectCodec.java:271) ~[mongo-java-driver-3.4.2.jar:?]

at com.mongodb.DBObjectCodec.writeValue(DBObjectCodec.java:200) ~[mongo-java-driver-3.4.2.jar:?]

at com.mongodb.DBObjectCodec.encode(DBObjectCodec.java:130) ~[mongo-java-driver-3.4.2.jar:?]

at com.mongodb.DBObjectCodec.encode(DBObjectCodec.java:61) ~[mongo-java-driver-3.4.2.jar:?]

at org.bson.codecs.BsonDocumentWrapperCodec.encode(BsonDocumentWrapperCodec.java:63) ~[mongo-java-driver-3.4.2.jar:?]

at org.bson.codecs.BsonDocumentWrapperCodec.encode(BsonDocumentWrapperCodec.java:29) ~[mongo-java-driver-3.4.2.jar:?]

at com.mongodb.connection.RequestMessage.addDocument(RequestMessage.java:253) ~[mongo-java-driver-3.4.2.jar:?]

at com.mongodb.connection.RequestMessage.addDocument(RequestMessage.java:205) ~[mongo-java-driver-3.4.2.jar:?]

at com.mongodb.connection.CommandMessage.encodeMessageBodyWithMetadata(CommandMessage.java:75) ~[mongo-java-driver-3.4.2.jar:?]

at com.mongodb.connection.RequestMessage.encodeWithMetadata(RequestMessage.java:160) ~[mongo-java-driver-3.4.2.jar:?]

at com.mongodb.connection.CommandProtocol.sendMessage(CommandProtocol.java:186) ~[mongo-java-driver-3.4.2.jar:?]

at com.mongodb.connection.CommandProtocol.execute(CommandProtocol.java:111) ~[mongo-java-driver-3.4.2.jar:?]

at com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:168) ~[mongo-java-driver-3.4.2.jar:?]

at com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:289) ~[mongo-java-driver-3.4.2.jar:?]

at com.mongodb.connection.DefaultServerConnection.command(DefaultServerConnection.java:176) ~[mongo-java-driver-3.4.2.jar:?]

at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:216) ~[mongo-java-driver-3.4.2.jar:?]

at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:187) ~[mongo-java-driver-3.4.2.jar:?]

at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:179) ~[mongo-java-driver-3.4.2.jar:?]

at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:92) ~[mongo-java-driver-3.4.2.jar:?]

at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:85) ~[mongo-java-driver-3.4.2.jar:?]

at com.mongodb.operation.CommandReadOperation.execute(CommandReadOperation.java:55) ~[mongo-java-driver-3.4.2.jar:?]

at com.mongodb.Mongo.execute(Mongo.java:836) ~[mongo-java-driver-3.4.2.jar:?]

at com.mongodb.Mongo$2.execute(Mongo.java:823) ~[mongo-java-driver-3.4.2.jar:?]

at com.mongodb.DB.executeCommand(DB.java:729) ~[mongo-java-driver-3.4.2.jar:?]

at com.mongodb.DB.command(DB.java:491) ~[mongo-java-driver-3.4.2.jar:?]

at com.mongodb.DB.command(DB.java:507) ~[mongo-java-driver-3.4.2.jar:?]

at com.mongodb.DB.command(DB.java:462) ~[mongo-java-driver-3.4.2.jar:?]

at org.springframework.data.mongodb.core.MongoTemplate$3.doInDB(MongoTemplate.java:406) ~[spring-data-mongodb-1.10.0.RELEASE.jar:?]

at org.springframework.data.mongodb.core.MongoTemplate$3.doInDB(MongoTemplate.java:404) ~[spring-data-mongodb-1.10.0.RELEASE.jar:?]

at org.springframework.data.mongodb.core.MongoTemplate.execute(MongoTemplate.java:462) ~[spring-data-mongodb-1.10.0.RELEASE.jar:?]

at org.springframework.data.mongodb.core.MongoTemplate.executeCommand(MongoTemplate.java:404) ~[spring-data-mongodb-1.10.0.RELEASE.jar:?]

at org.springframework.data.mongodb.core.MongoTemplate.aggregate(MongoTemplate.java:1572) ~[spring-data-mongodb-1.10.0.RELEASE.jar:?]

at org.springframework.data.mongodb.core.MongoTemplate.aggregate(MongoTemplate.java:1501) ~[spring-data-mongodb-1.10.0.RELEASE.jar:?]

[Jeff Yemin]

I suspect that you need support for aggregation cursors.  The relevant Spring Data MongoDB Jira issue is https://jira.spring.io/browse/DATAMONGO-1637.

Regards,

Jeff

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/4c8f27d0-5754-4dfa-890e-50d3cfdb1922%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.