MongoDB Cursor exceeds size limit - query

[PB]

Hi, I'm trying to runt this code, but I always get document size exception, when querying elements not in a list:

I also tried aggregation but I get a similar document size exception.

MongoClient mongoClient = new MongoClient(host, port);

MongoCollection<Document> col = mongoClient.getDatabase(db).getCollection(collection;

 

int offset = 0;

int limit = 10000;

BasicDBObject whereQuery = new BasicDBObject();
whereQuery.put(FIELD1,"FIELD1");
whereQuery.put(FIELD2, "FIELD2");
whereQuery.put(FIELD3, "FIELD3");
whereQuery.put(FIELD4 new BasicDBObject("$nin", list));

Set<String> set= new HashSet<>();

while (offset < totalCount) {

        List<Document> documents =                 col.find(whereQuery).skip(offset).limit(limit).into(new ArrayList<Document>());

         for (Document doc : documents) {

               set.add(doc.getString(field));

            }

                          offset += limit;

   }

//process last 

List<Document> documents = col.find(whereQuery).skip(offset).limit(limit).into(new ArrayList<Document>());

        for (Document doc : documents) {

               set.add(doc.getString(field));

            }
mongoClient.close();

 

 

When this line of code " col.find(whereQuery).skip(offset).limit(limit).into(new ArrayList<Document>()); " runs I get this error:

 

 

org.bson.BsonMaximumSizeExceededException: Document size of 18339335 is larger than maximum of 16793600. at org.bson.BsonBinaryWriter.validateSize(BsonBinaryWriter.java:418) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.BsonBinaryWriter.backpatchSize(BsonBinaryWriter.java:412) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.BsonBinaryWriter.doWriteEndArray(BsonBinaryWriter.java:153) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.AbstractBsonWriter.writeEndArray(AbstractBsonWriter.java:350) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connection.LevelCountingBsonWriter.writeEndArray(LevelCountingBsonWriter.java:78) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonArrayCodec.encode(BsonArrayCodec.java:85) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonArrayCodec.encode(BsonArrayCodec.java:37) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.EncoderContext.encodeWithChildContext(EncoderContext.java:91) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonDocumentCodec.writeValue(BsonDocumentCodec.java:136) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:115) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:41) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.configuration.LazyCodec.encode(LazyCodec.java:37) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.EncoderContext.encodeWithChildContext(EncoderContext.java:91) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonDocumentCodec.writeValue(BsonDocumentCodec.java:136) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:115) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:41) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.configuration.LazyCodec.encode(LazyCodec.java:37) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonDocumentWrapperCodec.encode(BsonDocumentWrapperCodec.java:60) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonDocumentWrapperCodec.encode(BsonDocumentWrapperCodec.java:29) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.EncoderContext.encodeWithChildContext(EncoderContext.java:91) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonDocumentCodec.writeValue(BsonDocumentCodec.java:136) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:115) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:41) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connection.RequestMessage.addDocument(RequestMessage.java:238) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connection.RequestMessage.addDocument(RequestMessage.java:188) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connection.CommandMessage.encodeMessageBodyWithMetadata(CommandMessage.java:136) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connection.RequestMessage.encode(RequestMessage.java:138) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connection.CommandMessage.encode(CommandMessage.java:57) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:244) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connection.UsageTrackingInternalConnection.sendAndReceive(UsageTrackingInternalConnection.java:99) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connection.DefaultConnectionPool$PooledConnection.sendAndReceive(DefaultConnectionPool.java:444) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connection.CommandProtocolImpl.execute(CommandProtocolImpl.java:72) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:200) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:269) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:131) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:123) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:242) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:233) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:136) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.operation.FindOperation$1.call(FindOperation.java:701) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.operation.FindOperation$1.call(FindOperation.java:695) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:457) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:401) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.operation.FindOperation.execute(FindOperation.java:695) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.operation.FindOperation.execute(FindOperation.java:83) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:179) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.client.internal.MongoIterableImpl.execute(MongoIterableImpl.java:132) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.client.internal.MongoIterableImpl.iterator(MongoIterableImpl.java:86) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.client.internal.MongoIterableImpl.forEach(MongoIterableImpl.java:110) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.client.internal.MongoIterableImpl.into(MongoIterableImpl.java:122) ~[mongo-java-driver-3.8.0.jar:na]

 

 

[Robert Cochran]

Hi!

I think you need to explain four (4) things:

1. What version of MongoDB are you running this query on?

2. What operating system are you running MongoDB on?

3. Can you show a few documents, such that the structure of each can be understood.

4. Can you explain in writing the exact goal of the query? What are you attempting to solve -- and how do you want to solve it, in words rather than code?

Thanks

Bob

[PB]

Hi Bob,

Answers to your questions:

1.

MongoDB server version: 4.0.0, spring-data-mongodb :2.0.8.RELEASE and mongo-java-driver: 3.8.0

 Spring version  5.0.7.RELEASE

2.

Windows OS

3.

My Mongo Database has a collection with 173 666 558 documents. Each of these documents has the following structure:

{ 

    "_id" : ObjectId("5b57527773ce360ce0ed4575"), 

    "field6" : "00:00:00", 

    "field7" : "00:00:00", 

    "field5" : "0000000000000000000", 

    "field4" : "3822", 

    "field8" : "3", 

    "field1" : "xxxxx",l

    "field2" : "1.0", 

    "field3" : "5002", 

    "_class" : "clazz"

}

4.

I have a String list with 967 989 elements. 

I want to know which documents in my collection (depicted in 3. ) have field1, field2 and field3 equal to FiELD1, FIELD2 and FIELD3 query parameters  and also field4 equal to a value not present in my list with 967989 elements also given as a query parameter.

[Robert Cochran]

Hi!

Thank you for posting these details. They are very helpful.

I believe the list you refer to of 967 989 elements is causing the query document itself to become larger than 16 megabytes in size. That in turn is causing your query to fail.

I'm not sure how to create a query that will satisfy your requirements. You may have to split your one list into several smaller lists and find some way of processing these lists in a progression of steps. 

Now that I've ventured my opinion of what the issue is, let me also say that I am not very experienced with MongoDB queries. So I could be giving you incorrect information. I could be entirely wrong.

You might want to enclose the query in a try...catch...finally block to ensure that the database connection is closed even if the query fails. 

Thanks so much

Bob

[Ramon Fernandez]

Please take a look at the Java Driver Quick Start Guide, which has some useful information that should help you here, specially around using cursors.

Cheers,

Ramón.

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/61476e6e-fd3e-4498-affc-cd5087aebcd0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PB]

Hi Bob,

Thank you for your answer.

Regarding the 16Mb limit, I agree with you on the size of the query parameter list, it has more than twice the size of 16 Mb and that is probably the cause. 

If I run the query by steps partioning the list, I will always get documents with field4 not present in the sublists.

So for now I still am stuck with this.

For managing database connections I use try-with-resources. 

[PB]

Hi Ramón,

Thanks, I had a look but did not help.

cheers

[Robert Cochran]

Hi PB, 

In the sample document you describe above, are all the fields except the _id field strings? Or are they some other data format, e.g. NumberDecimal, ISODate, etc.?

Thanks

Bob

[paulo Leite]

Hi Bob,

Yes, all the fields are of String type,

Regards,

PB

--

You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+unsubscribe@googlegroups.com.

To post to this group, send email to mongod...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/0c36e579-add7-4cff-a21f-ab4be5b2ca5b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Cumprimentos,

Paulo Leite

[Robert Cochran]

Hi PB,

So all the fields in each document occur in the order shown and with the names that are shown, is that correct? For example, after _id, we see "field6", "field7", "field5"... in that order?

Thanks

Bob

To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.

To post to this group, send email to mongod...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/0c36e579-add7-4cff-a21f-ab4be5b2ca5b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Cumprimentos,

Paulo Leite

[Wan Bachtiar]

Regarding the 16Mb limit, I agree with you on the size of the query parameter list, it has more than twice the size of 16 Mb and that is probably the cause.

Hi PB,

Based on the error message, this is the reason why org.bson.BsonBinaryWriter.validateSize() is throwing BsonMaximumSizeExceededException. The total size of your query is greater than the maximum BSON size allowed to be serialized and sent via the wire.

I would suggest to reconsider your data model, and reassess how the application interacts with the data.

Alternatively, you could store the list of Field4 into another collection, and utilise $lookup to perform a left outer join between the two collections.

Regards,
Wan.

​