aggregation result exceeds maximum document size (16MB)

Prasanna Ganesh

unread,

Mar 26, 2017, 1:22:39 AM3/26/17

to mongodb-user

Hi,

I have written a java program which takes distinct records from collection called 'A' and query another collection called 'B' which is a audit table in a batch of 1000 unique records from collection 'A'.

Currently i'm facing an issue in running below query on collection 'B' after executing certain number of batches.For example out of 134 batches to execute 133k records, it fails after 74 batches. Each batch contains set of 1000 records to query against collection B which might give 5000 recors atleast.

I understand there is size restrictions with aggregation query of 16MB and even mentioning allowDiskUse(true) in my query its failing with below exceptions. I saw in another post to use $out to resolve this issue but i'm facing difficulty in implementing it in my query because i want to return the result set as a list.

Really appreciate any help on this issue.

http://stackoverflow.com/questions/29644587/how-could-i-write-aggregation-without-exceeds-maximum-document-size

Query:

List<ReportsDocument> list = new ArrayList<ReportsDocument>();

TypedAggregation aggregation = Aggregation.newAggregation(ReportsDocument.class,
 match(query1),sort(Sort.Direction.ASC, "controlDocument.accountNumber")               
 ).withOptions(new org.springframework.data.mongodb.core.aggregation.AggregationOptions.Builder().allowDiskUse(true).build());


 AggregationResults<ReportsDocument> results = mongoTemplate.aggregate(aggregation,VCTCMongoCollections.REPORTING_HISTORY,ReportsDocument.class);

 list = results.getMappedResults();

return list;



 The full response is { "ok" : 0.0, "errmsg" : "aggregation result exceeds maximum document size (16MB)", "code" : 16389 }
        at org.springframework.data.mongodb.core.MongoTemplate.handleCommandError(MongoTemplate.java:2082)
        at org.springframework.data.mongodb.core.MongoTemplate.aggregate(MongoTemplate.java:1553)
        at org.springframework.data.mongodb.core.MongoTemplate.aggregate(MongoTemplate.java:1475)
        at com.visa.ip.vctc.utility.OneTimeReportNAB.computeOneTimeReport(OneTimeReportNAB.java:318)
        at com.visa.ip.vctc.utility.OneTimeReportNAB.run(OneTimeReportNAB.java:275)
        at com.visa.ip.vctc.utility.OneTimeReportNAB.main(OneTimeReportNAB.java:179)
Caused by: com.mongodb.MongoCommandException: Command failed with error 16389: 'aggregation result exceeds maximum document size (16MB)' on server xxxxx:27017. The full response is { "ok" : 0.0, "errmsg" : "aggregation result exceeds maximum document size (16MB)", "code" : 16389 }
        at com.mongodb.CommandResult.getException(CommandResult.java:80)
        at com.mongodb.CommandResult.throwOnError(CommandResult.java:94)
        at org.springframework.data.mongodb.core.MongoTemplate.handleCommandError(MongoTemplate.java:2076)
        ... 13 more

Wan Bachtiar

unread,

Apr 9, 2017, 11:20:23 PM4/9/17

to mongodb-user

com.mongodb.MongoCommandException: Command failed with error 16389: ‘aggregation result exceeds maximum document size (16MB)’

Hi Prasanna,

This is most likely due to the size of the aggregation query result itself exceeding the 16MB limit. The option allowDiskUse is only to enable the pipeline to write to temporary files during the stages. What you need is to use cursor in order to return the results in iterable manner.

If you’re using MongoDB driver v3.4, you can specify useCursor on the AggregateIterable object. for example:

AggregateIterable<Document> iterable = db.getCollection("collectionName").aggregate(aggPipeline).useCursor(true).allowDiskUse(true);

If you have further questions, please provide:

Specific MongoDB version that you’re using
Specific MongoDB Java version that you’re using

Regards,

Wan.

Prashant Patil

unread,

Nov 2, 2017, 6:01:34 PM11/2/17

to mongodb-user

Hi,

Attached is the screenshot of my eclipse environment, here you can see my jar files.

I am using Mono-java-driver 2.1.4.

we are using spring mongo driver . Even i am facing this issue. Can you please help me out with this?

Thanks,

Prashant.

Screenshot (4).png

Wan Bachtiar

unread,

Nov 3, 2017, 1:15:35 AM11/3/17

to mongodb-user

Can you please help me out with this?

Hi Prashant,

I’m not entirely clear if you’re having the same issue as the original question on this thread.
Could you please open a new discussion along with:

MongoDB Server version
The error message that you’re getting. (Actual error message, not eclipse user-interface screenshot)

Could you also clarify the version of MongoDB Java driver that you’re using ?
The earliest version that is available at http://mongodb.github.io/mongo-java-driver/ is v2.13.3 for mongo-java-driver.

Regards,
Wan.

Prashant Patil

unread,

Nov 3, 2017, 2:02:01 AM11/3/17

to mongodb-user

Hi Wan,

Thanks for the reply.

when i search for mongo in pom.xml i can see only the below dependency

<groupId>org.springframework.data</groupId>

<artifactId>spring-data-mongodb</artifactId>

<version>1.9.2.RELEASE</version>

</dependency>c

And in maven dependency in eclipse these are the jars related to mongo

mongo-java-driver-2.14.0-sources.jar

spring-data-mongodb-1.9.2.RELEASE-sources.jar

I am not really sure what are the differences between the above 2 jar files.

and can you please tell me how to check mongo server version ? i m new to this so m not understanding clearly.

It would be great help if you help me.

The error is :

com.mongodb.CommandFailureException: { "serverUsed" : "13.92.251.7:42525" , "ok" : 0.0 , "errmsg" : "aggregation result exceeds maximum document size (16MB)" , "code" : 16389 , "codeName" : "Location16389"}

at com.mongodb.CommandResult.getException(CommandResult.java:76)

at com.mongodb.CommandResult.throwOnError(CommandResult.java:140)

at com.mongodb.DBCollection.aggregate(DBCollection.java:1798)

at com.mongodb.DBCollection.aggregate(DBCollection.java:1778)

at com.flynava.jupiter.daoImpl.AnalystDaoImpl.getAnalystChannelRevenue(AnalystDaoImpl.java:237)

at com.flynava.jupiter.serviceImpl.AnalystServiceImpl.getAnalystChannelFare(AnalystServiceImpl.java:136)

at com.flynava.jupiter.controller.AnalystController.getAnalystChannelPie(AnalystController.java:46)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)

at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)

at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:114)

at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)

at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)

at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)

at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)

at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)

at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)

at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)

at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)

at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)

at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)