MongoDb aggregation 16MB buffer size limit

[Mukesh Kumar]

Hey guys,

Please tell me how can we resolve this issue, we are using aggregation and size of docs become greater than 16MB. We also tried "AllowDiskUse = true" but still it's throwing same error.

We are using php to fetch data greater than 16MB. Let me know if you needed more info.

[Asya Kamsky]

Change your aggregation so you are not constructing documents bigger than 16MBs. 

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/3f3c7ec5-8ca9-4bda-8b58-2e101ab2981a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Asya Kamsky
Lead Product Manager
MongoDB
Download MongoDB - mongodb.org/downloads
Free MongoDB Monitoring - cloud.mongodb.com
Free Online Education - university.mongodb.com
Get Involved - mongodb.org/community
We're Hiring! - https://www.mongodb.com/careers

[Mukesh Kumar]

Hey,

We can't change aggregation, as data will grow, of-course we will have more data, we need some solution for this, not change of aggregation.

On Thursday, December 24, 2015 at 12:33:56 PM UTC+5:30, Asya Kamsky wrote:

Change your aggregation so you are not constructing documents bigger than 16MBs. 

On Thursday, December 24, 2015, Mukesh Kumar <mukesh.m...@gmail.com> wrote:

Hey guys,

Please tell me how can we resolve this issue, we are using aggregation and size of docs become greater than 16MB. We also tried "AllowDiskUse = true" but still it's throwing same error.

We are using php to fetch data greater than 16MB. Let me know if you needed more info.

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+unsubscribe@googlegroups.com.

To post to this group, send email to mongod...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/3f3c7ec5-8ca9-4bda-8b58-2e101ab2981a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tim Hawkins]

Can you not use $out stage to write the output data to a seperate collection.

On Dec 24, 2015 3:49 PM, "Mukesh Kumar" <mukesh.m...@gmail.com> wrote:

Hey,

We can't change aggregation, as data will grow, of-course we will have more data, we need some solution for this, not change of aggregation.

On Thursday, December 24, 2015 at 12:33:56 PM UTC+5:30, Asya Kamsky wrote:

Change your aggregation so you are not constructing documents bigger than 16MBs. 

On Thursday, December 24, 2015, Mukesh Kumar <mukesh.m...@gmail.com> wrote:

Hey guys,

Please tell me how can we resolve this issue, we are using aggregation and size of docs become greater than 16MB. We also tried "AllowDiskUse = true" but still it's throwing same error.

We are using php to fetch data greater than 16MB. Let me know if you needed more info.

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.

To post to this group, send email to mongod...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/3f3c7ec5-8ca9-4bda-8b58-2e101ab2981a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Asya Kamsky
Lead Product Manager
MongoDB
Download MongoDB - mongodb.org/downloads
Free MongoDB Monitoring - cloud.mongodb.com
Free Online Education - university.mongodb.com
Get Involved - mongodb.org/community
We're Hiring! - https://www.mongodb.com/careers

--

You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.

To post to this group, send email to mongod...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/f2f62484-64a1-404d-9df2-f10eb38c68f8%40googlegroups.com.

[Frederick Cheung]

If you change the output options so that output is written to a collection rather than returned inline you should be able to have >16MB (as long as the individual documents are < 16 MB)

Fred

[Asya Kamsky]

You can claim that or you can provide more details and we can show you how you need to change aggregation. 

Your documents in the collection are smaller than 16MBs - the error means you are doing something in the aggregation that is creating these giant documents.  Fix your aggregation. 

Data will grow means your have more documents. Everyone's data grows. 

Asya

On Thursday, December 24, 2015, Mukesh Kumar <mukesh.m...@gmail.com> wrote:

Hey,

We can't change aggregation, as data will grow, of-course we will have more data, we need some solution for this, not change of aggregation.

On Thursday, December 24, 2015 at 12:33:56 PM UTC+5:30, Asya Kamsky wrote:

Change your aggregation so you are not constructing documents bigger than 16MBs. 

On Thursday, December 24, 2015, Mukesh Kumar <mukesh.m...@gmail.com> wrote:

Hey guys,

Please tell me how can we resolve this issue, we are using aggregation and size of docs become greater than 16MB. We also tried "AllowDiskUse = true" but still it's throwing same error.

We are using php to fetch data greater than 16MB. Let me know if you needed more info.

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.

To post to this group, send email to mongod...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/3f3c7ec5-8ca9-4bda-8b58-2e101ab2981a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Asya Kamsky
Lead Product Manager
MongoDB
Download MongoDB - mongodb.org/downloads
Free MongoDB Monitoring - cloud.mongodb.com
Free Online Education - university.mongodb.com
Get Involved - mongodb.org/community
We're Hiring! - https://www.mongodb.com/careers

--

You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.

To post to this group, send email to mongod...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/f2f62484-64a1-404d-9df2-f10eb38c68f8%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Asya Kamsky]

This is wrong - you cannot write out a document larger than 16 MBs. 

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/7403a1d2-b17c-49c2-86d8-d812ad12a8eb%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Tim Hawkins]

I think folks are talking about "inline" aggregation, where the results of the aggregation are returned as a single document, which is I belive the default behaviour.

If the OP is not using a "cursor" element argument to the aggregation function, or has not specified an '$out' collection, then it is my understanding that total size of the result set is restricted to the size of a single document, ie 16mb, I'm not sure that he is suggesting that the pipeline is emitting individual documents bigger than 16mb.

I may be wrong, but thats what i read from his problem description.

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CAOe6dJCXzDjuCiJndJm%3D9piuAZ1yixdKgt2yyAgABVCdCuoBjA%40mail.gmail.com.

[Asya Kamsky]

I don't think inline has been a default for a while - at least since 2.6 a cursor is returned.

We wouldn't have to guess if the poster told us what version of MongoDB they are running (or if they provided more details about what aggregation they are actually sending to the server).

Asya

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CAEbsasTWQLRFHwKvwomCzf6K2%3DHTO-1pPzn3ATtKPkQtr%2BZNyg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Tim Hawkins]

The legacy php driver documentation still shows that the aggregate command returns an array (document),  the mongodb 3.0  docs suggest that the default return of a cursor,  is a helper function behavior,  its possible that in php you have to request a cursor, as in pre 2.6 shell behaviour.

http://php.net/manual/en/mongocollection.aggregate.php

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CAOe6dJD1yHkyQn6hs%2BB3OmhF3tWqshPJpjwFQq9WM0P3H-eNCQ%40mail.gmail.com.

[Asya Kamsky]

Tim,

You may very well be right - if only there was someone on this list who knew definitively what the OP meant!    :)

Asya

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CAEbsasTJ1PgEg6MMH_7fdG4jty0zknUQv%2Bo%2BVTH2Ga4%2BcBuHjg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Tim Hawkins]

Ok,

Digging further

The standard command for aggregation in php is MongoCollection::aggregate,  which does just return a document.

However since version 1.5.0 of the php driver, a new function was added MongoCursor::aggregateCursor  which returns a cursor. I must admit that i have been using aggregation a lot in php and i have never noticed the cursor varient.

So if the OP just changes from ->aggregate($pipeline) to ->aggrigateCursor($pipeline)  and then iterates the resulting cursor, than they should be good to go.

http://php.net/manual/en/mongocollection.aggregatecursor.php

I think it is overlooked because all the tutorial use the pre 1.5.0 driver version.

I must admit i spent a bunch of time trying to add a "cursor" option to the aggrigation options array, and could not get it to work. I suspect most others woukd have expected it to work that way.

As per

https://jira.mongodb.org/browse/PHP-1165

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CAOe6dJB-W01s8Fh4rdSeqyMGP95fA5P1i2CjzBXr4kT-U6CBmQ%40mail.gmail.com.

[Mukesh Kumar]

Thanks guys for the update.
I am using Mongodb v2.6, And php driver version is- 1.6.11

To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+unsubscribe@googlegroups.com.

[Asya Kamsky]

Have you tried switching to using MongoCursor::aggregateCursor  as Tim suggested?

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/2a908c49-2267-486e-9612-00f07e83aff5%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Mukesh Kumar]

Hi Asya,

Yes I tried aggregateCursor, but it throws error "Cannot use object of type MongoCommandCursor as array" .

Please let me know if you need code snippet.

To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+unsubscribe@googlegroups.com.

[Jeremy Mikola]

On Wed, Dec 30, 2015 at 2:26 AM, Mukesh Kumar <mukesh.m...@gmail.com> wrote:

Hi Asya,

Yes I tried aggregateCursor, but it throws error "Cannot use object of type MongoCommandCursor as array" .

This may be due to your existing code that handled the MongoCollection::aggregate() return value, which was a single command result document returned as a PHP array (with inline aggregation results in the "result" field). The aggregateCursor() method instead returns an iterable MongoCommandCursor, which you'll want to work with similar to the standard MongoCursor returned by a find() query.

The examples in each doc pages I linked above should illustrate this in more detail.

[Mukesh Kumar]

Hi Jeremy,

Can you please elaborate more. I have given below the query which we use in php.

 $query = [
                    [
                        '$match' => [
                            '$and' => [
                                ['status' => [
                                    '$nin' => $not_order ]
                                ],
                                ['purchase_date' => [
                                    '$gte' => $startDate, '$lte' => $endDate]
                                ]
                            ]
                        ]
                    ],
                    [
                        '$match' => [
                            '$or' => [
                                ['seller_details.state' => null]
                            ]
 
                        ]
                    ]
                ];
 
                 $options = array("allowDiskUse" => true);
 
                return $listing->aggregateCursor($query, $options);
 
            });

Please update what we need to change.

[Jeremy Mikola]

On Mon, Jan 4, 2016 at 2:04 AM, Mukesh Kumar <mukesh.m...@gmail.com> wrote:

Hi Jeremy,

Can you please elaborate more. I have given below the query which we use in php.

                return $listing->aggregateCursor($query, $options);

That line is returning a MongoCommandCursor, which you must iterate on. The aforementioned error you shared ("Cannot use object of type MongoCommandCursor as array") hints that you're attempting to access an array offset on the return value instead. For instance, if you saved that return value to $retval, you may have code doing:

foreach ($retval["result"] as $document)

If $retval is now a MongoCommandCursor, you should be using:

foreach ($retval as $document)

Please review the documentation examples for aggregate() and aggregateCursor() to see how to properly work with each return value.

[Asya Kamsky]

This isn't the actual aggregation you are using, is it?

Because it's not really aggregating anything - it's doing a match and that can just be done with a find.

Asya

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/67061950-c56f-45c3-8332-af389674d9a9%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.