weird arithmetic

[Shi Shei]

I got the follwing exception:

Exception in thread "main" com.mongodb.MongoInternalException:
DBObject of size 15888915 is over Max BSON size 16777216
at com.mongodb.OutMessage.putObject(OutMessage.java:147)
at com.mongodb.OutMessage._appendQuery(OutMessage.java:85)
at com.mongodb.OutMessage.query(OutMessage.java:44)
at com.mongodb.OutMessage.query(OutMessage.java:38)
at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:303)
at com.mongodb.DBCursor._check(DBCursor.java:369)
at com.mongodb.DBCursor._hasNext(DBCursor.java:498)
at com.mongodb.DBCursor.hasNext(DBCursor.java:523)

Does this mean that Mongo thinks that 15888915 is greater than
16777216?

[gregor]

Which version of the driver are you using?

[Shi Shei]

I'm using the Java driver v2.7.2

[Nat]

Are you using sharding/replication? Does it happen consistently? If not, it could be that you have mixed server versions.

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com.
To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.

[Shi Shei]

Yes, I'm using sharding (4 shards, each consisting of 3 nodes). The
mongod versions are 2.0.1, 2.0.2 and 2.0.4.
Should the mongod versions always be identical?

[Shi Shei]

And yes, it happens consistently.

[Nat]

Nope it doesn't have to. How about your mongos version?

[Shi Shei]

All mongos are 2.0.1

[Max Schireson]

I am guessing here which I hate to do but I wonder if that is showing the raw document size and with the padding factor for inserts it exceeds the maximum?

You can see what the padding factor is by running db.coll.stats - you may want to check that?

-- Max

[Shi Shei]

The padding factor is around 1.0 on 3 shards and 1.3 on one shard (see
below).
Using the "right" query, I get the following exception, where max BSON
size is greater than 2.6 times of the DBObject size:

Exception in thread "main" com.mongodb.MongoInternalException:

DBObject of size 6442515 is over Max BSON size 16777216

at com.mongodb.OutMessage.putObject(OutMessage.java:147)
at com.mongodb.OutMessage._appendQuery(OutMessage.java:85)
at com.mongodb.OutMessage.query(OutMessage.java:44)
at com.mongodb.OutMessage.query(OutMessage.java:38)
at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:303)
at com.mongodb.DBCursor._check(DBCursor.java:369)
at com.mongodb.DBCursor._hasNext(DBCursor.java:498)
at com.mongodb.DBCursor.hasNext(DBCursor.java:523)

mongos> db.offer.stats()
{
"sharded" : true,
"flags" : 1,
"ns" : "offerStore.offer",
"count" : 94043357,
"numExtents" : 251,
"size" : 182913750120,
"storageSize" : 283038796064,
"totalIndexSize" : 33238383360,
"indexSizes" : {
"_id_" : 3348210992,
"bokey_1" : 7212466576,
"categoryBokey" : 6988288832,
"clickCount" : 2180187632,
"mappedCatalogCategory" : 3768432864,
"missingSince" : 2873095456,
"onlineProductIds" : 2961935872,
"shopId" : 3905765136
},
"avgObjSize" : 1944.9938406601118,
"nindexes" : 8,
"nchunks" : 9494,
"shards" : {
"offerStoreDE2" : {
"ns" : "offerStore.offer",
"count" : 32521119,
"size" : 58552151128,
"avgObjSize" : 1800.434699925301,
"storageSize" : 76310707744,
"numExtents" : 66,
"nindexes" : 8,
"lastExtentSize" : 2146426864,
"paddingFactor" : 1.3199999992045253,
"flags" : 1,
"totalIndexSize" : 11378604608,
"indexSizes" : {
"_id_" : 1178930144,
"shopId" : 1295757008,
"mappedCatalogCategory" : 1227626400,
"missingSince" : 1076313168,
"clickCount" : 718089904,
"categoryBokey" : 2366886592,
"bokey_1" : 2562423808,
"onlineProductIds" : 952577584
},
"ok" : 1
},
"offerStoreDE3" : {
"ns" : "offerStore.offer",
"count" : 19263196,
"size" : 43881514208,
"avgObjSize" : 2277.9975974910913,
"storageSize" : 99921403248,
"numExtents" : 77,
"nindexes" : 8,
"lastExtentSize" : 2146426864,
"paddingFactor" : 1.0099999975193636,
"flags" : 1,
"totalIndexSize" : 7413522592,
"indexSizes" : {
"_id_" : 782426848,
"shopId" : 898910320,
"mappedCatalogCategory" : 846354992,
"missingSince" : 607182464,
"clickCount" : 518088592,
"categoryBokey" : 1543097360,
"bokey_1" : 1553873328,
"onlineProductIds" : 663588688
},
"ok" : 1
},
"offerStoreDE4" : {
"ns" : "offerStore.offer",
"count" : 24164368,
"size" : 48108572960,
"avgObjSize" : 1990.888938622355,
"storageSize" : 67725000288,
"numExtents" : 62,
"nindexes" : 8,
"lastExtentSize" : 2146426864,
"paddingFactor" : 1.009999998815975,
"flags" : 1,
"totalIndexSize" : 8557336816,
"indexSizes" : {
"_id_" : 799988896,
"shopId" : 1031361520,
"mappedCatalogCategory" : 1005320960,
"missingSince" : 710052896,
"clickCount" : 566752144,
"categoryBokey" : 1806503552,
"bokey_1" : 1812921712,
"onlineProductIds" : 824435136
},
"ok" : 1
},
"offerstore" : {
"ns" : "offerStore.offer",
"count" : 18094674,
"size" : 32371511824,
"avgObjSize" : 1789.007739183364,
"storageSize" : 39081684784,
"numExtents" : 46,
"nindexes" : 8,
"lastExtentSize" : 2146426864,
"paddingFactor" : 1.049999999691875,
"flags" : 1,
"totalIndexSize" : 5888919344,
"indexSizes" : {
"_id_" : 586865104,
"shopId" : 679736288,
"mappedCatalogCategory" : 689130512,
"missingSince" : 479546928,
"clickCount" : 377256992,
"categoryBokey" : 1271801328,
"bokey_1" : 1283247728,
"onlineProductIds" : 521334464
},
"ok" : 1
}
},
"ok" : 1

}



On Apr 19, 2:07 pm, Max Schireson <mschire...@gmail.com> wrote:
> I am guessing here which I hate to do but I wonder if that is showing the
> raw document size and with the padding factor for inserts it exceeds the
> maximum?
>
> You can see what the padding factor is by running db.coll.stats - you may
> want to check that?
>
> -- Max

[Nat]

It could be a bug. Did you reuse Mongo object? If not, can you reuse it. Also after you instantiate it, can you call Mongo.checkMaster() before making any insert.

> > mongodb-user+unsubscribe@googlegroups.com.

[Shi Shei]

I did not make an insert. It's just a read query of this type:
{_id:{$in:[1,2,3,4]}}

The error occurs when the inlist becomes too big.

> > > > mongodb-user...@googlegroups.com.

[Shi Shei]

When I reuse the mongo object, I don't get the exception.
The code before (without reuse of mongo object) stopped when I put
around 200000 id's in the inlist.

Now it works even when I put 1000000 id's into the list.

> > > > >        at com.mongodb.DBCursor._check(DBCursor.java:369)...
>
> read more »

[Shi Shei]

Exceeding the inlist size results in an appropriate exception:

Exception in thread "main" com.mongodb.MongoException: could not
initialize cursor across all shards because : combinatorial limit of
$in partitioning of result set exceeded @ offerStoreDE4/
s115:27018,s132:27018,s125:27018
at com.mongodb.MongoException.parse(MongoException.java:82)
at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:312)

at com.mongodb.DBCursor._check(DBCursor.java:369)
at com.mongodb.DBCursor._hasNext(DBCursor.java:498)
at com.mongodb.DBCursor.hasNext(DBCursor.java:523)

> > > > > > I got the follwing exception:...
>
> read more »

[Shi Shei]

So, how to cope with the issue, when the mongo object can't be reused,
i.e. you just want to query once?

> > > > >              ...
>
> read more »

[Nat]

Why can't you reuse Mongo object? If you don't reuse it, the connection pool will not be reused.

[Shi Shei]

For example when it's the first usage of the Mongo object. Then it's
not yet reused and will fail.

[Nat Luengnaruemitchai]

Can you file the bug report? I will submit a patch.

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.

[Shi Shei]

Done. Please see:
https://jira.mongodb.org/browse/SERVER-5662

On Apr 20, 2:26 am, Nat Luengnaruemitchai <nat.lu...@gmail.com> wrote:
> Can you file the bug report? I will submit a patch.
>

> ...
>
> read more »