Inserting lots of documents exceeds 16MB size limit

[Tomer Froumin]

When I use bulk insert to add a lot of new documents and the total size exceeds 16MB I get an error and I see a traceback in the MongoDB logs stating it received a request that exceeds the maximum BSON size.

The same thing happens when I try to update existing documents, in my case there are several hundred documents of various size (several KB each).

I could reproduce the error on several versions of reactivemongo, 0.12.2 and 0.12.7.

In 0.12.2 there was a max size parameter but it seems to be ignored, and the interface changed in 0.12.7 and there is no limit check.

Example code to reproduce:

val coll: BSONCollection = db.collection("mycoll")
val bsonList = List(BSONDocument(...), BSONDocument(...), ...)
// 0.12.2
coll.bulkInsert(bsonList.toStream, true)
// 0.12.7
coll.insert(ordered = false).many(bsonList)

And the MongoDB output:

2017-11-08T11:06:13.780+0000 I -        [conn1889] Assertion: 10334:BSONObj size: 47309568 (0x2D1E300) is invalid. Size must be between 0 and 16793600(16MB) First element: update: "myCollection" src/mongo/bson/bsonobj.cpp 58

2017-11-08T11:06:13.790+0000 I CONTROL  [conn1889]

 0x5611cacc9bc1 0x5611cac54a17 0x5611c9f648e5 0x5611c9f649c4 0x5611c9ffe798 0x5611ca2add40 0x5611ca3d23d9 0x5611ca3d44a6 0x5611c9fd340d 0x5611c9fd3d4d 0x5611cac2fc32 0x7f373c99b064 0x7f373c6d062d

----- BEGIN BACKTRACE -----

{"backtrace":[{"b":"5611C9753000","o":"1576BC1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"5611C9753000","o":"1501A17","s":"_ZN5mongo10logContextEPKc"},{"b":"5611C9753000","o":"8118E5","s":"_ZN5mongo23msgassertedWithLocationEiPKcS1_j"},{"b":"5611C9753000","o":"8119C4","s":"_ZN5mongo30msgassertedNoTraceWithLocationEiPKcS1_j"},{"b":"5611C9753000","o":"8AB798","s":"_ZNK5mongo7BSONObj14_assertInvalidEv"},{"b":"5611C9753000","o":"B5AD40","s":"_ZN5mongo9DbMessage9nextJsObjEv"},{"b":"5611C9753000","o":"C7F3D9"},{"b":"5611C9753000","o":"C814A6","s":"_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE"},{"b":"5611C9753000","o":"88040D","s":"_ZN5mongo23ServiceEntryPointMongod12_sessionLoopERKSt10shared_ptrINS_9transport7SessionEE"},{"b":"5611C9753000","o":"880D4D"},{"b":"5611C9753000","o":"14DCC32"},{"b":"7F373C993000","o":"8064"},{"b":"7F373C5E8000","o":"E862D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.4.4", "gitVersion" : "888390515874a9debd1b6c5d36559ca86b44babd", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-97-generic", "version" : "#120-Ubuntu SMP Tue Sep 19 17:28:18 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "b" : "5611C9753000", "elfType" : 3, "buildId" : "59518566540C4D32B0F28CDEADF506ADAECCE6D1" }, { "b" : "7FFD97751000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "015BE691E41FD883340407166A01E630241297AD" }, { "b" : "7F373D8CF000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "21115992A1F885E1ACE88AADA60F126AD9759D03" }, { "b" : "7F373D4D3000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "32E9A5B9EED626E93DEEB00A49033F78652DB9A3" }, { "b" : "7F373D2CB000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "E74B9E72FAD3C15B664D6F83CCFEC1463D7CC138" }, { "b" : "7F373D0C7000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "D0F3172357FA7D0D7E7E531F4AA2843F9EB66EBD" }, { "b" : "7F373CDC6000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "10CFE72872EF5AAECDA534C4D5ABC27649149A83" }, { "b" : "7F373CBB0000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "D5FB04F64B3DAEA6D6B68B5E8B9D4D2BC1A6E1FC" }, { "b" : "7F373C993000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "B0550E19F1DFB30EB1F14BA360444EFFDF64609A" }, { "b" : "7F373C5E8000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "8DBCBCA713CA58E2C7E4970E8218D536154BB64E" }, { "b" : "7F373DB30000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "29E98A1FB7D8AE233FC360FEEFA38E93638C3D42" } ] }}

 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x5611cacc9bc1]

 mongod(_ZN5mongo10logContextEPKc+0x177) [0x5611cac54a17]

 mongod(_ZN5mongo23msgassertedWithLocationEiPKcS1_j+0x1C9) [0x5611c9f648e5]

 mongod(_ZN5mongo30msgassertedNoTraceWithLocationEiPKcS1_j+0x0) [0x5611c9f649c4]

 mongod(_ZNK5mongo7BSONObj14_assertInvalidEv+0x208) [0x5611c9ffe798]

 mongod(_ZN5mongo9DbMessage9nextJsObjEv+0x110) [0x5611ca2add40]

 mongod(+0xC7F3D9) [0x5611ca3d23d9]

 mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x6D6) [0x5611ca3d44a6]

 mongod(_ZN5mongo23ServiceEntryPointMongod12_sessionLoopERKSt10shared_ptrINS_9transport7SessionEE+0x1ED) [0x5611c9fd340d]

 mongod(+0x880D4D) [0x5611c9fd3d4d]

 mongod(+0x14DCC32) [0x5611cac2fc32]

 libpthread.so.0(+0x8064) [0x7f373c99b064]

 libc.so.6(clone+0x6D) [0x7f373c6d062d]

-----  END BACKTRACE  -----

2017-11-08T11:06:13.791+0000 I -        [conn1889] AssertionException handling request, closing client connection: 10334 BSONObj size: 47309568 (0x2D1E300) is invalid. Size must be between 0 and 16793600(16MB) First element: update: "myCollection"

2017-11-08T11:06:13.791+0000 I -        [conn1889] end connection 10.244.1.50:50398 (208 connections now open)

[Cédric Chantepie]

Hi,

Both .insert and .bulkInsert are covered by the tests, so to go further it would need a reproducer *with* fixtures.