Bulk insert: maxBsonSize exceeded -- wrong max size

[Heikki Salokanto]

Using 0.12.7-SNAPSHOT.

Doing a bulk insert (1), I'm getting occasional:

reactivemongo.core.errors.GenericDriverException: 

  MongoError['size of document #19 exceed the maxBsonSize: 1295 > 1000']

Now, maxBsonSize should be 16777216. But there's another constant, maxBulkSize that, I think, is 1000. I think the purpose is to allow a bulk insert of at most 1,000 documents at a time, and at most 16 MiB each.

So I've got a feeling that maxBsonSize and maxBulkSize are getting mixed up somewhere, but couldn't easily find where.

I can file a bug if I haven't misunderstood how that should work.

(1)

database.collection[BSONCollection]("test_collection")

  .insert(ordered = false)

  .many(Seq(/* a few objects here, some of them larger than 1 kB */))

Cheers,

Heikki

[Cédric Chantepie]

The message should rather be about "remaining bson size", the size of the last document exceeding the remaining size for the current bulk.

Please first share a reproducer (test use case) to check.

The new bulk support (including update) is still in dev.

[Heikki Salokanto]

Hi Cédric,

I don't think I quite understand how maxBsonSize and maxBulkSize should work.

At the moment, it fails if any single document in the bulk exceeds 1000 bytes, which is a relatively nasty limitation. I think this was not the case in 0.12.3; at least I only started seeing these exceptions after upgrading to 0.12.7-SNAPSHOT.

See: https://gist.github.com/salokanto/bfd706149699d3f549aa5dc3a2264fcb

-Heikki

[Cédric Chantepie]

On Thursday, 5 October 2017 11:30:45 UTC+2, Heikki Salokanto wrote:

I think this was not the case in 0.12.3; at least I only started seeing these exceptions after upgrading to 0.12.7-SNAPSHOT.

As the new bulk support is under development with this 0.12.7-SNAPSHOT, that's different from any previous release.

 

See: https://gist.github.com/salokanto/bfd706149699d3f549aa5dc3a2264fcb

You can try the latest SNAPSHOT (after cleaning any IVY/Maven cache). The indicated test case doesn't raise exception with it.

[robert.w...@web.de]

Hi, got the same issue.

Seems to be a bug in 

reactivemongo.api.collections.UpdateOps at line 105 you will find :

...
 meta <- metadata
 maxSz <- maxBsonSize
 res <- {
 val bulkProducer = BulkOps.bulks(
 updates, meta.maxBulkSize, maxSz) { up =>
 elementEnvelopeSize + pack.bsonSize(up.q) + pack.bsonSize(up.u)
...

It looks like maxBsonSize and maxBulkSize got mixed up. I think it should be

 updates, maxSz, meta.maxBulkSize) { up =>

The same issue at reactivemongo.api.collections.InsertOps at line 94:

...
val bulkProducer = BulkOps.bulks(
    docs, bulkSz, maxSz) { pack.bsonSize(_) }
...

but it should be

val bulkProducer = BulkOps.bulks(
    docs, maxSz, bulkSz) { pack.bsonSize(_) }

I locally swapped it and tested it and this should fix it.

If you like I can check it in (would be my first tiny contribution).

Cheers, Rob.

[Cédric Chantepie]

That's what's is already fixed in the latest SNAPSHOT, so I suggest you check it's properly re-fresh from Sonatype SNAPSHOTS.

[Cédric Chantepie]

On Thursday, 5 October 2017 17:26:33 UTC+2, Cédric Chantepie wrote:

That's what's is already fixed in the latest SNAPSHOT, so I suggest you check it's properly re-fetch rom Sonatype SNAPSHOTS.

[robert.w...@web.de]

Looks like we overlapped. Perfect.

From which branch is the SNAPSHOT build, because I checked out the master and its not there yet.

Thanks.

[Cédric Chantepie]

That's not visible until the final PR is merged.