Size limit to updates?

[dunno]

Hi,

I have a large update which clocks in at >120MBs. Yes, I understand
that the max document size is 16MB, but in my case, any WIP/
intermediate or final document isn't actually that large. The reason
for the update "bloat" is that I'm using dot notation on everything:

$set: {
Asset.SceneNodes.0x1234567890abcdef.Metadata.Subdata.
0xfedcba0987654321.LocalTransform.Position.X: 0,
Asset.SceneNodes.0x1234567890abcdef.Metadata.Subdata.
0xfedcba0987654321.LocalTransform.Position.Y: 0,
Asset.SceneNodes.0x1234567890abcdef.Metadata.Subdata.
0xfedcba0987654321.LocalTransform.Position.X: Z,
// ...ad nauseum
}

As you can see, the verbosity is quite high for what should amount to
very little final document size. The problem manifests as the MongoDB
C-driver's socket read receives a 0 from mongod.exe, thus closing the
socket. The mongod process still exists and responds to other sockets
(we have a connection pool going on), but the connection that the
update went through is dead now.

If you're wondering if I could use object notation, the answer is
probably a "no". To be able to update properly using object notation,
I'd need to know the before and after to calculate the diff (I need to
create an undo buffer out of this). And fetching the document from the
mongod process takes 5 seconds since it's an ~8MB document. If I just
set the field with the object I have, I stomp over potentially
existing data (stuff that I may not know about).

On the other hand, as for the option of breaking up the interior scene
nodes into individual documents...that's probably even worse as I then
have to update individual documents separately (each node gets
different data), which comes with a lot more overhead and eventual
latency.

My question is, has the buffer filled up and has mongod decided to
close the connection for good? Is there a variable somewhere where I
can jack this up and make a local patch? Since this is on the PC, I
don't care that much for larger temporary buffers (memory).

Alternatively, is there a way to force the update to interpret object
notation as dot notation? That would work as well.

Thanks for any help!

[Wes Freeman]

I know this isn't your question, but I'm a little confused how fetching an 8MB document (and saving it) is slower than sending a 120MB update.

Wes

--
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com
To unsubscribe from this group, send email to
mongodb-user...@googlegroups.com
See also the IRC channel -- freenode.net#mongodb

[dunno]

Good question. At the risk of sounding a little snarky, I don't get how fetching an 8MB document takes 5 seconds. I'm not crawling either, since I'm specifically grabbing the document by _id. That and everything should be able to fit in memory, since I can replicate this exact issue within the first 200MB of data that gets exchanged.

On the other hand, a 120MB push with the "fields" parameter set (which is normally tiny...until they undo...that's the next step) takes much less time than that.

Oh, and just as a note, the mongod process is running on the same computer as the client application. So it's all through the loopback device anyway...which is why I don't get how it takes 5 seconds to read 8MB of data.

mongodb-user+unsubscribe@googlegroups.com

[Wes Freeman]

Can you tell what it's doing for 5 seconds (watching mongostat, iostat, and top, or something)? Is it pegging a CPU core? The C driver looks like it just does a memcpy for finds. Maybe something isn't optimized for the loopback interface--small MTU or something (still shouldn't take that long, I know).

mongodb-user...@googlegroups.com

[dunno]

*sigh*

Thanks for bouncing ideas around. And you're right, it does look like there's a ginormous performance hit somewhere outside of the DB. And that's definitely the eventual "proper" fix we have to put in place, which is to pull the document down first to do the diff.

Now back to the original question: any hacks I can put into our local copy to get this working for the time being? =)

Thanks!

[Wes Freeman]

I've not tried this out, but the max size is set here: https://github.com/mongodb/mongo/blob/master/src/mongo/bson/util/builder.h

You probably should (as a hack) bump the max user size and buffer size up to 256 or something. Totally untested by me--you probably don't need to worry about other places in the code, though, because you're not doing $push to add to the object; it's just getting read in and processed by update.

Wes

mongodb-user...@googlegroups.com