BSON Performance query

[Abhijit Bhopale]

Hi Team,

I am using protocol buffer as a serialization protocol but found that it is very slow in case of bulk data with nested structures. serialization & De-serialization is even slower than XML based serialization .

I have query for BSON how efficient it in in case of bulk data (400MB to 700MB size) having nested structures compared to protocol buffer?

[Dwight Merriman]

it should be quite fast to serialize & de-serialize.  not sure what programming language you are using.

the buffer will be larger as in bson the field names and data types for each field are in the encoding.  if you look at bsonspec.org you can see the encoding spec and from that get a feel for how large the documents will be.

700MB of data is fine but note that should consist of many bson documents (which could be appended all together in a single file) -- many of the bson encoders (depends on language) are non-streaming and assume a bson document fits in ram.  In MongoDB there is a 16MB limit per document but that is database-related and not a bson limit per se.

also note there is an xml to bson converter at https://github.com/dwight/bsontools.  it is very beta , but needs beta testers if you want to try it.

[Abhijit Bhopale]

Thank you Dwright.

I am using C++(GNU 4.1.2 ) with VXWorks 6.8 and other end would be  C# .Net 4.5 (Windows 7). Any suggestion which BSON library & version should be used?

Need to use memory stream to store serialized data on vXworks side but wont be streaming through network. Network bandwidth is not an issue more concerned about the the serialization & De-serialization time. I guess xml to bson converter would not be useful in this scenario. 

[Dwight Merriman]

you can use the bson libraries in the  MongoDB C++ and C# driver.  if not a database problem just don't bother with the rest of the driver.

[sandy]

Hi Abhijit,

Did you manage to implement BSON? I also have a requirement of serialising/deserialising 700MB of JSON. Would you be able share your experience on response times.

Which Library did you use?

Thanks

Sandy