New array type proposal

[Claudio Bisegni]

Hi, I have a proposal for a new array type to bson spec. Now array are a bson document list. Image if I need to encode hundred or thousand of same elements (also structured) the write and read need to pass to write and read of full bson document for every elements. In high performance environments will be usefull to store one time the array elements structure and then store sequentially the value of the structured data. In high energy physics experiment this situation is the rule. Keep I mind for example a sample of signal. I can have a structure of two element, double for value and integer for time. An efficient way to store this into a new array type in bson is needed. Can we discuss this upgrade? I don't want go out the standardization.

Thanks in advanced

[Dwight Merriman]

agree this should be discussed when a bson spec rev occurs

[Claudio Bisegni]

Hi Dwight,
i agree with you, and i have also other idea. I hope the spec rev will occur soon. Any information about that?

[Dwight Merriman]

soon would be good to start talking about a revision

i think great care should be taken with the mod as specs shouldn't have new versions very often.  but it's time to start talking about the next rev.

in the meantime you may find BinData useful as a workaround for the array case you mention.  obviously, some limitations in what you can do that way though.

[Claudio Bisegni]

Yes now we use BinData sub type to achieve that. Anyway we are focusing all our control system on bson e we need to have a very fast and useful serialization format. Another idea is to separate description for the key and the serialized data. In this case in our environment we can get all serialization the first time(data+description) after that also the serialized data is need(to get new value). this is the idea.

[StephanC]

The redundancy in BSON arrays, where the key (field name) is repeated for each element, is an efficiency problem for many possible applications in science, where often some kind of sensor or sensor array gathers huge amounts of data. A relatively simple idea might be to add as a subtype "strings" of all the basic types. The present string covers the basic type "byte." One could add as subtypes a "double string" as well as an "int32 string" and an "int64 string" which contain sequences of double, int32, and int64, respectively. This would not be as general as fully structured arrays, but much easier to add in a revision of BSON (and MongoDB).