Custom index key

[Panagiotis Stalidis]

Hello all,

I am trying to build a new index type. This index uses a std::bitset for the key.
So far i have managed to make this work by transforming the bitset to a string back and forth but the process is very slow.
Can anyone help me with how i can use the bitset as a key without transforming it to a string?

thanks

[David Storch]

Hi Panagiotis,

It seems to me that if you want to serialize the std::bitset into a BSON index key, it would make the most sense to use the BinData type. This is simply a int32 giving the number of bytes followed by a byte array storing arbitrary data. See type 5 in the BSON spec here: http://bsonspec.org/spec.html.

Can you call std::bitset::to_ulong() or std::bitset::to_ullong() and then put the resulting bytes inside a BinData element?

Best,

Dave

[Panagiotis Stalidis]

Hi David,

unfortunately for me bitset::to_ulong() fails if the bitset is larger than ulong (i think its 32bits) and i am dealing with bitsets of 1024 or 2048 bits.

I have since substituted std::bitset with boost::dynamic_bitset and use boost::to_block_range which is more effective storage wise (not 1 byte for every bit) and a little faster.
Still the transformation back and forth costs more than 20x the key comparison and are a major bottleneck.

I will look into the BinData type, though I remember trying something like that and failing. I probably didn't build the BSON object correctly...

thanks for the suggestion,
Panagiotis

[David Storch]

Hi Panagiotis,

I see. Do you know why exactly the string conversion is so slow? Ideally you should be able to write code which simply copies the bytes of interest into a BinData BSONElement, by providing a pointer and a length. See BSONObjBuilder::appendBinData():

https://github.com/mongodb/mongo/blob/46cd31827e62a02e826b44f6129f06623e8f2f78/src/mongo/bson/bsonobjbuilder.h#L556-L573

Let me know if there's any other specific info I could provide which would be helpful.

Best,

Dave