Understanding the binary format

[Martin Ichilevici de Oliveira]

Hello,

I am trying to understand the structure of a BSON document generated by MongoDB/WiredTiger by comparing the hexdump with the formal specification (http://bsonspec.org/spec.html).

This question went unanswered in the mongodb-dev group, but in hindsight I think here is a more appropriate place.

I inserted the following into an empty mongo collection (without compression):

{"name": John, age: NumberInt(10)}

{"name": Paul, age: NumberInt(25)}

And here is the hexdump

$ hexdump -C collection-0-1541750112074217368.wt
00000000  41 d8 01 00 01 00 00 00  d8 08 23 b7 00 00 00 00  |A.........#.....|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00001000  00 00 00 00 00 00 00 00  26 04 00 00 00 00 00 00  |........&.......|
00001010  8a 00 00 00 04 00 00 00  07 04 00 00 00 10 00 00  |................|
00001020  f0 22 09 6b 01 00 00 00  05 81 bb 2e 00 00 00 07  |.".k............|
00001030  5f 69 64 00 5d fd 43 d4  53 6c 1e 93 e1 fb fb 1d  |_id.].C.Sl......|
00001040  02 6e 61 6d 65 00 05 00  00 00 6a 6f 68 6e 00 10  |.name.....john..|
00001050  61 67 65 00 0a 00 00 00  00 05 82 bb 2e 00 00 00  |age.............|
00001060  07 5f 69 64 00 5d fd 43  dc 53 6c 1e 93 e1 fb fb  |._id.].C.Sl.....|
00001070  1e 02 6e 61 6d 65 00 05  00 00 00 70 61 75 6c 00  |..name.....paul.|
00001080  10 61 67 65 00 19 00 00  00 00 00 00 00 00 00 00  |.age............|

[...]

I understand that the bytes underlined are the types and bytes in blue are the objectids, yellow are the names and green are the ages.

However, I do not understand what the bytes between them (in red) represent -- I know it is something binary, but what?

I also tried to parse that same file using libbson (using example-client.c). It connects to Mongo and retrives that collection. However, the data received is slightly different! Using gdb, I stepped through the parsing and I realized that those bytes were actually:

00 03 31 00 2e 00 00 00

Now, that I can understand: the first 00 is the end of the previous document, "03 31 00" is the beginning of the new document and "2e 00 00 00" is the document's size.

So what is the meaning of the binary data in Wired Tiger and how can I "convert it" to the format that libbson received?

Thanks,

Martin

[Alexander Gorrod]

Hi Martin

Thanks for your clear question and your interest in MongoDB and WiredTiger. The (annoying for you, I suspect) answer is that it isn't supported to decode the data directly from WiredTiger. WiredTiger isn't aware of the BSON object format - it is passed a binary blob of data from MongoDB which is stores and retrieves as requested. The WiredTiger data format will include additional information than the BSON data which means that it isn't possible to directly retrieve BSON from a WiredTiger file on disk. You need the associated metadata and index structures to do that (even disregarding the various compression schemes implemented in WiredTiger). The only reliably way to retrieve data is via the regular WiredTiger interface.

The level of detail you are looking for should be available in the BSON handling code in MongoDB, which can be found here:

https://github.com/mongodb/mongo/tree/master/src/mongo/bson

Are you trying to achieve something specific, or just a general understanding of the data format encoding?

- Alex

[Martin Ichilevici de Oliveira]

Hi Alex,

Thanks for your reply. I'm trying to build my own BSON parser as part of a research project, so I really need to understand the structure. Is the metadata included in the same collection*.wt file or is it somewhere else?

Can you point me to a few functions in that directory that do the reading/parsing of the metadata/actual data? Can I (initially) ignore the index*.wt file?

Thank you,

Martin

--
You received this message because you are subscribed to the Google Groups "wiredtiger-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wiredtiger-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wiredtiger-users/2c09257b-c007-4942-a29e-bbc50c1184a4%40googlegroups.com.

[Donald Anderson]

Hi Martin,

I don't know if this helps you, but there is a way to dump the individual key/value pairs that are stored in WiredTiger.  To achieve that you need to have the "wt" program, to get it you may have to build the corresponding WiredTiger release from scratch (see src/third_party/wiredtiger/README in your mongodb source tree).

If you have "wt" you can do this:

$ ../wt -h test_cursor03.1 list

colgroup:test_cursor

file:test_cursor.wt

table:test_cursor

$ ../wt -h test_cursor03.1 dump -x file:test_cursor.wt

WiredTiger Dump (WiredTiger Version 3.2.2)

Format=print

Header

file:test_cursor.wt

access_pattern_hint=none,allocation_size=4KB,app_metadata=,assert=(commit_timestamp=none,durable_timestamp=none,read_timestamp=none),block_allocation=best,block_compressor=,cache_resident=false,checksum=uncompressed,colgroups=,collator=,columns=,dictionary=0,encryption=(keyid=,name=),exclusive=false,extractor=,format=btree,huffman_key=,huffman_value=,ignore_in_memory_cache_size=false,immutable=false,internal_item_max=0,internal_key_max=0,internal_key_truncate=true,internal_page_max=4KB,key_format=S,key_gap=10,leaf_item_max=0,leaf_key_max=0,leaf_page_max=32KB,leaf_value_max=0,log=(enabled=true),lsm=(auto_throttle=true,bloom=true,bloom_bit_count=16,bloom_config=,bloom_hash_count=8,bloom_oldest=false,chunk_count_limit=0,chunk_max=5GB,chunk_size=10MB,merge_custom=(prefix=,start_generation=0,suffix=),merge_max=15,merge_min=0),memory_page_image_max=0,memory_page_max=5MB,os_cache_dirty_max=0,os_cache_max=0,prefix_compression=false,prefix_compression_min=4,source=,split_deepen_min_child=0,split_deepen_per_child=0,split_pct=90,type=file,value_format=S

Data

303030303030303030303030303030303030303000

303030303030303030303030303030303030303000

303030303030303030303432393439363732393600

303030303030303030303432393439363732393600

303030303030303030303835383939333435393200

303030303030303030303835383939333435393200

303030303030303030303835393031393637333600

303030303030303030303835393031393637333600

The outputs of wt dump (after the "Data" marker) are alternating key value pairs.  I've used the -x option to get hex output.  This particular example is not terribly interesting, keys and values match each other.  And no BSON here.  But if you dump a .wt collection produced by MongoDB, you'll see lots of raw BSON.  This gets around having to decode WT's own metadata and data format encoding.

- Don

To unsubscribe from this group and stop receiving emails from it, send an email to wiredtig...@googlegroups.com.

[Donald Anderson]

Martin, 

Sorry about the formatting in my last post, and don't know how to edit it to fix it up.  I wanted to point out that the "-h test_cursor03.1" command options represent the home directory for WiredTiger's data (basically where all the .wt files are, including WiredTiger.wt).  So in this example test_cursor03.1 is the name of the WiredTiger home directory.

Good luck!

- Don

[Martin Ichilevici de Oliveira]

Hi Donald,

Thanks for pointing me to the wt program, I did not know about it. I will investigate how it works and report back if I have any other questions.

Thanks,

Martin

--

You received this message because you are subscribed to the Google Groups "wiredtiger-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wiredtiger-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wiredtiger-users/38f2f620-cb04-4685-8eed-4d5c9d088b10%40googlegroups.com.

[G F]

Is there any example of how to access mongodb *.wt files of wired tiger storage as bson ?

Aside from the mongodb server code, where should I start looking in there for high speed access to mongo db ?.

[Donald Anderson]

Hello,

I'm not aware of examples that do this directly.  There are BSON decoders around, here's one I found via google : https://github.com/mongodb/js-bson .  I can't vouch for whether it works.  Anyway you could use the "wt dump" command as I described in a previous response and use its output with a BSON decoder you find.  If you don't want to use "wt dump", you can do what it does (it's pretty easy, it opens a cursor, gets each key/value pair and formats it).  It is in the WiredTiger source directory as src/utilities/util_dump.c .

- Don