Bitcoin Block Parser written in Golang

[Sidney Zhang]

I am starting to work on a block parser in Golang as a side exercise.

Does anyone have experiences with working on block parsers? Any resources that you would like to share before I begin?

[JP Richardson]

I haven't, no. But these guys have a lot of bitcoin/golang work completed: https://github.com/conformal So, that may help as a reference. You've probably already seen that though :)

[JP Richardson]

This is good: http://2.bp.blogspot.com/-DaJcdsyqQSs/UsiTXNHP-0I/AAAAAAAATC0/kiFRowh-J18/s1600/blockchain.png

[Sidney Zhang]

Sick. I have been reading that too. My gut feel is that it shouldn't be that hard with all these documentations and prior examples

[Brooks Boyd]

I have experience working with parsing binary files in general (Bitcoin messages/transactions/blocks, Minecraft NBT files, Myst/Riven Mohawk archives, etc.), and would be glad to help explain that if you need help getting started!

In general, parsing binary data is made easier by a "readable stream" sort of class that has an internal buffer of data, a pointer to where it is in the buffer, and the means to pull out different structures of data at the current pointer. Since Bitcoin has binary structures like "variable integers" and "variable strings", which don't always take up the same amount of space in the binary file, you need to read the stream in order to determine how far forward to skip after reading that structure.

For the NodeJS libraries I've been working with, Node has a Buffer class that's good for storing fixed-size data, but not for reading a stream like that. I made my own Message class for adding that functionality (https://github.com/MidnightLightning/crypto-p2p-manager/blob/master/examples/Message.js) if you want to take a peek at that.

Python has the Struct classes/methods to make binary data parsing easier, though I'm not sure what Golang has to work with by default.

[Sidney Zhang]

Hey Brooks.

Thanks for the response. I actually made some progress today and parsed my first 100k blocks. Although they are just sitting in memory.

What I wanted to ask wasn't actually about the parser itself. I wanted to talk to you about database schema to back this data. I am having a bit of dificulty understanding how this data could be stored in leveldb. It is a simple KV and has no indexes. I don't exactly understand how that could work.

Is there a schema graph that you could perhaps point me to?

[Brooks Boyd]

bitcoinjs-server uses a LevelDB data store, and stores the block and transaction data as a JSON-encoded text string as a value under the "key" of the hash of that object. Thus if you know the block or transaction hash, you just look up that key. For "indexes" in a K/V store, you have to add other pointers to that data. For bitcoinjs-server, you can also refer to blocks by height, so there's an additional entry with key of chain height, and value of the block hash. So it takes two lookups to look up a block by height.

In looking at some other organizations used for K/V stores, I started coming up with alternative schema ideas (https://github.com/MidnightLightning/bitcoinjs-server/blob/leveldb-fixing/doc/db-schema.md), focused on using LevelDB to get a range of values. If you use a namespace-character in the key of the entry, you can create a sort of hierarchy in the keys themselves. Then the body of the entry is more simple, and you could cherry-pick the data you need, if you only need, say, the block's difficulty bits, not the whole object.

[Sidney Zhang]

A small update. I have just finished my blockchain parser. I am going to start thinking about storage options in the next 2 days. Brooks, could I bother you and see if you might be free for a skype session? I am tossing up between leveldb and nosql. The structure your outlined (your leveldb schema) actually looks pretty good. I have been using leveldb at work recently and it seems blazing fast. So I am quite interested to try it.

[JP Richardson]

I've experimented with LevelDB in the past few weeks. It seems pretty cool. I wrote an article recently highlighting some of the cool things (specifically, RPC) that you can do with Node.js/LevelDB:http://procbits.com/2014/01/06/poor-mans-firebase-leveldb-rest-and-websockets What's really cool is the ecosystem of plugins that exist for it: https://github.com/rvagg/node-levelup/wiki/Modules 

[Sidney Zhang]

@jp, Leveldb does seem like that it would limit you to one api server per db which probably make launching nodes a bit hard?

[Brooks Boyd]

You could make a separate database server/application running a connection to a LevelDB back-end, something like what JP posted about in his article, using an RPC stream between two servers that only talk to each other, you could have several LevelDB-based database servers all talking to a load-balancer, and several front-end API servers talking to the load-balancer to get the data, if scaling up really big.