Voltron Cryptocoin library

[Brooks Boyd]

The issue I've struggled with, with most of the cryptocoin binaries (including the canonical bitcoind application, and the bitcoinjs-server NodeJS implementation I've worked with the most) is how monolithic they are; they include mining, wallet, listening server and sending server, JSON-RPC API, etc. Even if you only need a portion of those features, you must deal with the overhead of having all the other components there. And if you're trying to find bugs/memory leaks or trying to tweak it to work with another cryptocoin, there's a lot more code to wade through.

While the source code for those applications usually splits the source out into different files that do their own roles, my goal is to create a set of libraries in more the UNIX mentality of "a small application that does one thing and does it well", and then bundling little applications together into a piecemeal whole that's exactly what you need. I've started to build this in NodeJS (chosen since it has the potential then to be deployable to desktop, server, or run in-browser), and have these three libraries as proof-of-concept so far:

crypto-p2p-node: Manages a single connection to a single peer. Its primary purpose is to replace the 'data' events a bare Node socket emits, and instead collect the chunked data from the peer and break it up by magic byte sequence, and instead emit 'message' events when a whole message is received. Emits 'message' events so you can subscribe to all messages, and custom events per command in the form of "commandMessage" (e.g. "versionMessage", "verackMessage", "getaddrMessage", etc.) to subscribe to messages only of that type. Works with any network communication protocol that has the same envelope structure as Bitcoin (4 byte magic byte identifier, 12 byte command string, 4 byte payload length, 4 byte checksum, then payload).

crypto-p2p-manager: Manages several crypto-p2p-node connections. It maintains lists of active peers, a pool of known peers to potentially connect to, and weeds out bad peers who are unresponsive on a basic level. Provides functions for sending messages to specific sub-sets of active peers.

btc-p2p: Knows how to parse the message payloads of Bitcoin protocol messages, emitted by a crypto-p2p-manager. It ensures Bitcoin protocol rules like sending a VERSION message right after connecting, and sending a VERACK after receiving a VERSION message are followed. It uses the GETADDR/ADDR messages to feed new potential peers to the crypto-p2p-manager pool. This will emit events for new blocks and transactions being transmitted for some parent library to assemble (a blockchain-maintaining library) and store (database).

What do you all think of that structure/methodology? I put several examples in the three libraries (especially crypto-p2p-node) on how they could easily be flexed to conform to other cryptocurrency schemes. That way, there doesn't need to be a monolithic configuration file at the top level of the application that tries to reach deep into the application to change the logic to another cryptocurrency (the way that namecoin has a "namecoin.cpp" source file that overrides the defaults set by the "main.cpp" Bitcoin file which is still in its source). There could be a "ltc-p2p" module that applies the Litecoin logic, and then Litecoin apps get build on that library, rather than building on the btc-p2p library and overriding methods.

Even without the block and transaction relaying, I realized one simple application could already be built on the btc-p2p library as-is: a seed/bootstrap peer distribution app, like a DNS lookup server, for peers wanting to get in the network. Once I add the block and transaction relaying ability to the btc-p2p library, I plan to work towards a website service like blockexplorer/blockchain.info. Anyone else have interests in this direction or interest in helping create modified versions of these libraries for other cryptocurrencies? Also, the names aren't anything I'm stuck on; anyone got any better naming convention for these little components?

Brooks

[Sidney Zhang]

Hey Brooks,

I agree with you 100%.

First of all, I definitely like the idea of working towards a service like blockexplorer/blockchain.info. Partially, that's what I have been working on which is an blockchain data API so that developers have a thin http interface into the blockchain without having to host their own node etc. 

Secondly, I definitely would like to get involved in the library, out of interest to learn more about the network protocol. Although I always have reservations about alternative node implementation. I want to refer to this article and the discussions between peter todd and mike hearne

https://bitcointalk.org/index.php?topic=260595.0

Even coinbase had issues with chain splitting as they were using a alternative node implementation.

"This will emit events for new blocks and transactions being transmitted for some parent library to assemble (a blockchain-maintaining library) and store (database)." <- I feel like a safe way of doing this is to have a process that reads from bitcoind and relays information through bitcoind. (I am actually working on an implementation of this in Golang). This is also how blockexplorer and bitcoin-abe does it. Instead of directly connecting to peers, the process can make calls to bitcoind to get latest blocks, mempool etc and therefore maintain a database and we can also make RPC calls to relay transactions through bitcoind.

I am not 100% set on my opinion regarding this. Although I am definitely cautious. I want to get your thoughts on this. Do you think its safer to write a process that talks to bitcoind rather than something to connects directly to peers?

[Brooks Boyd]

Regarding the "reimplementing the protocol" I have mixed feelings on it; placing that much reverence on the bitcoind implementation as The Canonical Implementation and in that sense goes against the theme of decentralization the Bitcoin morality generally leans to. The discussions in that thread talk about unintentional invalidation of blocks, leading to unintentional forks away from the "main" chain as the primary reason against. From a technical perspective I completely agree; how and when a block is considered valid and how and when a side chain is considered dead and a reorganization takes place should be in the fundamental definition of the cryptocurrency. However, I don't think how that should be defined is "the right way is the way bitcoind does it", but rather explicitly spelled out in a developer's guide. Then yes, new implementations should be able to be tested against a bitcoind node (fully-sync a bitcoind node, then cut it off from the network, and have it only talk to a peer using the new implementation. Does the new implementation sync up properly?)

I'm reminded of a few months back when there was much murmuring in the community about the possibility that the core developers would add "blacklisting" of "tainted coins" to the bitcoind implementation. The murmuring then was generally along the lines of "if you don't like it, don't use the update, and the chain will hard-fork". If how and when a transaction/block is accepted is variable and the definition of the right way is "how bitcoind does it", then situations like that become an issue; is the "right" way to do it "how version 123 of bitcoind does it" or "version 456 of bitcoind does it"? If the logic of verification does need to change, the change should be like "after block 1234, this new logic applies", not "when version 1234 of bitcoind gets released, this new logic applies".

From an implementation perspective, I'm looking for alternatives to running bitcoind alongside a web service for ease-of-depoyment. Since bitcoind is a compiled application, if deploying your web service to a new architecture, you have to first figure out if you need to recompile that binary. Also, it requires another port to be open on the server for the JSON-RPC calls to flow, even if you're only talking to yourself and not allowing other communications. Running bitcoind (because it's monolithic) forces downloading of the blockchain (AFAIK), and while storage is cheap if you're running on your own hardware, most virtual machine slices for web services have only a few GB of storage, so that becomes an issue. All those reasons are an issue if bitcoind is on the same server as your webservice; you could run it on a separate server, which would alleviate some of the issues (if you want to move your application to new hardware, it might be able to still point to your trusted bitcoind node from its new home), but becomes a more complex system if dealing with two different processes on two different servers (harder for not-as-experienced developers to manage if that level of complexity is required from get-go), and more complex to scale (if you need to add virtual servers for your web service to handle its load you also need a plan for when you need to add more bitcoind nodes to handle your own front-end nodes' traffic).

So, overall, I agree with the idea of "don't trust huge amounts of money to a non-standard implementation", and rule that most of the ideas I have for how these alternate implementations would work would be for non-wallet purposes. If there are many blockexplorer/blockchain.info-style services out there, the better to verify with! If there's an application that makes it easier to make a side database of known peers, so make deploying a new DNS seeding service easier, we'll get a more robust network around the globe. If there's implementations that only focus on a small area of the implementation (i.e. just the forking/reorganizing logic), it makes it easier to unit-test and experiment outside the main network, and perhaps discover bugs in the bitcoind logic. Right now, I do agree that your own wallet and funds are safest if they're part of a bitcoind node on running on your own hardware. However, I'd like to work towards the future where developers of alternate implementations can unit-test against each other and bitcoind, and there's a known/tested setup for running two different implementations side-by-side and watching for forking errors between them. Forking differences I think are one of the most fatal errors to the system, since it invalidates everyone's funds (which fork is "true"?). Errors that spend all a user's funds because a decimal place moved suck for that one user, and need to be fixed, but aren't fatal to the system overall. Are there any other problem areas like hard forks that you'd consider fatal if alternate implementations fell into?

[JP Richardson]

I'm with both of you on the desire for an OSS blockchain.info service. A community backed effort is needed and I think that Sidney has great start in this area.

Brooks you have an excellent point in regards to the existing reverence towards bitcoind. I'm reminded of a statement that Andreas Antonopoulos made on one of his most recent Let's Talk Bitcoin podcasts, something along the lines of fixing or changing Bitcoin is akin to replacing the software on a 10 billion dollar Boeing 747; you can't mess it up or people die (lose money). He goes on to say that it's critical to make the important changes now as that same airplane will be worth 100 billion dollars or more, thus mistakes are more costly in the future. How does this relate to what we're doing? There is a real opportunity here to build these modules and services and grow a community from them. There is old Chinese proverb that says: "The best time to plant a tree was 20 years ago. The second best time is now." Clearly no one in this discussion is asking whether it makes sense to do this now or later; rather, what I'm suggesting is that we can't wait for the core Bitcoin team to take action. You've identified problems in the bitcoind software, problems that I agree with. Yes, there is that risk that a fork could happen. But learning by failure and experience can be great. What I really like is your idea of building these small components that other developers can plug into their apps and build on those. You're doing it in such a way too that can benefit altcoins as well. 

As far the structure of your code goes, I think that it's a good start. I think that as you start to use it, the proper structure will start to emerge. Let me know if you think that I can help in some way.

[Brooks Boyd]

Excellent! Since you like this overall structure, should I add these three libraries to the "cryptocoinjs" organization and publish them under that organization's banner? What do you think about their naming? 

[JP Richardson]

Ya Brook, bring 'em over :) Regarding the name, I think the "crypto-" prefix is redundant, but other than that, the naming is good :)

[JP Richardson]

*Brooks (Sorry, typo in your name!)

[Brooks Boyd]

Okay, done! I agree the name is redundant given the name of the org; I had named them that way to differentiate the ones that were specific to a particular (alt)coin, and ones that could be configured for any, but given the name of the org, people could probably assume if it's not namespaced with a coin name, it's probably universal/configurable.

[Sidney Zhang]

Guys, Awesome google hangout today.

I am going to start working on the Javascript Blockparser. I am very excited to be working on all these networking stuff and a bit sad that now I have to write javascript instead of Go hahaha

I will hollar at your guys when I get something going!

[Brooks Boyd]

Indeed; great meeting, and I've posted some updates to the repos in line with moving to Buffers rather than (Typed) Arrays:

I updated "binstring" to be using Buffers by default. It works as before, but now the default output is a Buffer; do you think that should count as a "major" version change?

I created "crypto-hashing" as an example of how a Crypto/Hash library would look, to provide a common interface for the hashing functions. It uses binstring to coerce any input into a Buffer, and then outputs the requested format, making the "options" argument the same structure in binstring and crypto-hashing. More algorithms could easily be added as named functions that call the doCrypto() function for their logic.

And I went back and made the examples in "p2p-node" a little more robust: I added the DNS lookup functionality, so if the first peer doesn't respond, it tries again until it finds a live peer. It also monitors for a connection event and hangs up on peers who don't answer a connection attempt, or don't sent a VERACK after a VERSION message. Now you don't have to keep re-running the example script until you find a live server; it will do that for you!

I've started working on transaction processing on the "getdata" branch of btc-p2p if you want to take a peek at that or contribute there, before deciding how to incorporate it into the main branch.

I was thinking about our goals of creating various types of nodes other than "full" nodes, and I realized we should probably come up with a document/guide outlining what basic functionality a node of any type is expected to do (do initial handshake, respond to certain messages, relay certain transactions, etc.), and then start using the "services" bitflags that are part of the VERSION message to mean different functionality (bitcoind only uses one flag for "is a node"). I'll create that document somewhere; I'm thinking part of the Wiki for the btc-p2p repo, since that library is meant to implement the absolute baseline functionality to be a bitcoin node; anything else on top of that will be additional services. What do you think of that?

Brooks

[JP Richardson]

Regarding "binstring", no it should be a minor version change since the API is still unstable (< 1.0.0). On the "crypto-hashing" library, just glanced over it... I want to let the idea sink in a bit; we'll also need to make sure that https://github.com/dominictarr/crypto-browserify has the functionality that we need if we go the "crypto-hashing" route. I'm also in favor of the necessity of a document outlining the basic functionlaity for these crypto currency nodes; I think the btc-p2p wiki should would work well.

--
You received this message because you are subscribed to the Google Groups "Bitcoin Hackers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to BitcoinHacker...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Follow me on Twitter: http://twitter.com/jprichardson

[Brooks Boyd]

I've started a document for peer requirements here: https://github.com/cryptocoinjs/btc-p2p/wiki/Peer-requirements

Let me know what you think and feel free to make modifications!

Brooks

[Sidney Zhang]

Hey Brooks,

I looked through the code today. Look good! Good job! I am getting this when I run node app.js in examples under btc-p2p

/Users/alumn/projects/bitcoinmafia/cryptocoinjs/btc-p2p/lib/BTCNetwork.js:93

    self.manager.launch(seeds);

        ^

TypeError: Cannot read property 'manager' of undefined

    at BTCNetwork.launch (/Users/alumn/projects/bitcoinmafia/cryptocoinjs/btc-p2p/lib/BTCNetwork.js:93:9)

    at /Users/alumn/projects/bitcoinmafia/cryptocoinjs/btc-p2p/examples/app.js:54:5

    at asyncCallback (dns.js:68:16)

    at Object.onanswer [as oncomplete] (dns.js:147:9)

[Brooks Boyd]

Oops, I fixed that in the "getdata" branch, but not in master. I just pushed a fix to the github repository; do a "git pull origin" and that should be fixed.

Thanks!

[Sidney Zhang]

Nice working now.

I am going to write a module to parse various binary data. That should come handy!

[Brooks Boyd]

After playing with this a bit, I think I've settled on an object structure that works well for parsing and handling of messages, by separating them into two separate methods. There's now a "parseVersionMessage" method and a "handleVersionMessage" method on the BTCNetwork object. The first changes the raw binary Buffer into a parsed JSON object, and the second applies that learned data to the Peer that sent it, doing other logic in the app. Separating out the parsing functionality allowed me to write some unit tests for it as well.

To accomplish that, I did create a separate object for analyzing binary data and attempting to get structure out of it (named Struct). Its a buffer with a pointer, and has methods that match the names of Buffer native methods, but before it does a read, it looks ahead to make sure the buffer is long enough to get the structure requested. It also auto-increments the pointer, so you can not waste more time keeping track of how many bytes you just moved in the buffer. The Struct object emits "error" events when it has an issue, and since those can happen anywhere in the parsing, I added a try/catch block to catch those errors and abort further parsing. However that's not the best way to do it (since I'm not checking to see if the exception comes from the Struct object indicating it's out of data, or a fatal exception from Node for some other reason). I think it's possible to use Node Domains for this, but have to do more tinkering. 

Once I get that domain setup working, and add in the parsing/handling of Block messages, I think that side branch will be set to merge back into the main branch as a new version, and can start working on apps on top of it; namely the blockchain and database pieces!

Brooks

[Brooks Boyd]

Okay, I've published the updated binstring as v0.2.0 now. Do you have any further thoughts about how the "crypto-hashing" just provides a consistent interface for the hashing functions?