node-dht: a Distributed Hash Table (Kademlia) library with NAT-traversing super powers

[Bill Casarin]

Hey guys,

This is just a little something I have been tinkering with. I was trying to think of a good way to test node's capabilities and it turns out distributed hash tables work quite well in an evented environment like node. I found a great little library called libcage which is evented and was able to get it to live nicely within node's event loop.

The goal of node-dht is to provide a flexible library for building networked applications that can communicate via DHTs. node-dht can be used as a decentralized, peer-to-peer key value store with the ability to traverse NATs if needed. 

Each node in the DHT has a 160-bit hash associated with it. node-dht also has the ability to send and receive data to and from specific nodes (identified by the hash). All communication is done via UDP but there is also an API for reliable UDP (RDP) communication between nodes.

node-dht follows "the node way" and doesn't build any abstractions for you, by default the DHT is pretty dumb, it's up to the developer to build interesting services on top of node-dht.

Most of all node-dht breaks DHTs out of the blackbox that they are usually presented as, and provides a very clean way of interacting and experimenting with DHTs.

(pseudo) code!

var dht = require('dht');

// global nodes to enable NAT-traversal (optional)

var globalNode = dht.createNode(10000).setGlobal();

var globalNode2 = dht.createNode(10001).setGlobal();

var bobNode = dht.createNode(10002);

var aliceNode = dht.createNode(10003); 

// have global nodes join to form our bootstrap nodes

globalNode2.join("localhost", 10000, function (success) {

   // joined if success == true

  

});

// have bobNode join the network

bobNode.join("localhost", 10000, function (success) {

  // have aliceNode join the network after bob has joined

  aliceNode.join("localhost", 10000, function (success) {

     // put data into the network (k/v store)

     aliceNode.put(key, value, ttl)

     // send data to specific nodes

     aliceNode.send(bobNode.id, data)

  });

});

node-dht is currently in a very experimental stage and most of these feature are being implemented as we speak.

Let me know what you think!

Download: http://github.com/jb55/node-dht

[Astro]

Bill Casarin wrote:
> This is just a little something I have been tinkering with. I was trying to
> think of a good way to test node's capabilities and it turns out distributed
> hash tables work quite well in an evented environment like node. I found a
> great little library called libcage which is evented and was able to get it
> to live nicely within node's event loop.

That is an interesting project. However, I bet lines of code can be
reduced by implementing the protocol in JavaScript instead of an
external library that has to deal with asynchronicity itself.

> The goal of node-dht is to provide a flexible library for building networked
> applications that can communicate via DHTs. node-dht can be used as a
> decentralized, peer-to-peer key value store with the ability to traverse
> NATs if needed.

I browsed the NAT detection code a bit. Can you elaborate how it is
going to work in the end?

> Most of all node-dht breaks DHTs out of the blackbox that they are usually
> presented as, and provides a very clean way of interacting and experimenting
> with DHTs.

Do you have any particular use-cases in mind? DHT power comes with many
participants. Unless widespread use has been achieved somebody who wants
to use it for data-heavy projects could overload the network.

> Let me know what you think!

I discovered that Kademlia is very prone to NodeID spoofing and can thus
be used to facilitate DDoS attacks. Telehash.org has a straight-forward
solution to this: make NodeIDs immediately verifyable, ie. by creating
it from the SHA1 hash of a peer's IP+port number.

[Bill Casarin]

On Mon, Aug 16, 2010 at 11:51 AM, Astro <as...@spaceboyz.net> wrote:

Bill Casarin wrote:
> This is just a little something I have been tinkering with. I was trying to
> think of a good way to test node's capabilities and it turns out distributed
> hash tables work quite well in an evented environment like node. I found a
> great little library called libcage which is evented and was able to get it
> to live nicely within node's event loop.

That is an interesting project. However, I bet lines of code can be
reduced by implementing the protocol in JavaScript instead of an
external library that has to deal with asynchronicity itself.

I thought about this, currently node-dht's core is fairly large chunk of C++ code.
A javascript implementation with a smaller C++ core for hotspots would be an
interesting future project it people find this useful. I'm currently only binding to the
top level abstractions provided by libcage (key/value put/get, node send/recv, working
on the rdp send/recv). Internally there is some more interesting stuff going on like
dht tables and tunneling tables, I might expose these if it makes sense to do so.
 

> The goal of node-dht is to provide a flexible library for building networked
> applications that can communicate via DHTs. node-dht can be used as a
> decentralized, peer-to-peer key value store with the ability to traverse
> NATs if needed.

I browsed the NAT detection code a bit. Can you elaborate how it is
going to work in the end?

The author of libcage had this to say about it:

libcage try to detect the NAT type to access two nodes, which have global IP address.
So, there must be at least two nodes having global IP address on the DHT network.

If you plan to use libcage on the Internet of IPv4, please make prepare two nodes, which have global IP address as bootstrapping nodes when you deploy it.

Please use set_global() function, which set the state 'GLOBAL', for the two nodes.
Other nodes will determine 'GLOBAL' or 'NATTED' by communicating with two nodes whose states are 'GLOBAL'.

Of course, you can disable the detection of NATs to pass 'false' to the 3rd argument of cage::open().
If you use PF_INET6, which indicates IPv6, the detection is automatically disabled.

 

> Most of all node-dht breaks DHTs out of the blackbox that they are usually
> presented as, and provides a very clean way of interacting and experimenting
> with DHTs.

Do you have any particular use-cases in mind? DHT power comes with many
participants. Unless widespread use has been achieved somebody who wants
to use it for data-heavy projects could overload the network.

I thought a little about possible use cases. In my head I imagined a situation where you'd have
possibly thousands of nodes within and across datacenters, hiveminding them together to share
data and distribute messages. I mostly started it as en experiment to see what was possible.
 

> Let me know what you think!

I discovered that Kademlia is very prone to NodeID spoofing and can thus
be used to facilitate DDoS attacks. Telehash.org has a straight-forward
solution to this: make NodeIDs immediately verifyable, ie. by creating
it from the SHA1 hash of a peer's IP+port number.

I haven't really looked into how libcage hashes the node id, I'd image if it doesn't do this
it wouldn't be too hard implement. Thanks for the input!
 

--

You received this message because you are subscribed to the Google Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com.
To unsubscribe from this group, send email to nodejs+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/nodejs?hl=en.

[Eric Fong]

Hi

Want to know what will happen when data are larger than a physical
memory?

If it is fast enough, it will be good for share memory (memcached) for
multi-node in multi-machine.

NAT and P2P feature make it even better than memcached.

If add Bonjour, auto peer detection, that will even more usage.

Eric

> > nodejs+un...@googlegroups.com<nodejs%2Bunsu...@googlegroups.com>
> > .