kademlia-dht

kademlia-dht is a javascript, network-agnostic implementation of the
Distributed Hash Table
storage container abstraction, employing the
Kademlia algorithms and data
structures.

From a local point of view, a DHT is similar to a classic hash table. It
provides two main operations, set and get allowing you, respectively,
to store a key/value pair, and to retrieve a value from the key. The
"distributed" aspect comes from the fact the pairs are stored accross a
network of interconnected nodes (eg. over the Internet), making it suitable
to provide information to a large number of users. Typical usage of DHTs
include file-sharing (eg. Bitorrent).

kademlia-dht is implemented with Node.js, but does not depend on system
resources, like the network. Instead, the implementation of the network layer
(called by Kademlia the Remote Procedure Calls) is left to the user or
higher-level libraries. That means this DHT implementation is theorically
adaptable to the browser JS without too much hassle.

Note: this implementation is, for now partially complete. It basically
lacks two things:

* the handling of time-driven behaviors: key/value pairs expiration,
bucket refresh, replication, and pairs republish.
* an implementation of the Rpc usable out-of-the-box (it will be done as a
separate library) even though it's possible to use your own already;

Installation

npm install kademlia-dht

Example usage

This example creates two nodes on a mock network, stores a value on
one node, then get it on the other node.

``js
'use strict';

var kad = require('kademlia-dht');

// Store a value on one side and get it back on the other side.
//
function demo(dht1, dht2) {
dht1.set('beep', 'boop', function (err) {
dht2.get('beep', function (err, value) {
console.log('%s === %s', 'boop', value);
});
});
}

// Spawn a node. A node is composed of two elements: the local Dht and the Rpc.
//
function spawnNode(endpoint, seeds, cb) {
kad.MockRpc.spawn(endpoint, function (err, rpc) {
if (err) return cb(err);
kad.Dht.spawn(rpc, seeds, function (err, dht) {
if (err) return cb(err);
cb(err, dht);
});
});
}

spawnNode('localhost:9876', [], function (err, dht1) {
spawnNode('localhost:4321', [dht1.rpc.endpoint], function (err, dht2) {
demo(dht1, dht2);
});
});
`

Output:

boop === boop

MockRpc is provided for demonstration and tests. It works by simulating a
basic network of interconnected nodes. Every instance of MockRpc can send
messages to any other instances of the same node.js process. For a real
use case, you must implement your own Rpc or use a higher-level library. The
Rpc interface is explained below.

This example is available as example/demo.js.

Notes

A Dht cannot ensure a complete consistency of the returned data. The fact new
nodes arrive and other exit the network at any time causes variations in the
lookup algorithm: all nodes resulting from the lookup are not necessarily
up-to-date all the time. You must take into account this variability when
processing returned values.

Moreover, a Dht can hardly ensure the authenticity of the values. For instance,
a technique known as the Sybil attack allows attackers to cluster around
a specific key, accepting any store operation, but then returning a malicious
value or no value at all instead of the expected one. This kind of attack
is inherent to the structure of such a distributed system, even though some
tentative implementations try to counteract via trust-based systems.

The following documents have been the basis of this Kademlia implementation:

* http://pdos.csail.mit.edu/~petar/papers/maymounkov-kademlia-lncs.pdf
* http://xlattice.sourceforge.net/components/protocol/kademlia/specs.html

API

$3

#### new Dht(rpc, id)

Create directly the node from an Id. You should never use this directly.

#### Dht.spawn(rpc, seeds, callback)

rpc Rpc* The RPC object to use to send messages to the network.
seeds Array* An array of known endpoints (eg. IPs).
callback(err, dht) Function* Called when the Dht is ready.

Create a brand new Dht. It will try to connect to the seeds, and grab
knowledge about its nearest nodes. You cannot use a Dht that's not connected
to some network, hence the need for seeds.

The Dht takes ownership of the rpc, and it should not be used for anything
else in your application.

#### dht.set(key, value, callback)

key String*
value Any*
callback Function* Called with arguments (err) once the key/value pair
has been replicated onto the network.

#### dht.get(key, callback)

key String*
callback(err, value) Function* Called once the value is found.

If the key cannot be found on the network, this is not an error. The value
will merely be null.

#### dht.peek(key)

key String*

Return immediately the associated value if known locally. You can use this
method to avoid the callback overhead, but only very few key/value pair are
available with this technique.

#### dht.close()

Close the Dht and the inner node. No other operation is then possible.

$3

This describe the excepted interface for the rpc object provided to
Dht.spawn().

#### rpc.send(message, endpoint, payload, cb)

message String* Type of message, always one of: "ping", "store",
"findNode" or "findValue".
endpoint Any* A value representing the 'endpoint' of the node. Generally
it will be a pair {ipAddress, port}, when the Rpc is implemented over
TCP or UDP.
payload Object* An object containing information attached to the
message.
cb(err, result) Function* To be called once the message got a reply.

This function must implement a timeout mechanism, and callback with an error
if so happens. This ensures the Dht does not hang up on a request and properly
take into account unresponsive nodes.

#### rpc.receive(message, handler)

message String* Type of message to handle.
handler(payload)` Function* Function locally handling the specified
message.

Register a local handler.

The receive handler must be a synchronous operation. The handling function
returns a result object that must be forwarded to the node initiating the
request. The handler may throw, in which case an error should be forwarded to
the remote node; make sure, though, that no local information leaks such as
stacktraces, because they can contain personal or sensitive information.

#### rpc.close()

Close the Rpc endpoint and prevent any further call to message handlers.