dull - npm explorer

dull
====

Playing around with multilevel-http, node-hashring and express-swim
to build a riak-like distributed leveldb, any message passing between
cluster nodes is http-based.

``sh $ npm install -g dull $ mkdir -p data/node1 data/node2 data/node3 $ dull --port 3001 --path ./data/node1 $ dull --port 3002 --path ./data/node2 --join 127.0.0.1:3001 $ dull --port 3003 --path ./data/node3 --join 127.0.0.1:3001 $ curl -X PUT -d '{ "cap": { "n": 3 } }' http://localhost:3001/bucket/people $ curl http://localhost:3001/buckets/data $ curl -X PUT -d '{ "name": "Andrea", "lastname": "Gariboldi", age: 33 }' -H 'Content-Type: application/json' http://localhost:3002/bucket/people/data/andrea $ curl http://localhost:3002/bucket/people/data/andrea $ curl -X DELETE http://localhost:3002/bucket/people/data/andrea $ mkdir data/node4 $ dull --port 3004 --path ./data/node4 $ curl -X POST -d '127.0.0.1:3001' http://localhost:3004/gossip/join $ curl -X DELETE http://localhost:3004/gossip/leave $ curl http://localhost:3002/bucket/people/keys $ curl -X PUT --data-binary @examples/v8.png -H 'Content-Type: image/png' http://127.0.0.1:3001/bucket/people/data/v8.png`

`CLI arguments`

`$3`

--host (default: 127.0.0.1

): listen address.
*

--port (default: 3000

): listen port.
*

--path (default: ./data

): directory to store data files.
*

--timeout (default: 1000

ms): timeout for request between nodes.
*

--join (default: (empty)

): another node already in the cluster to join it.
$3

*

--cap.n (default: 3

): number of replicas for a given key.
*

--cap.r (default: 2

): number of replicas that should respond to a read request, for it to be considered successful.
*

--cap.w (default: 2

): number of replicas that should respond to a write request, for it to be considered successful.
$3

*

--vclock.small (default: 10

): number of entries to consider a vector clock small.
*

--vclock.big (default: 50

): number of entries to consider a vector clock big.
*

--vclock.young (default: 20

): max number of seconds to consider a vector clock entry young.
*

--vclock.old (default: 86400

): min number of seconds to consider a vector clock entry old.
$3

*

--ring.vnode_count (default: 40

): The amount of virtual nodes per server.
*

--ring.max_cache_size (default: 5000

): We use a simple LRU cache inside the module to speed up frequent key lookups, you can customize the amount of keys that need to be cached.
$3

*

--swim.verbose (default: false

): make the gossip protocol very verbose.
*

--swim.period_length (default: 3000

): period length (in milliseconds).
*

--swim.ping_timeout (default: 1000

): timeout of a ping request (in milliseconds).
*

--swim.failing_timeout (default: 9000

): timeout of a suspected state before failing a node (in milliseconds).
*

--swim.message_ttl (default: 10000

): the time a message should be kept in the message queue (in milliseconds).
*

--swim.pingreq_nodes (default: 2

): number of random nodes to select for a ping-req.
*

--swim.tune_gossip (default: 2

): tune maximum message retransmission (keep it "small").
*

--swim.gossip_messages (default: 10

): max piggybacked messages per request.
HTTP API
A node in dull is the string host:port.
$3

#### join a node to a cluster`curl -X POST -d http:///gossip/join`

#### make a node leave the cluster`curl -X DELETE http:///gossip/leave`

#### list active nodes in the cluster`curl http:///gossip/nodes`

`$3`

#### create or update bucket and options`curl -X PUT -d http:///bucket/`

#### delete a bucket`curl -X DELETE http:///bucket/`

#### list buckets`curl http:///buckets/keys curl http:///buckets/data curl http:///buckets/values`

`$3`

#### put a KV`curl -X PUT -d http:///bucket//data/`

this defaults to Content-Type: application/json. You can put any other value type like this:

`curl -X PUT -d -H 'Content-Type: ' http:///bucket//data/`

or

`curl -X PUT --data-binary @ -H 'Content-Type: ' http://127.0.0.1:3001/bucket//data/`

#### get a value`curl -v http:///bucket//data/`

you will get an header x-dull-vclock like this:

`x-dull-vclock: {"127.0.0.1:3002":2,"127.0.0.1:3001":1}`

that you should pass back updating a value like this:

`curl -X PUT -d -H 'Content-Type: ' -H 'x-dull-vclock: {"127.0.0.1:3002":2,"127.0.0.1:3001":1}' http:///bucket//data/`

those are used in read-repair.

#### delete a KV`curl -X DELETE http:///bucket//data/`

you will get an header x-dull-vclock like this:

`x-dull-vclock: {"127.0.0.1:3002":2,"127.0.0.1:3001":1}`

that you should pass back if you want to recreate the value like this:

`curl -X PUT -d -H 'Content-Type: ' -H 'x-dull-vclock: {"127.0.0.1:3002":2,"127.0.0.1:3001":1}' http:///bucket//data/`

Deleted objects are marked with a thumbstone record that may be in conflict with a put if you don't pass in a vector-clock. This thumbstone record is also used when generating keys: dull will return only non-deleted keys as one would expect, resolving those vector clocks. It will return also keys that has conflicts, so that the client may have the chance to decide it.

#### list all keys known to dull`curl http:///bucket//keys`

#### increment/decrement a pn-counter`curl -X PUT -d http:///counter//data/`

#### get the counter value`curl http:///counter//data/`

`$3`

#### x-dull-clientid

Used to identify the actor in vector clocks, if no actor is passed in an uuid is used.

#### x-dull-vclock

used to pass vclocks between client and the server: the server will return a vector clock in read or delete of a key, that should be used to modify the key.

#### x-dull-thumbstone

it is used to identify a deleted key sibling on a 303 response.

`$3`

You can check multilevel-http

#### Buckets`curl http:///buckets/`

#### Data`curl http:///mnt//``

Things to implement

$3

http://basho.com/why-vector-clocks-are-hard/
* vclock header compression

$3

http://vimeo.com/43903960
https://github.com/aphyr/meangirls
http://docs.basho.com/riak/latest/dev/using/data-types/

$3

https://github.com/aphyr/jepsen

$3

http://en.wikipedia.org/wiki/Merkle_tree
https://github.com/c-geek/merkle

$3

leveldb range of keys to push to a node when it comes back

CLI arguments

$3

(default: 127.0.0.1

): listen address.
*

--port (default: 3000

): listen port.
*

--path (default: ./data

): directory to store data files.
*

--timeout (default: 1000

ms): timeout for request between nodes.
*

--join (default: (empty)

): another node already in the cluster to join it.
$3

*

--cap.n (default: 3

): number of replicas for a given key.
*

--cap.r (default: 2

): number of replicas that should respond to a read request, for it to be considered successful.
*

--cap.w (default: 2

): number of replicas that should respond to a write request, for it to be considered successful.
$3

*

--vclock.small (default: 10

): number of entries to consider a vector clock small.
*

--vclock.big (default: 50

): number of entries to consider a vector clock big.
*

--vclock.young (default: 20

): max number of seconds to consider a vector clock entry young.
*

--vclock.old (default: 86400

): min number of seconds to consider a vector clock entry old.
$3

*

--ring.vnode_count (default: 40

): The amount of virtual nodes per server.
*

--ring.max_cache_size (default: 5000

): We use a simple LRU cache inside the module to speed up frequent key lookups, you can customize the amount of keys that need to be cached.
$3

*

--swim.verbose (default: false

): make the gossip protocol very verbose.
*

--swim.period_length (default: 3000

): period length (in milliseconds).
*

--swim.ping_timeout (default: 1000

): timeout of a ping request (in milliseconds).
*

--swim.failing_timeout (default: 9000

): timeout of a suspected state before failing a node (in milliseconds).
*

--swim.message_ttl (default: 10000

): the time a message should be kept in the message queue (in milliseconds).
*

--swim.pingreq_nodes (default: 2

): number of random nodes to select for a ping-req.
*

--swim.tune_gossip (default: 2

): tune maximum message retransmission (keep it "small").
*

--swim.gossip_messages (default: 10

): max piggybacked messages per request.
HTTP API
A node in dull is the string host:port.
$3

#### join a node to a cluster`curl -X POST -d http:///gossip/join`

#### make a node leave the cluster`curl -X DELETE http:///gossip/leave`

#### list active nodes in the cluster`curl http:///gossip/nodes`

`$3`

#### create or update bucket and options`curl -X PUT -d http:///bucket/`

#### delete a bucket`curl -X DELETE http:///bucket/`

#### list buckets`curl http:///buckets/keys curl http:///buckets/data curl http:///buckets/values`

`$3`

#### put a KV`curl -X PUT -d http:///bucket//data/`

this defaults to Content-Type: application/json. You can put any other value type like this:

`curl -X PUT -d -H 'Content-Type: ' http:///bucket//data/`

or

`curl -X PUT --data-binary @ -H 'Content-Type: ' http://127.0.0.1:3001/bucket//data/`

#### get a value`curl -v http:///bucket//data/`

you will get an header x-dull-vclock like this:

`x-dull-vclock: {"127.0.0.1:3002":2,"127.0.0.1:3001":1}`

that you should pass back updating a value like this:

`curl -X PUT -d -H 'Content-Type: ' -H 'x-dull-vclock: {"127.0.0.1:3002":2,"127.0.0.1:3001":1}' http:///bucket//data/`

those are used in read-repair.

#### delete a KV`curl -X DELETE http:///bucket//data/`

you will get an header x-dull-vclock like this:

`x-dull-vclock: {"127.0.0.1:3002":2,"127.0.0.1:3001":1}`

that you should pass back if you want to recreate the value like this:

`curl -X PUT -d -H 'Content-Type: ' -H 'x-dull-vclock: {"127.0.0.1:3002":2,"127.0.0.1:3001":1}' http:///bucket//data/`

#### list all keys known to dull`curl http:///bucket//keys`

#### increment/decrement a pn-counter`curl -X PUT -d http:///counter//data/`

#### get the counter value`curl http:///counter//data/`

`$3`

#### x-dull-clientid

Used to identify the actor in vector clocks, if no actor is passed in an uuid is used.

#### x-dull-vclock

used to pass vclocks between client and the server: the server will return a vector clock in read or delete of a key, that should be used to modify the key.

#### x-dull-thumbstone

it is used to identify a deleted key sibling on a 303 response.

`$3`

You can check multilevel-http

#### Buckets`curl http:///buckets/`

#### Data`curl http:///mnt//``

Things to implement

$3

http://basho.com/why-vector-clocks-are-hard/
* vclock header compression

$3

http://vimeo.com/43903960
https://github.com/aphyr/meangirls
http://docs.basho.com/riak/latest/dev/using/data-types/

$3

https://github.com/aphyr/jepsen

$3

http://en.wikipedia.org/wiki/Merkle_tree
https://github.com/c-geek/merkle

$3

leveldb range of keys to push to a node when it comes back