A peer to peer wire communication
npm install interblockchain-devp2pnode-* named) (all outdated).
Using the library
The first step is to include the library into your project with:
`
npm install
`
Then to import the ntrnetwork class and instantiate it. To receive the network transmitted transactions, you need to subscribe a callback which will receive as parameter network transactions data.
To be added to your code:
`
broadcaster = require ('ntrnetwork').broadcaster;
broadcaster.subscribe(myCallback);
myCallback(transaction) {
// do something with the transaction
}
`
Run/Build
This library needs to run in an ECMAScript 2016 and nodeJS version 8.11.2 and up.
$3
- clone the source code into your project with
`
git clone https://github.com/Interblockchain/internodes.git
`
- install dependencies registered in the package.json file
located in the root directory containing the Interblockchain wire protocol.
`
npm install
`
- Include the dependencies into your project (as illustrated in the peer-communication.interblockchain.js)
`
const devp2p = require('../src');
const LRUCache = require('lru-cache');
const ms = require('ms');
const assert = require('assert');
const { randomBytes } = require('crypto');
const rlp = require('rlp-encoding');
const Buffer = require('safe-buffer').Buffer;
const mac = require('getmac');
`
Usage/Examples
An example implementation is included in the test directory, in the file:
peer-communication.interblockchain.js
All components of this library are implemented as Node EventEmitter objects
and make heavy use of the Node.js network stack.
You can react on events from the network like this:
`
dpt.on('peer:added', (peer) => {
// Do something...
})
`
Basic example to connect to some bootstrap nodes and get basic peer info:
- simple
Communicate with peers to read and write new transactions:
- peer-communication
____
Interblockchain Wire Protocol (NTR)
Is an upper layer protocol to transmit message among Interblockchain nodes, see ./src/interblockchain
$3
When a new peer is added to the network, the peer:added event is fired. The event handler attached to this event receives as parameter the connected peer object. The first task of this event handler is to send a status object to the just connected peer. This is sent to the connected peer with the
sendStatus() function as illustrated below.
`
rlpx.on('peer:added', (peer) => {
ntr.sendStatus({
networkId: CHAIN_ID,
networkName: "interblockchain",
Note:"test in progress"
});
// Do something with this messsage :-)
}
`
When other nodes send their own status message, the latter is trapped by an event handler associated to the status event. This event is fired only once per peer connected.
`
eth.once('status', () => {
// Send an initial message
ntr.sendMessage()
})
`
Each message received from other peers is hanlded by an event handler associated to the message event.
`
ntr.on('message', async (code, payload) => {
if (code === devp2p.NTR.MESSAGE_CODES.TX) {
// Do something with this messsage :-)
}
})
`
$3
$3
Send initial status message.
- status - Status message to send, format { networkId: CHAIN_ID}.
$3
Send initial status message.
- code - The message code, see MESSAGE_CODES for available message types.
- payload - Payload as a list, will be rlp-encoded.
$3
Events emitted:
| Event | Description |
| ------------- |:----------------------------------------:|
| message | Message received |
| status | Status info received |
_______
RLPx Transport Protocol
Connect to a peer, organize the communication, see ./src/rlpx/
$3
Create your RLPx object, e.g.:
`
const rlpx = new devp2p.RLPx(PRIVATE_KEY, {
dpt: dpt,
maxPeers: 25,
capabilities: [
interblockchain
],
listenPort: null
})
`
$3
$3
Manages the handshake (ECIES) and the handling of the peer communication (Peer).
$3
Creates new RLPx object
- privateKey - Key for message encoding/signing.
- options.timeout - Peer ping timeout in ms (default: 10s).
- options.maxPeers - Max number of peer connections (default: 10).
- options.clientId - Client ID string (default example: ethereumjs-devp2p/v2.1.3/darwin-x64/nodejs).
- options.remoteClientIdFilter - Optional list of client ID filter strings (e.g. ['go1.5', 'quorum']).
- options.capabilities - Upper layer protocol capabilities, e.g. [devp2p.ETH.eth63, devp2p.ETH.eth62].
- options.listenPort - The listening port for the server or null for default.
- options.dpt - DPT object for the peers to connect to (default: null, no DPT peer management).
$3
Manually connect to peer without DPT.
- peer - Peer to connect to, format { id: PEER_ID, address: PEER_ADDRESS, port: PEER_PORT }.
$3
Broadcast a message to all connected peers.
- code - The message code, see MESSAGE_CODES for available message types.
- message - can be an Array, a string
For other connection/utility functions like listen, getPeers see ./src/rlpx/index.js.
$3
Events emitted:
| Event | Description |
| ------------- |:----------------------------------------:|
| peer:added | Handshake with peer successful |
| peer:removed | Disconnected from peer |
| peer:error | Error connecting to peer |
| listening | Forwarded from server |
| close | Forwarded from server |
| error | Forwarded from server |
$3
- RLPx: Cryptographic Network & Transport Protocol
- devp2p wire protocol
___
Distributed Peer Table (DPT) / Node Discovery
Maintain/manage a list of peers, see ./src/dpt/, also
includes node discovery (./src/dpt/server.js)
$3
Create your peer table:
`
const dpt = new DPT(Buffer.from(PRIVATE_KEY, 'hex'), {
endpoint: {
address: '0.0.0.0',
udpPort: null,
tcpPort: null
}
})
`
Add some bootstrap nodes (or some custom nodes with dpt.addPeer()):
`
dpt.bootstrap(bootnode).catch((err) => console.error('Something went wrong!'))
`
$3
#### DPT (extends EventEmitter)
Distributed Peer Table. Manages a Kademlia DHT K-bucket (Kbucket) for storing peer information
and a BanList for keeping a list of bad peers. Server implements the node discovery (ping,
pong, findNeighbours).
##### new DPT(privateKey, options)
Creates new DPT object
- privateKey - Key for message encoding/signing.
- options.refreshInterval - Interval in ms for refreshing (calling findNeighbours) the peer list (default: 60s).
- options.createSocket - A datagram (dgram) createSocket function, passed to Server (default: dgram.createSocket.bind(null, 'udp4')).
- options.timeout - Timeout in ms for server ping, passed to Server (default: 10s).
- options.endpoint - Endpoint information to send with the server ping, passed to Server (default: { address: '0.0.0.0', udpPort: null, tcpPort: null }).
#### dpt.bootstrap(peer) (async)
Uses a peer as new bootstrap peer and calls findNeighbouts.
- peer - Peer to be added, format { address: [ADDRESS], udpPort: [UDPPORT], tcpPort: [TCPPORT] }.
#### dpt.addPeer(object) (async)
Adds a new peer.
- object - Peer to be added, format { address: [ADDRESS], udpPort: [UDPPORT], tcpPort: [TCPPORT] }.
For other utility functions like getPeer, getPeers see ./src/dpt/index.js.
$3
Events emitted:
| Event | Description |
| ------------- |:----------------------------------------:|
| peer:added | Peer added to DHT bucket |
| peer:removed | Peer removed from DHT bucket |
| peer:new | New peer added |
| listening | Forwarded from server |
| close | Forwarded from server |
| error | Forwarded from server |
$3
- Node discovery protocol
- RLPx - Node Discovery Protocol
- Kademlia Peer Selection
___________
Safe Buffer API
usage
The goal of this package is to provide a safe replacement for the node.js Buffer.
It's a drop-in replacement for Buffer. You can use it by adding one require line to
the top of your node.js modules:
`js
var Buffer = require('safe-buffer').Buffer
// Existing buffer code will continue to work without issues:
new Buffer('hey', 'utf8')
new Buffer([1, 2, 3], 'utf8')
new Buffer(obj)
new Buffer(16) // create an uninitialized buffer (potentially unsafe)
// But you can use these new explicit APIs to make clear what you want:
Buffer.from('hey', 'utf8') // convert from many types to a Buffer
Buffer.alloc(16) // create a zero-filled buffer (safe)
Buffer.allocUnsafe(16) // create an uninitialized buffer (potentially unsafe)
`
api
$3
* array {Array}
Allocates a new Buffer using an array of octets.
`js
const buf = Buffer.from([0x62,0x75,0x66,0x66,0x65,0x72]);
// creates a new Buffer containing ASCII bytes
// ['b','u','f','f','e','r']
`
A TypeError will be thrown if array is not an Array.
$3
* arrayBuffer {ArrayBuffer} The .buffer property of a TypedArray or
a new ArrayBuffer()
* byteOffset {Number} Default: 0
* length {Number} Default: arrayBuffer.length - byteOffset
When passed a reference to the .buffer property of a TypedArray instance,
the newly created Buffer will share the same allocated memory as the
TypedArray.
`js
const arr = new Uint16Array(2);
arr[0] = 5000;
arr[1] = 4000;
const buf = Buffer.from(arr.buffer); // shares the memory with arr;
console.log(buf);
// Prints:
// changing the TypedArray changes the Buffer also
arr[1] = 6000;
console.log(buf);
// Prints:
`
The optional byteOffset and length arguments specify a memory range within
the arrayBuffer that will be shared by the Buffer.
`js
const ab = new ArrayBuffer(10);
const buf = Buffer.from(ab, 0, 2);
console.log(buf.length);
// Prints: 2
`
A TypeError will be thrown if arrayBuffer is not an ArrayBuffer.
$3
* buffer {Buffer}
Copies the passed buffer data onto a new Buffer instance.
`js
const buf1 = Buffer.from('buffer');
const buf2 = Buffer.from(buf1);
buf1[0] = 0x61;
console.log(buf1.toString());
// 'auffer'
console.log(buf2.toString());
// 'buffer' (copy is not changed)
`
A TypeError will be thrown if buffer is not a Buffer.
$3
* str {String} String to encode.
* encoding {String} Encoding to use, Default: 'utf8'
Creates a new Buffer containing the given JavaScript string str. If
provided, the encoding parameter identifies the character encoding.
If not provided, encoding defaults to 'utf8'.
`js
const buf1 = Buffer.from('this is a tést');
console.log(buf1.toString());
// prints: this is a tést
console.log(buf1.toString('ascii'));
// prints: this is a tC)st
const buf2 = Buffer.from('7468697320697320612074c3a97374', 'hex');
console.log(buf2.toString());
// prints: this is a tést
`
A TypeError will be thrown if str is not a string.
$3
* size {Number}
* fill {Value} Default: undefined
* encoding {String} Default: utf8
Allocates a new Buffer of size bytes. If fill is undefined, the
Buffer will be zero-filled.
`js
const buf = Buffer.alloc(5);
console.log(buf);
//
`
The size must be less than or equal to the value of
require('buffer').kMaxLength (on 64-bit architectures, kMaxLength is
(2^31)-1). Otherwise, a [RangeError][] is thrown. A zero-length Buffer will
be created if a size less than or equal to 0 is specified.
If fill is specified, the allocated Buffer will be initialized by calling
buf.fill(fill). See [buf.fill()][] for more information.
`js
const buf = Buffer.alloc(5, 'a');
console.log(buf);
//
`
If both fill and encoding are specified, the allocated Buffer will be
initialized by calling buf.fill(fill, encoding). For example:
`js
const buf = Buffer.alloc(11, 'aGVsbG8gd29ybGQ=', 'base64');
console.log(buf);
//
`
Calling Buffer.alloc(size) can be significantly slower than the alternative
Buffer.allocUnsafe(size) but ensures that the newly created Buffer instance
contents will never contain sensitive data.
A TypeError will be thrown if size is not a number.
$3
* size {Number}
Allocates a new non-zero-filled Buffer of size bytes. The size must
be less than or equal to the value of require('buffer').kMaxLength (on 64-bit
architectures, kMaxLength is (2^31)-1). Otherwise, a [RangeError][] is
thrown. A zero-length Buffer will be created if a size less than or equal to
0 is specified.
The underlying memory for Buffer instances created in this way is *not
initialized*. The contents of the newly created Buffer are unknown and
may contain sensitive data. Use [buf.fill(0)][] to initialize such
Buffer instances to zeroes.
`js
const buf = Buffer.allocUnsafe(5);
console.log(buf);
//
// (octets will be different, every time)
buf.fill(0);
console.log(buf);
//
`
A TypeError will be thrown if size is not a number.
Note that the Buffer module pre-allocates an internal Buffer instance of
size Buffer.poolSize that is used as a pool for the fast allocation of new
Buffer instances created using Buffer.allocUnsafe(size) (and the deprecated
new Buffer(size) constructor) only when size is less than or equal to
Buffer.poolSize >> 1 (floor of Buffer.poolSize divided by two). The default
value of Buffer.poolSize is 8192 but can be modified.
Use of this pre-allocated internal memory pool is a key difference between
calling Buffer.alloc(size, fill) vs. Buffer.allocUnsafe(size).fill(fill).
Specifically, Buffer.alloc(size, fill) will never use the internal Buffer
pool, while Buffer.allocUnsafe(size).fill(fill) will use the internal
Buffer pool if size is less than or equal to half Buffer.poolSize. The
difference is subtle but can be important when an application requires the
additional performance that Buffer.allocUnsafe(size) provides.
$3
* size {Number}
Allocates a new non-zero-filled and non-pooled Buffer of size bytes. The
size must be less than or equal to the value of
require('buffer').kMaxLength (on 64-bit architectures, kMaxLength is
(2^31)-1). Otherwise, a [RangeError][] is thrown. A zero-length Buffer will
be created if a size less than or equal to 0 is specified.
The underlying memory for Buffer instances created in this way is *not
initialized*. The contents of the newly created Buffer are unknown and
may contain sensitive data. Use [buf.fill(0)][] to initialize such
Buffer instances to zeroes.
When using Buffer.allocUnsafe() to allocate new Buffer instances,
allocations under 4KB are, by default, sliced from a single pre-allocated
Buffer. This allows applications to avoid the garbage collection overhead of
creating many individually allocated Buffers. This approach improves both
performance and memory usage by eliminating the need to track and cleanup as
many Persistent objects.
However, in the case where a developer may need to retain a small chunk of
memory from a pool for an indeterminate amount of time, it may be appropriate
to create an un-pooled Buffer instance using Buffer.allocUnsafeSlow() then
copy out the relevant bits.
`js
// need to keep around a few small chunks of memory
const store = [];
socket.on('readable', () => {
const data = socket.read();
// allocate for retained data
const sb = Buffer.allocUnsafeSlow(10);
// copy the data into the new allocation
data.copy(sb, 0, 0, 10);
store.push(sb);
});
`
Use of Buffer.allocUnsafeSlow() should be used only as a last resort after
a developer has observed undue memory retention in their applications.
A TypeError will be thrown if size is not a number.
$3
The rest of the Buffer API is exactly the same as in node.js.
See the docs.
Related links
- Node.js issue: Buffer(number) is unsafe
- Node.js Enhancement Proposal: Buffer.from/Buffer.alloc/Buffer.zalloc/Buffer() soft-deprecate
Why is
Buffer unsafe?
Today, the node.js Buffer constructor is overloaded to handle many different argument
types like String, Array, Object, TypedArrayView (Uint8Array, etc.),
ArrayBuffer, and also Number.
The API is optimized for convenience: you can throw any type at it, and it will try to do
what you want.
Because the Buffer constructor is so powerful, you often see code like this:
`js
// Convert UTF-8 strings to hex
function toHex (str) {
return new Buffer(str).toString('hex')
}
`
But what happens if toHex is called with a Number argument?
$3
If an attacker can make your program call the Buffer constructor with a Number
argument, then they can make it allocate uninitialized memory from the node.js process.
This could potentially disclose TLS private keys, user data, or database passwords.
When the Buffer constructor is passed a Number argument, it returns an
UNINITIALIZED block of memory of the specified size. When you create a Buffer like
this, you MUST overwrite the contents before returning it to the user.
From the node.js docs:
> new Buffer(size)
>
> - size Number
>
> The underlying memory for Buffer instances created in this way is not initialized.
> **The contents of a newly created Buffer are unknown and could contain sensitive
> data.** Use buf.fill(0) to initialize a Buffer to zeroes.
(Emphasis our own.)
Whenever the programmer intended to create an uninitialized Buffer you often see code
like this:
`js
var buf = new Buffer(16)
// Immediately overwrite the uninitialized buffer with data from another buffer
for (var i = 0; i < buf.length; i++) {
buf[i] = otherBuf[i]
}
`
$3
Yes. It's surprisingly common to forget to check the type of your variables in a
dynamically-typed language like JavaScript.
Usually the consequences of assuming the wrong type is that your program crashes with an
uncaught exception. But the failure mode for forgetting to check the type of arguments to
the Buffer constructor is more catastrophic.
Here's an example of a vulnerable service that takes a JSON payload and converts it to
hex:
`js
// Take a JSON payload {str: "some string"} and convert it to hex
var server = http.createServer(function (req, res) {
var data = ''
req.setEncoding('utf8')
req.on('data', function (chunk) {
data += chunk
})
req.on('end', function () {
var body = JSON.parse(data)
res.end(new Buffer(body.str).toString('hex'))
})
})
server.listen(8080)
`
In this example, an http client just has to send:
`json
{
"str": 1000
}
`
and it will get back 1,000 bytes of uninitialized memory from the server.
This is a very serious bug. It's similar in severity to the
the Heartbleed bug that allowed disclosure of OpenSSL process
memory by remote attackers.
$3
#### bittorrent-dht
Mathias Buus and I
(Feross Aboukhadijeh) found this issue in one of our own packages,
bittorrent-dht. The bug would allow
anyone on the internet to send a series of messages to a user of bittorrent-dht and get
them to reveal 20 bytes at a time of uninitialized memory from the node.js process.
Here's
the commit
that fixed it. We released a new fixed version, created a
Node Security Project disclosure, and deprecated all
vulnerable versions on npm so users will get a warning to upgrade to a newer version.
#### ws
That got us wondering if there were other vulnerable packages. Sure enough, within a short
period of time, we found the same issue in ws, the
most popular WebSocket implementation in node.js.
If certain APIs were called with Number parameters instead of String or Buffer as
expected, then uninitialized server memory would be disclosed to the remote peer.
These were the vulnerable methods:
`js
socket.send(number)
socket.ping(number)
socket.pong(number)
`
Here's a vulnerable socket server with some echo functionality:
`js
server.on('connection', function (socket) {
socket.on('message', function (message) {
message = JSON.parse(message)
if (message.type === 'echo') {
socket.send(message.data) // send back the user's message
}
})
})
`
socket.send(number) called on the server, will disclose server memory.
Here's the release where the issue
was fixed, with a more detailed explanation. Props to
Arnout Kazemier for the quick fix. Here's the
Node Security Project disclosure.
$3
It's important that node.js offers a fast way to get memory otherwise performance-critical
applications would needlessly get a lot slower.
But we need a better way to signal our intent as programmers. **When we want
uninitialized memory, we should request it explicitly.**
Sensitive functionality should not be packed into a developer-friendly API that loosely
accepts many different types. This type of API encourages the lazy practice of passing
variables in without checking the type very carefully.
#### A new API: Buffer.allocUnsafe(number)
The functionality of creating buffers with uninitialized memory should be part of another
API. We propose Buffer.allocUnsafe(number). This way, it's not part of an API that
frequently gets user input of all sorts of different types passed into it.
`js
var buf = Buffer.allocUnsafe(16) // careful, uninitialized memory!
// Immediately overwrite the uninitialized buffer with data from another buffer
for (var i = 0; i < buf.length; i++) {
buf[i] = otherBuf[i]
}
`
$3
We sent a PR to node.js core (merged as
semver-major) which defends against one case:
`js
var str = 16
new Buffer(str, 'utf8')
`
In this situation, it's implied that the programmer intended the first argument to be a
string, since they passed an encoding as a second argument. Today, node.js will allocate
uninitialized memory in the case of new Buffer(number, encoding), which is probably not
what the programmer intended.
But this is only a partial solution, since if the programmer does new Buffer(variable)
(without an encoding parameter) there's no way to know what they intended. If variable
is sometimes a number, then uninitialized memory will sometimes be returned.
$3
We could deprecate and remove new Buffer(number) and use Buffer.allocUnsafe(number) when
we need uninitialized memory. But that would break 1000s of packages.
~~We believe the best solution is to:~~
~~1. Change new Buffer(number) to return safe, zeroed-out memory~~
~~2. Create a new API for creating uninitialized Buffers. We propose: Buffer.allocUnsafe(number)~~
#### Update
We now support adding three new APIs:
- Buffer.from(value) - convert from any type to a buffer
- Buffer.alloc(size) - create a zero-filled buffer
- Buffer.allocUnsafe(size) - create an uninitialized buffer with given size
This solves the core problem that affected ws and bittorrent-dht which is
Buffer(variable) getting tricked into taking a number argument.
This way, existing code continues working and the impact on the npm ecosystem will be
minimal. Over time, npm maintainers can migrate performance-critical code to use
Buffer.allocUnsafe(number) instead of new Buffer(number).
___
RLP Encoding - decoding API
$3
`javascript
var RLP = require('rlp');
var assert = require('assert');
var nestedList = [ [], [[]], [ [], [[]] ] ];
var encoded = RLP.encode(nestedList);
var decoded = RLP.decode(encoded);
assert.deepEqual(nestedList, decoded);
`
$3
rlp.encode(plain) - RLP encodes an Array, Buffer or String and returns a Buffer.
rlp.decode(encoded, [skipRemainderCheck=false]) - Decodes an RLP encoded Buffer, Array or String and returns a Buffer or an Array of Buffers. If skipRemainderCheck is enabled, rlp` will just decode the first rlp sequence in the buffer. By default, it would throw an error if there are more bytes in Buffer than used by rlp sequence.