A space-efficient object serialization library for node.js
npm install msgpacknode-msgpack is an addon for NodeJS that provides an
API for serializing and de-serializing JavaScript objects using the
MessagePack library. The performance of this
addon compared to the native JSON object isn't too bad, and the space
required for serialized data is far less than JSON.
node-msgpack is currently slower than the built-in JSON.stringify() andJSON.parse() methods. In recent versions of node.js, the JSON functions
have been heavily optimized. node-msgpack is still more compact, and we are
currently working performance improvements. Testing shows that, over 500k
iterations, msgpack.pack() is about 5x slower than JSON.stringify(), andmsgpack.unpack() is about 3.5x slower than JSON.parse().
Old performance numbers are below.
The following tests were performed with 500,000 instances of
the JavaScript object {'abcdef' : 1, 'qqq' : 13, '19' : [1, 2, 3, 4]}:
* JSON.stringify() 7.17 seconds
* JSON.parse(JSON.stringify()) 22.18 seconds
* msgpack.pack() 5.80 seconds
* msgpack.unpack(msgpack.pack()) 8.62 seconds
Note that node-msgpack produces and consumes Buffer objects, and a such does
not incur encoding/decoding overhead when performing I/O with native strings.
This module provides two methods: pack(), which consumes a JavaScript object
and produces a node Buffer object; and unpack(), which consumes a node Buffer
object and produces a JavaScript object. Packing of all native JavaScript types
(undefined, boolean, numbers, strings, arrays and objects) is supported, as
is the node Buffer type.
The below code snippet packs and then unpacks a JavaScript object, verifying
the resulting object at the end using assert.deepEqual().
``javascript
var assert = require('assert');
var msgpack = require('msgpack');
var o = {"a" : 1, "b" : 2, "c" : [1, 2, 3]};
var b = msgpack.pack(o);
var oo = msgpack.unpack(b);
assert.deepEqual(oo, o);
`
As a convenience, a higher level streaming API is provided in the
msgpack.Stream class, which can be constructed around a net.Streammsg
instance. This object emits events when an object has been received.
`javascript
var msgpack = require('msgpack');
// ... get a net.Stream instance, s, from somewhere
var ms = new msgpack.Stream(s);
ms.addListener('msg', function(m) {
sys.debug('received message: ' + sys.inspect(m));
});
`
The JavaScript type system does not map cleanly on to the MsgPack type system,
though it's pretty close.
When packing, JavaScript values are mapped to MsgPack types as follows
* undefined and null values map to MSGPACK_OBJECT_NILboolean
* values map to MSGPACK_OBJECT_BOOLEANnumber
* values map differently depending on their valueMSGPACK_OBJECT_DOUBLE
* Floating point values map to MSGPACK_OBJECT_POSITIVE_INTEGER
* Positive values map to MSGPACK_OBJECT_NEGATIVE_INTEGER
* Negative values map to string
* values map to MSGPACK_OBJECT_RAW; all strings are serializedArray.isArray()
with UTF-8 encoding
* Array values (as defined by ) map toMSGPACK_OBJECT_ARRAY
; each element in the array is packed individuallyMSGPACK_OBJECT_RAW
the rules in this list
* NodeJS Buffer values map to MSGPACK_OBJECT_MAP
* Everything else maps to , where we iterate over
the object's properties and pack them and their values as per the
mappings in this list
When unpacking, MsgPack types are mapped to JavaScript values as follows
* MSGPACK_OBJECT_NIL values map to the null valueMSGPACK_OBJECT_BOOLEAN
* values map to boolean valuesMSGPACK_OBJECT_POSITIVE_INTEGER
* , MSGPACK_OBJECT_NEGATIVE_INTEGER andMSGPACK_OBJECT_DOUBLE
values map to number valuesMSGPACK_OBJECT_ARRAY
* values map to arrays; each object in the array isMSGPACK_OBJECT_RAW
packed individually using the rules in this list
* values are mapped to string values; these values areMSGPACK_OBJECT_MAP
unpacked using either UTF-8 or ASCII encoding, depending on the contents
of the raw buffer
* values are mapped to JavaScript objects; keys and
values are unpacked individually using the rules in this list
Strings are particularly problematic here, as it's difficult to get hints down
into the packing and unpacking codepaths about how to interpret a particular
string or MSGPACK_OBJECT_RAW. If you have strict requirements about theBuffer.write()
encoding of your strings, it's recommended that you populate a Buffer object
yourself (e.g. using ) and pack that buffer rather than the
string. This will ensure that you can control what gets packed.
When unpacking, things are trickier as there is no way to know the encoding
used when a string was packed. There is an an open
ticket for the MsgPack
format to address this.
As a convenience and for debugging, bin/json2msgpack and bin/msgpack2json
are provided to convert JSON data to and from MessagePack data, reading from
stdin and writing to stdout.
% echo '[1, 2, 3]' | ./bin/json2msgpack | xxd -
0000000: 9301 0203 ....
% echo '[1, 2, 3]' | ./bin/json2msgpack | ./bin/msgpack2json
[1,2,3]
There are two ways to install msgpack.
npm install msgpack
This should build and install msgpack for you. Then just require('msgpack')`.
You will need node-gyp:
npm install -g node-gyp
Then from the root of the msgpack repo, you can run:
node-gyp rebuild
To run all tests use:
./run_tests
To run a specific test:
./run_tests test/lib/msgpack.js
To run benchmarks:
./run_tests test/benchmark/benchmark.js