NodeJS bindings for fastcdc-rs
npm install fastcdcfastcdc: NodeJS bindings for @nlfielder's fastcdc-rs package.
This module implements the fast content defined chunking algorithm (FastCDC) described in the following paper:
W. Xia et al. "FastCDC: a Fast and Efficient Content-Defined Chunking Approach for Data Deduplication" Usenix 2016
Content defined chunking takes a blob of bytes (like a file for example) and cuts it into more or less uniform sized chunks along content boundaries. Because these chunk boundaries are determined by the contents of a file, not regular intervals they are robust to local modifications like insertions and deletions. This makes content defined chunking useful when deduplicating date, comparing files and synchronizing files over remote network storage.
``javascript
const fastCDC = require('./index.js')
const buffer = new Uint8Array(10 * 1024)
for (let i = 0; i < buffer.length; ++i) {
buffer[i] = Math.random() * 256
}
console.log(fastCDC(buffer, {
min: 1024,
avg: 4096,
max: 65536
}))
`
Output
`
[ 0, 2594, 6454, 8106, 9244, 10240 ]
If your system has an up-to-date build of nodejs and rust you should be able to install this package using
`
npm install fastcdc
Computes a list of content defined chunk boundaries using fastcdc-rs.
Arguments
* bytes is either an ArrayBuffer or a Uint8Array containing the bytes to be hashedopts
* is a set of optional arguments to the chunker. If it is a number, then it assumed to be the average chunk size in bytes (default is 1024). Otherwise, if an object is passed then it is assumed to be a configuration with the following properties:min
* the minimum chunk sizeavg
* the average chunk sizemax
* the maximum chunk size
Returns An array of chunk boundaries (0 and bytes.length` inclusive) representing the chunks of the file