@dstanesc/wasm-chunking-node-eval

Wasm Content Defined Chunking Evaluation

Wasm library for content-based slicing. Convenience wrapper on existing rolling hash implementations in rust, such as provided by fastcdc, asuran chunker, etc.

Build Requirements

- Rust toolchain
- Wasm-pack

Test

``sh
wasm-pack test --headless --firefox
`

Wasm Build

`sh
wasm-pack build --out-dir pkg/webpack --out-name chunking --target bundler --scope dstanesc
wasm-pack build --out-dir pkg/node --out-name chunking --target nodejs --scope dstanesc
`

Install for NodeJS

`
npm install @dstanesc/wasm-chunking-node-eval
`

Install for Webpack bundling

`
npm install @dstanesc/wasm-chunking-webpack-eval
`

Usage

`js
import {compute_chunks_buzhash, compute_chunks_fastcdc} from "@dstanesc/wasm-chunking-node-eval";

const buf = ...
// mask 0b11111111111111
const offsets_buz = compute_chunks_buzhash(buf, 15).values();
// chunk sizes: min 16 KiB, avg 32 KiB, max 64 KiB
const offsets_fast = compute_chunks_fastcdc(buf, 16384, 32768, 65536).values();
``

Licenses

Licensed under either of

* Apache License, Version 2.0, (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
* MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)

at your option.

Fastcdc is distributed under the terms of the MIT license

Asuran Chunker is distributed under the terms of the BSD 2 Clause + Patent License.