High-performance Reed-Solomon error correction library optimized for Apple Silicon
npm install @digitaldefiance/node-rs-accelerateHigh-performance Reed-Solomon error correction library optimized for Apple Silicon (M1/M2/M3/M4).



Reed-Solomon codes are a family of error-correcting codes that enable data recovery from corruption or loss. They are widely used in:
- Distributed storage systems (RAID, cloud storage)
- Communications (satellite, deep space)
- QR codes and barcodes
- Blockchain and cryptocurrency (data availability)
This library provides systematic Reed-Solomon encoding and erasure decoding with hardware acceleration through:
- Apple Accelerate framework for CPU SIMD operations
- Metal Performance Shaders for GPU acceleration
- ARM NEON with advanced instructions (vtbl, veor3)
- Grand Central Dispatch for parallel processing
- Blazing fast: Up to 30 GB/s encoding throughput on Apple Silicon
- Hardware accelerated: Leverages Accelerate, Metal, and NEON
- Systematic encoding: Original data appears unchanged in output
- Flexible field sizes: GF(2^8) for up to 256 shards, GF(2^16) for up to 65536 shards
- Multiple matrix types: Vandermonde and Cauchy matrix constructions
- Streaming API: Process large files without loading into memory
- Optional compression: gzip, deflate, or brotli before encoding
- Hash validation: SHA-256 integrity checking
- Full TypeScript support: Complete type definitions and JSDoc
- Node.js >= 16.0.0
- macOS >= 11.0 (Big Sur)
- Apple Silicon (M1/M2/M3/M4) processor
- Xcode Command Line Tools (for native compilation)
``bashUsing yarn
yarn add @digitaldefiance/node-rs-accelerate
The native addon will be compiled automatically during installation.
$3
`typescript
import { isMetalAvailable, initMetal } from '@digitaldefiance/node-rs-accelerate';console.log('Metal GPU available:', isMetalAvailable());
console.log('Metal initialized:', initMetal());
`Quick Start
$3
`typescript
import {
ReedSolomonEncoder,
ReedSolomonDecoder,
ShardInfo
} from '@digitaldefiance/node-rs-accelerate';// Create encoder: 10 data shards + 4 parity shards
// Can recover from up to 4 lost shards
const encoder = new ReedSolomonEncoder({
dataShards: 10, // K = 10
parityShards: 4, // M = 4
shardSize: 1024 // 1KB per shard
});
// Encode 10KB of data (10 shards × 1KB)
const data = new Uint8Array(10 * 1024);
// Fill with your data...
const encoded = encoder.encode(data);
console.log(
Data shards: ${encoded.dataShards.length}); // 10
console.log(Parity shards: ${encoded.parityShards.length}); // 4// Create decoder with matching configuration
const decoder = new ReedSolomonDecoder({
dataShards: 10,
parityShards: 4,
shardSize: 1024
});
// Simulate losing 4 shards (indices 3, 5, 7, 9)
const availableShards: ShardInfo[] = [
{ index: 0, data: encoded.dataShards[0], isData: true },
{ index: 1, data: encoded.dataShards[1], isData: true },
{ index: 2, data: encoded.dataShards[2], isData: true },
// index 3 lost
{ index: 4, data: encoded.dataShards[4], isData: true },
// index 5 lost
{ index: 6, data: encoded.dataShards[6], isData: true },
// index 7 lost
{ index: 8, data: encoded.dataShards[8], isData: true },
// index 9 lost
{ index: 10, data: encoded.parityShards[0], isData: false },
{ index: 11, data: encoded.parityShards[1], isData: false },
];
// Decode from any 10 shards
const decoded = decoder.decode(availableShards);
console.log('Data recovered successfully!');
`$3
`typescript
const encoder = new ReedSolomonEncoder({
dataShards: 10,
parityShards: 4,
shardSize: 1024,
compression: {
enabled: true,
level: 6, // 0-9, higher = better compression
algorithm: 'gzip' // 'gzip', 'deflate', or 'brotli'
}
});const encoded = encoder.encode(data);
console.log(
Original: ${encoded.originalSize} bytes);
console.log(Compressed: ${encoded.compressedSize} bytes);
`$3
`typescript
const encoder = new ReedSolomonEncoder({
dataShards: 10,
parityShards: 4,
shardSize: 1024,
enableHashValidation: true
});const encoded = encoder.encode(data);
console.log('Data hashes:', encoded.dataHashes);
console.log('Parity hashes:', encoded.parityHashes);
// Include hashes when decoding for validation
const shards: ShardInfo[] = encoded.dataShards.map((data, i) => ({
index: i,
data,
isData: true,
hash: encoded.dataHashes![i]
}));
const decoder = new ReedSolomonDecoder({
dataShards: 10,
parityShards: 4,
shardSize: 1024,
enableHashValidation: true
});
// Will throw CorruptionError if hash doesn't match
const decoded = decoder.decode(shards);
`$3
`typescript
import { createReadStream, createWriteStream } from 'fs';
import { pipeline } from 'stream/promises';const encoder = new ReedSolomonEncoder({
dataShards: 10,
parityShards: 4,
shardSize: 64 * 1024 // 64KB per shard
});
// Stream encode a large file
await pipeline(
createReadStream('large-file.dat'),
encoder.encodeStream(),
// Handle encoded chunks...
);
`API Overview
$3
| Class | Description |
|-------|-------------|
|
ReedSolomonEncoder | Encodes data into K data shards + M parity shards |
| ReedSolomonDecoder | Decodes data from any K shards |
| StreamingEncoder | Transform stream for encoding |
| StreamingDecoder | Transform stream for decoding |$3
| Type | Description |
|------|-------------|
|
EncoderConfig | Configuration for encoder |
| DecoderConfig | Configuration for decoder |
| EncodedData | Result of encoding operation |
| ShardInfo | Information about a single shard |
| CompressionConfig | Compression settings |$3
| Enum | Values | Description |
|------|--------|-------------|
|
GaloisField | GF256, GF65536 | Field size (max shards) |
| MatrixType | Vandermonde, Cauchy | Matrix construction |
| PrimitivePolynomialGF256 | DEFAULT, ANSI, CCSDS, etc. | GF(2^8) polynomial |
| PrimitivePolynomialGF65536 | DEFAULT, ALT1, ALT2, ALT3 | GF(2^16) polynomial |$3
| Error | Code | Description |
|-------|------|-------------|
|
ConfigurationError | INVALID_CONFIG | Invalid configuration parameters |
| InsufficientShardsError | INSUFFICIENT_SHARDS | Not enough shards for decoding |
| CorruptionError | DATA_CORRUPTION | Data corruption detected |
| NativeError | NATIVE_ERROR | Native code operation failed |
| MetalError | METAL_ERROR | GPU operation failed |$3
| Function | Description |
|----------|-------------|
|
validateConfig(config) | Validate encoder/decoder configuration |
| estimateMemoryUsage(config) | Estimate memory usage in bytes |
| shouldUseGPU(config) | Check if GPU should be used |
| isMetalAvailable() | Check if Metal GPU is available |
| initMetal() | Initialize Metal GPU |Performance
$3
| Configuration | Throughput | Speedup vs JS |
|--------------|------------|---------------|
| (10,4) 64KB shards | 17.4 GB/s | 97x |
| (10,4) 1MB shards | 30.3 GB/s | 167x |
| (20,10) 64KB shards | 15.5 GB/s | 218x |
| (50,20) 64KB shards | 12.9 GB/s | 358x |
$3
- vtbl table lookup: 10x speedup for GF multiplication
- Interleaved processing: 6-7x speedup for multiply-accumulate
- GCD parallel encoding: 13-50x speedup depending on configuration
- Cache-optimized access: Minimizes cache misses
$3
GPU acceleration is automatically enabled when beneficial. Manual control:
`typescript
const encoder = new ReedSolomonEncoder({
dataShards: 10,
parityShards: 4,
shardSize: 1024 * 1024, // 1MB
useGPU: true, // Force GPU
gpuThreshold: 10 * 1024 // Use GPU when shardSize >= 10KB
});
`Recommendation: Let the library auto-detect. CPU with SIMD optimizations often outperforms GPU for typical configurations due to data transfer overhead.
Common Configurations
| Use Case | K | M | Overhead | Fault Tolerance |
|----------|---|---|----------|-----------------|
| Basic redundancy | 10 | 4 | 40% | 4 shards |
| High availability | 10 | 10 | 100% | 10 shards |
| Storage efficient | 20 | 4 | 20% | 4 shards |
| Maximum protection | 100 | 50 | 50% | 50 shards |
Development
$3
`bash
Clone the repository
git clone https://github.com/digitaldefiance/node-rs-accelerate.git
cd node-rs-accelerateInstall dependencies
yarn installBuild native addon and TypeScript
yarn build
`$3
`bash
All tests
yarn testProperty-based tests
yarn test:propertiesIntegration tests
yarn test:integrationMemory leak tests
yarn test:memory
`$3
`bash
Full benchmark suite
yarn benchmarkIndividual benchmarks
node benchmarks/encoding.js
node benchmarks/decoding.js
node benchmarks/gf_operations.js
node benchmarks/simd_benchmark.js
``- API Reference - Complete API documentation
- Performance Guide - Performance characteristics and optimization tips
- Optimization Guide - Implementation details for ARM NEON optimizations
- Troubleshooting - Common issues and solutions
MIT License - see LICENSE for details.
Contributions are welcome! Please see the implementation plan for areas that need work.
- Apple's Accelerate framework for high-performance numerical computing
- Metal Performance Shaders for GPU acceleration
- The Reed-Solomon algorithm by Irving S. Reed and Gustave Solomon (1960)