A TypeScript project with tsup bundler for Rust WASM MD5 calculation
npm install fast-md5-web---
π Possibly the fastest MD5 calculation library for web environments - powered by Rust WebAssembly with Web Worker support for truly non-blocking computation. 8x faster in single-thread mode and 16x faster with multi-thread workers than spark-md5 in specific scenarios.
β‘ ESM-only package - Supports modern browsers, Node.js, and Deno with ES modules. No CommonJS support.
- 𧡠Web Worker Pool - True parallel processing with configurable worker threads, prevents UI blocking
- π SharedArrayBuffer Support - Zero-copy data transfer between main thread and workers for maximum performance
- π¦ Rust WebAssembly - Native performance with Rust compiled to WebAssembly
- β‘ High Performance - 8x faster in single-thread mode, 16x faster with multi-thread workers than spark-md5 for batch processing
- π¦ ESM-only - Modern ES modules for browsers, Node.js, and Deno (no CommonJS)
- π TypeScript Support - Full TypeScript declarations and type safety
- π Streaming Processing - Incremental MD5 calculation for large files (200MB+) without memory overflow
- π― Flexible Output - Support for 16-bit and 32-bit MD5 hash lengths
- π Auto Fallback - Automatically falls back to message passing when SharedArrayBuffer is unavailable
- π§ Smart Memory Management - Intelligent shared memory allocation with fragmentation control
- π Progress Tracking - Real-time progress updates for large file processing
- ποΈ Concurrency Control - Configurable maximum concurrent tasks to prevent system overload
- π¦ Batch Processing - Optimized batch processing with priority queues and task management
bash
npm install fast-md5-web
`$3
typescript
// Direct import from npm (recommended)
import { Md5CalculatorPool, WasmInit, Md5Calculator } from 'npm:fast-md5-web';// Or add to deno.json import map:
// {
// "imports": {
// "fast-md5-web": "npm:fast-md5-web"
// }
// }
`> β οΈ ESM Only: This package only supports ES modules. It works in modern browsers, Node.js (with
in package.json), and Deno. CommonJS is not supported.π Quick Start
`typescript
import { Md5CalculatorPool, WasmInit, Md5Calculator } from 'fast-md5-web';// Method 1: Using Worker Pool with SharedArrayBuffer (Recommended for processing multiple files)
const pool = new Md5CalculatorPool(
navigator.hardwareConcurrency, // Worker count
undefined, // SharedMemoryConfig (optional)
3 // Max concurrent tasks
);
// Enable shared memory for large files
pool.enableSharedMemory(64 1024 1024, 2 1024 1024); // 64MB memory, 2MB chunks
// Process large files with progress tracking
const largeFile = new File([/ large data /], 'large-file.bin');
const hash = await pool.calculateMd5(
largeFile,
32, // MD5 length
60000, // Timeout
(progress) => {
console.log(
Progress: ${progress.toFixed(1)}%);
},
1 // Priority (number)
);
console.log('MD5:', hash);// Batch processing
const files = [file1, file2, file3];
const results = await pool.calculateMd5Batch(
files,
32, // MD5 length
(completed, total) => {
console.log(
Progress: ${completed}/${total} files completed);
}
);
console.log('Batch results:', results);// Check pool status including memory usage
const status = pool.getPoolStatus();
console.log('Pool status:', {
workers: status.totalWorkers,
activeTasks: status.activeTasks,
memoryEnabled: status.sharedMemoryEnabled,
memoryUsage: status.sharedMemoryUsage ?
${(status.sharedMemoryUsage.used / 1024 / 1024).toFixed(2)}MB / ${(status.sharedMemoryUsage.total / 1024 / 1024).toFixed(2)}MB : 'N/A'
});// Clean up
pool.destroy();
// Method 1b: Traditional Worker Pool (without SharedArrayBuffer)
const traditionalPool = new Md5CalculatorPool(4); // Uses message passing by default
// Method 2: Direct WASM usage (Recommended for single small files)
await WasmInit();
const calculator = new Md5Calculator();
// Convert single file to Uint8Array
const file = new File(['Hello, World!'], 'example.txt');
const arrayBuffer = await file.arrayBuffer();
const data = new Uint8Array(arrayBuffer);
const hash = await calculator.calculate_md5_async(data, 32);
console.log('MD5:', hash);
// Method 3: Deno usage
// For Deno, you can import directly from npm:
import { Md5CalculatorPool, WasmInit, Md5Calculator } from 'npm:fast-md5-web';
// Or use with import maps in deno.json:
// {
// "imports": {
// "fast-md5-web": "npm:fast-md5-web"
// }
// }
// Then: import { Md5CalculatorPool, WasmInit, Md5Calculator } from 'fast-md5-web';
// Read file in Deno
const fileData = await Deno.readFile('./example.txt');
const data = new Uint8Array(fileData);
// Calculate MD5
await WasmInit();
const calculator = new Md5Calculator();
const hash = await calculator.calculate_md5_async(data, 32);
console.log('MD5:', hash);
`π API Reference
$3
Manages a pool of Web Workers for parallel MD5 calculation of multiple files with advanced features.
typescript
interface SharedMemoryConfig {
enabled: boolean;
memorySize: number;
chunkSize: number;
}interface PoolStatus {
totalWorkers: number;
availableWorkers: number;
pendingTasks: number;
activeTasks: number;
maxConcurrentTasks: number;
sharedMemoryEnabled: boolean;
sharedMemoryUsage?: {
total: number;
used: number;
available: number;
fragmentation: number;
};
}
class Md5CalculatorPool {
constructor(poolSize?: number, sharedMemoryConfig?: SharedMemoryConfig, maxConcurrentTasks?: number);
// Single file processing with streaming support
async calculateMd5(
data: Uint8Array | File,
md5Length?: number,
timeout?: number,
onProgress?: (progress: number) => void,
priority?: number
): Promise;
// Batch processing
async calculateMd5Batch(
files: (Uint8Array | File)[],
md5Length?: number,
timeout?: number, // singleFileTimeoutSetting
onProgress?: (completed: number, total: number) => void
): Promise;
// Shared memory management
enableSharedMemory(memorySize?: number, chunkSize?: number): boolean;
disableSharedMemory(): void;
// Task management
cancelTask(taskId: string): boolean;
// Pool status and cleanup
getPoolStatus(): PoolStatus;
destroy(): void;
}
`$3
Direct WASM MD5 calculator for single file processing.
typescript
class Md5Calculator {
constructor();
async calculate_md5_async(data: Uint8Array, md5Length: number): Promise;
set_log_enabled(enable: boolean): void;
is_log_enabled(): boolean;
}
`$3
-
- File data as byte array
- md5Length: number - Hash length (16 for 128-bit half, 32 for full 128-bit)
- poolSize: number - Number of worker threads (default: navigator.hardwareConcurrency)ποΈ Development
`bash
Install dependencies
npm installBuild WASM and TypeScript
npm run buildDevelopment mode with watch
npm run devType checking
npm run type-checkLinting
npm run lint
npm run lint:fixFormatting
npm run format
npm run format:checkClean build artifacts
npm run clean
`β οΈ Production Environment Requirements
To use SharedArrayBuffer for optimal performance in a production environment, your website must meet the following conditions:
1. Must be served over HTTPS - This is essential for creating a secure context.
2. Must set Cross-Origin Isolation HTTP headers:
Cross-Origin-Opener-Policy: same-origin
Cross-Origin-Embedder-Policy: require-corp
β±οΈ Timeout Setting
In the
method, the timeout parameter controls the calculation timeout in milliseconds.
- Default value: 60000ms (1 minute)
- Set to 0 to disable timeoutExample:
`typescript
// Disable timeout
await pool.calculateMd5(file, 32, 0);
`π Project Structure
βββ src/
β βββ index.ts # Main TypeScript entry
β βββ md5-worker.ts # Web Worker implementation
βββ wasm/ # Rust WASM source
β βββ src/
β β βββ lib.rs # Main Rust library
β β βββ utils.rs # Utility functions
β βββ Cargo.toml # Rust dependencies
β βββ pkg/ # Generated WASM package
βββ dist/ # Build output
βββ package.json
βββ tsup.config.ts # Build configuration
βββ tsconfig.json # TypeScript configuration
β‘ Performance
$3
Performance Comparison:
- Single-thread mode: ~8x faster than spark-md5 β‘
- Multi-thread workers: ~16x faster than spark-md5 π
- Live Demo: View performance test results
$3
- WebAssembly: Leverages Rust's performance for MD5 calculation
- Streaming Processing: Handles large files without loading entire content into memory
- Incremental Hashing: WASM-based incremental MD5 calculation for memory efficiency
- Smart Concurrency: Adaptive task scheduling with priority queues
- Zero-Copy Transfer: SharedArrayBuffer eliminates data copying overhead$3
- Chunked Processing: Configurable chunk sizes for optimal memory usage
- Shared Memory Pool: Efficient memory allocation and reuse
- Fragmentation Monitoring: Real-time memory fragmentation tracking
- Automatic Cleanup: Proper resource cleanup and garbage collection$3
- SharedArrayBuffer: Zero-copy data transfer eliminates serialization overhead
- Web Worker Pool: Parallel processing of multiple files prevents main thread blocking
- Rust WebAssembly: Native performance with zero-cost abstractions
- Chunked Processing: Automatic optimization for files > 1MB
- Memory Efficient: Streaming processing with controlled memory usage
- Multi-file Processing: Optimized for handling multiple files simultaneously with worker pool
- Auto Fallback: Graceful degradation to message passing when SharedArrayBuffer is unavailable
$3
#### For Large Files (>50MB)
typescript
// Enable shared memory with appropriate chunk size
pool.enableSharedMemory(128 1024 1024, 4 1024 1024); // 128MB, 4MB chunks// Use high priority for critical files
const hash = await pool.calculateMd5(
largeFile,
32,
60000, // timeout
(progress) => {
console.log(
Processing: ${progress.toFixed(1)}%);
},
10 // high priority (higher number = higher priority)
);
`#### For Batch Processing
typescript
// Sort files by size for optimal scheduling
const sortedFiles = files.sort((a, b) => b.size - a.size);const results = await pool.calculateMd5Batch(
sortedFiles,
32, // MD5 length
(completed, total) => {
console.log(
Progress: ${completed}/${total} files completed);
}
);
`#### Memory Monitoring
typescript
const status = pool.getPoolStatus();
if (status.sharedMemoryUsage && status.sharedMemoryUsage.fragmentation > 3) {
console.warn('High memory fragmentation detected');
// Consider recreating the pool
}
`$3
Processing 10MB file with 4 workers:
- With SharedArrayBuffer: ~1ms data transfer + ~150ms processing = ~151ms total
- Without SharedArrayBuffer: ~50ms data transfer + ~150ms processing = ~200ms total
- Performance gain: ~25% faster overall, 50x faster data transfer
Memory Usage Comparison:
- Traditional mode: 2x memory usage (original + copied data)
- SharedArrayBuffer mode: 1x memory usage (shared data)
- Memory savings: Up to 50% reduction in memory usage
Browser Compatibility
| Feature | Chrome | Firefox | Safari | Edge | Notes |
|---------|--------|---------|--------|---------|-------|
| Basic MD5 | β
| β
| β
| β
| Full support |
| Web Workers | β
| β
| β
| β
| Full support |
| Streaming Processing | β
| β
| β
| β
| File API required |
| SharedArrayBuffer | β
| β
| β
| β
| HTTPS + headers required |
| WebAssembly | β
| β
| β
| β
| Full support |
| Progress Tracking | β
| β
| β
| β
| Full support |
$3
For optimal performance with large files, SharedArrayBuffer requires:
html
`Or server headers:
Cross-Origin-Opener-Policy: same-origin
Cross-Origin-Embedder-Policy: require-corp
$3
When SharedArrayBuffer is unavailable:
- Automatically falls back to standard message passing
- Slightly reduced performance for very large files
- All features remain functional
π§ Troubleshooting
$3
#### SharedArrayBuffer Not Available
Problem: SharedArrayBuffer is undefined or not working
Solutions:
1. Ensure HTTPS is enabled (required for security)
2. Add required headers:
Cross-Origin-Opener-Policy: same-origin
Cross-Origin-Embedder-Policy: require-corp
3. Check browser support (Chrome 68+, Firefox 79+, Safari 15.2+)
4. The library automatically falls back to message passing#### Memory Issues with Large Files
Problem: Out of memory errors or slow performance
Solutions:
typescript
// Reduce chunk size for memory-constrained environments
pool.enableSharedMemory(32 1024 1024, 1 1024 1024); // 32MB, 1MB chunks// Reduce concurrent tasks
const pool = new Md5CalculatorPool(2, 1); // 2 workers, 1 concurrent task
// Monitor memory usage
const status = pool.getPoolStatus();
if (status.memoryUsed > 100 1024 1024) { // > 100MB
console.warn('High memory usage detected');
}
`#### Worker Initialization Failures
Problem: Workers fail to start or crash
Solutions:
1. Check browser console for detailed error messages
2. Ensure proper WASM loading:
typescript
try {
await WasmInit();
console.log('WASM initialized successfully');
} catch (error) {
console.error('WASM initialization failed:', error);
}
`
3. Verify file paths and module resolution
4. Check Content Security Policy (CSP) settings#### ESM Import Issues
Problem: Module import errors or CommonJS compatibility
Solutions:
1. Ensure
in package.json for Node.js
2. Use .mjs file extension
3. For bundlers, ensure ESM support is enabled
4. This package does NOT support CommonJS#### Performance Not as Expected
Problem: Slower than expected performance
Solutions:
1. Enable SharedArrayBuffer for large files
2. Adjust worker count based on CPU cores:
`typescript
const optimalWorkers = Math.min(navigator.hardwareConcurrency, 8);
const pool = new Md5CalculatorPool(optimalWorkers);
`
3. Use appropriate chunk sizes:
- Small files (<1MB): Direct processing
- Medium files (1-50MB): 1MB chunks
- Large files (>50MB): 2-4MB chunks
4. Process files in batches rather than individually$3
Enable detailed logging for troubleshooting:
typescript
// Enable WASM logging
const calculator = new Md5Calculator();
calculator.set_log_enabled(true);// Monitor pool status
const pool = new Md5CalculatorPool(4);
setInterval(() => {
const status = pool.getPoolStatus();
console.log('Pool Status:', {
workers: status.totalWorkers,
active: status.activeTasks,
pending: status.pendingTasks,
memory: status.sharedMemoryUsage ?
${(status.sharedMemoryUsage.used / 1024 / 1024).toFixed(2)}MB : 'N/A',
fragmentation: status.sharedMemoryUsage ?
${status.sharedMemoryUsage.fragmentation} : 'N/A'
});
}, 5000);
`$3
#### Node.js
- Requires Node.js 16+ with ES modules support
- Add
to package.json
- Web Workers use worker_threads under the hood#### Deno
- Use
npm:` specifier for imports#### Browsers
- Modern browsers (Chrome 68+, Firefox 79+, Safari 15.2+)
- HTTPS required for SharedArrayBuffer
- Check CSP policies for worker and WASM support
We welcome contributions! Please read our Contributing Guide for detailed information on how to submit pull requests, report issues, and contribute to the project.
MIT License - see the LICENSE file for details.