File de-duplication utilities
npm install @akumzy/dd
⚠️ Documentaion is very much incomplete but you can always checkout the test file for usage guide.
This package provides all the necessaries utility functions to implements file de-duplication ranging from splitting file content to blocks, generating hash and merging the file blocks together without corrupting the file.
``bash`
npm install @akumzy/dd
# Or
yarn add @akumzy/dd
`ts
import Dedupe from '@akumzy/dd
``
- Dedupe.deduplicate is used for creating file blocks and hash.
`ts`
Dedupe.deduplicate(
path: string,
chunk?: number,
algorithm?: string
): Promise
- path: File absolute path.
- chunk: Block size in bytes
- default: 500 * 1024 (500kb)sha256
- algorithm: Algorithm to use for hash file.
- default:
Example
`js`
const ONE_MAGA_BYTES = 1024 * 1024
let filePath = 'path-to-file'
let info = await deduplicate(filePath, ONE_MAGA_BYTES)
console.log(info) // =>
// {
// hash: 'f27a663ef8df8091e94d07ba090449a34b68461b9af5557377423057ce902484',
// blocks: [
// {
// order: 1,
// start: 0,
// end: 281,
// hash: '673c83b6fd07ba91954d4bc88f631bbe31bb9675b6c44b635801157c0ba94861'
// }
// ]
// }
- Dedupe.createBlocks is used for writing blocks to disk.
`ts`
Dedupe.createBlocks(
options: Deduplicate.CreateBlocksOptions
): Promise
Example
`ts``
await createBlocks({
input: filePath,
bucket: directoryToSaveBlocks,
blocks: info.blocks
})