shingles

Fast, privacy-aware client-side near-duplicate detection in the browser
A lightweight Rust library compiled to WebAssembly (via wasm-bindgen) for computing MinHash signatures and Locality-Sensitive Hashing (LSH) indices to find similar/duplicate texts entirely in the browser, with optional salting for privacy-preserving similarity checks.

No server needed. Runs offline. Perfect for PWAs, note-taking apps, plagiarism checkers, private document comparison, or client-side data cleaning.

![npm](https://www.npmjs.com/package/shingles)
![License: MIT](https://opensource.org/licenses/MIT)

✨ Features

- MinHash signatures for estimating Jaccard similarity between texts
- Locality-Sensitive Hashing (LSH) with banding for fast candidate duplicate detection
- Privacy-preserving mode: Optional per-document salt for blind similarity comparison (same salt → comparable signatures :-> different salt → no leakage)
- Efficient shingling (k-grams on bytes, UTF-8 safe)
- Zero-copy slices where possible for fast WASM ↔ JS interop
- Small footprint WASM binary stays tiny (~500KB–1MB gzipped)
- Offline-first works in browsers, PWAs, even low-end devices

Installation

$3

bash

npm install shingles

or

yarn add shingles

or

pnpm add shingles





Usage (JavaScript / TypeScript)

shingles

 is designed to be used in a background thread (Web Worker) for large datasets, but the API is simple enough for main-thread use on smaller collections.

javascript

import init, { Deduplicator } from 'shingles';



async function main() {

  await init(); 



  // 128 hashes divided into 16 bands (Threshold ~0.7-0.8)

  // use a BigInt for the seed (n suffix required in JS)

  const seed = 42n;

  const dedup = new Deduplicator(128, 16, seed); 



  const privacySalt = "user-secret-key-123";



  // add documents

  const docs = [

    { id: 1, text: "The quick brown fox jumps over the lazy dog." },

    { id: 2, text: "The quick brown fox leaps over the lazy dog!" }, // near dupe

    { id: 3, text: "I love coding in Rust and WebAssembly." }        // different

  ];



  for (const doc of docs) {

    const shingless = Deduplicator.get_shingless(doc.text, 5);

    const signature = dedup.compute_minhash(shingless, privacySalt);

    dedup.add_document(doc.id, signature);

  }



  // query for near-duplicates

  const queryText = "A quick brown fox jumps over a lazy dog.";

  const queryshingless = Deduplicator.get_shingless(queryText, 5);

  const querySig = dedup.compute_minhash(queryshingless, privacySalt);



  // LSH finds candidates in O(1) time instead of O(N)

  const candidates = dedup.query_candidates(querySig);

  console.log("Candidate IDs:", candidates); // Output: [1, 2]



  // fine-grained similarity check

  // compare the query against the most likely candidate (ID: 1)

  const sig1 = dedup.compute_minhash(Deduplicator.get_shingless(docs[0].text, 5), privacySalt);

  const similarity = dedup.estimate_jaccard(querySig, sig1);

  

  console.log(

Estimated Similarity: ${(similarity * 100).toFixed(2)}%

);

}



main();





Why Privacy Salt?



When two parties use the same secret salt, their MinHash signatures become comparable without revealing the original text.  

Different salts produce completely incomparable signatures → ideal for secure, private similarity checks (e.g., two users compare notes without sending plaintext).



Performance



- Tested on 1,000–5,000 short documents: signature computation + indexing < 500ms (browser, mid-range device)

- Scales to 10k+ docs with Web Workers (recommended for heavy use)

- Much faster than pure JS alternatives for large texts



Tuning the "Sensitivity" (S-Curve)



The relationship between Bands, Rows, and Similarity follows an S-curve. By adjusting the parameters in new Deduplicator(hashes, bands, seed), you control how "fuzzy" the matching is.

| Goal        | Hashes | Bands | Threshold (T) |

|-------------|--------|-------|---------------|

| Strict      | 128    | 8     | ≈ 0.84        |

| Balanced    | 128    | 16    | ≈ 0.70        |

| Permissive  | 128    | 32    | ≈ 0.55        |



Formula: $$T≈(1/b)1/r$$ where b is bands and r is rows per band



Building from Source

bash

Install wasm-pack if needed

cargo install wasm-pack



Build for web target

wasm-pack build --target web





Roadmap



- Web Worker helpers for non-blocking heavy operations

- Exact similarity (Jaccard on shingles) only on candidates

- Streaming/large-text support

- Advanced PSI extensions for stronger privacy

- TypeScript types & better docs



Contributing



Contributions welcome! Feel free to open issues or PRs.



1. Fork the repo

2.

cargo build / wasm-pack build`
3. Make changes
4. Submit PR

License

MIT © Preetham Pemmasani

---

If you use this in a project, star the repo and let me know — I'd love to hear about it!
Follow me on X: @ppmpreetham

Installation

bash
npm install shingles

or

yarn add shingles

or

pnpm add shingles

Usage (JavaScript / TypeScript)

is designed to be used in a background thread (Web Worker) for large datasets, but the API is simple enough for main-thread use on smaller collections.

javascript
import init, { Deduplicator } from 'shingles';

async function main() {
await init();

// 128 hashes divided into 16 bands (Threshold ~0.7-0.8)
// use a BigInt for the seed (n suffix required in JS)
const seed = 42n;
const dedup = new Deduplicator(128, 16, seed);

const privacySalt = "user-secret-key-123";

// add documents
const docs = [
{ id: 1, text: "The quick brown fox jumps over the lazy dog." },
{ id: 2, text: "The quick brown fox leaps over the lazy dog!" }, // near dupe
{ id: 3, text: "I love coding in Rust and WebAssembly." } // different
];

for (const doc of docs) {
const shingless = Deduplicator.get_shingless(doc.text, 5);
const signature = dedup.compute_minhash(shingless, privacySalt);
dedup.add_document(doc.id, signature);
}

// query for near-duplicates
const queryText = "A quick brown fox jumps over a lazy dog.";
const queryshingless = Deduplicator.get_shingless(queryText, 5);
const querySig = dedup.compute_minhash(queryshingless, privacySalt);

// LSH finds candidates in O(1) time instead of O(N)
const candidates = dedup.query_candidates(querySig);
console.log("Candidate IDs:", candidates); // Output: [1, 2]

// fine-grained similarity check
// compare the query against the most likely candidate (ID: 1)
const sig1 = dedup.compute_minhash(Deduplicator.get_shingless(docs[0].text, 5), privacySalt);
const similarity = dedup.estimate_jaccard(querySig, sig1);

console.log(

);
}

main();

Why Privacy Salt?

When two parties use the same secret salt, their MinHash signatures become comparable without revealing the original text.
Different salts produce completely incomparable signatures → ideal for secure, private similarity checks (e.g., two users compare notes without sending plaintext).

Performance

- Tested on 1,000–5,000 short documents: signature computation + indexing < 500ms (browser, mid-range device)
- Scales to 10k+ docs with Web Workers (recommended for heavy use)
- Much faster than pure JS alternatives for large texts

Tuning the "Sensitivity" (S-Curve)

The relationship between Bands, Rows, and Similarity follows an S-curve. By adjusting the parameters in new Deduplicator(hashes, bands, seed), you control how "fuzzy" the matching is.
| Goal | Hashes | Bands | Threshold (T) |
|-------------|--------|-------|---------------|
| Strict | 128 | 8 | ≈ 0.84 |
| Balanced | 128 | 16 | ≈ 0.70 |
| Permissive | 128 | 32 | ≈ 0.55 |

Formula: $$T≈(1/b)1/r$$ where b is bands and r is rows per band

Building from Source

bash

Install wasm-pack if needed

cargo install wasm-pack

Build for web target

wasm-pack build --target web

Roadmap

- Web Worker helpers for non-blocking heavy operations
- Exact similarity (Jaccard on shingles) only on candidates
- Streaming/large-text support
- Advanced PSI extensions for stronger privacy
- TypeScript types & better docs

Contributing

Contributions welcome! Feel free to open issues or PRs.

1. Fork the repo
2.