Bigmap for Keyv
npm install @keyv/bigmap> Bigmap for Keyv




djb2Hash for fast key lookups.@cacheable/memory for scalable in-memory caching.``bash`
npm install --save keyv @keyv/bigmap
BigMap is a scalable Map implementation that overcomes JavaScript's built-in Map limit of approximately 17 million entries. It uses a distributed hash approach with multiple internal Map instances.
`typescript
import { BigMap } from '@keyv/bigmap';
// Create a new BigMap
const bigMap = new BigMap
// Set values
bigMap.set('key1', 100);
bigMap.set('key2', 200);
// Get values
const value = bigMap.get('key1'); // 100
// Check if key exists
bigMap.has('key1'); // true
// Delete a key
bigMap.delete('key1'); // true
// Get size
console.log(bigMap.size); // 1
// Clear all entries
bigMap.clear();
`
By default, BigMap uses 4 internal Map instances. You can configure this:
`typescript`
const bigMap = new BigMap
Note: Changing the storeSize after initialization will clear all entries.
Provide your own hash function for key distribution:
`typescript
const customHashFunction = (key: string, storeSize: number) => {
return key.length % storeSize;
};
const bigMap = new BigMap
storeHashFunction: customHashFunction
});
`
Hashery is a powerful hashing library that provides multiple hash algorithms. You can use it for better key distribution and it is available as an export:
`typescript
import { BigMap, Hashery } from '@keyv/bigmap';
const hashery = new Hashery();
// Using Hashery's toNumberSync for deterministic key distribution
const bigMap = new BigMap
storeHashFunction: (key: string, storeSize: number) => {
return hashery.toNumberSync(key, { min: 0, max: storeSize - 1 });
}
});
// You can also use different algorithms
const hasheryFnv1 = new Hashery({ defaultAlgorithmSync: 'fnv1' });
const bigMapWithFnv1 = new BigMap
storeHashFunction: (key: string, storeSize: number) => {
return hasheryFnv1.toNumberSync(key, { min: 0, max: storeSize - 1 });
}
});
`
Hashery supports multiple synchronous hash algorithms:
- djb2 - Fast hash function (default)
- fnv1 - Excellent distribution for hash tables
- murmer - MurmurHash algorithm
- crc32 - Cyclic Redundancy Check
BigMap supports all standard Map iteration methods:
`typescript
const bigMap = new BigMap
bigMap.set('a', 1);
bigMap.set('b', 2);
for (const [key, value] of bigMap) {
console.log(key, value);
}
`
`typescript
bigMap.forEach((value, key) => {
console.log(key, value);
});
// With custom context
const context = { sum: 0 };
bigMap.forEach(function(value) {
this.sum += value;
}, context);
`
`typescript
// Iterate over keys
for (const key of bigMap.keys()) {
console.log(key);
}
// Iterate over values
for (const value of bigMap.values()) {
console.log(value);
}
// Iterate over entries
for (const [key, value] of bigMap.entries()) {
console.log(key, value);
}
`
`typescript
interface User {
id: number;
name: string;
}
const userMap = new BigMap
userMap.set('user1', { id: 1, name: 'Alice' });
`
BigMap is designed to handle millions of entries:
`typescript
const bigMap = new BigMap
// Add 20+ million entries without hitting Map limits
for (let i = 0; i < 20000000; i++) {
bigMap.set(key${i}, i);
}
console.log(bigMap.size); // 20000000
`
BigMap can be used as a storage adapter for Keyv, providing a scalable in-memory store with TTL support.
The createKeyv function creates a Keyv instance with BigMap as the storage adapter.
Parameters:
- options (optional): BigMap configuration optionsstoreSize
- (number): Number of internal Map instances. Default: 4storeHashFunction
- (StoreHashFunction): Custom hash function for key distribution
Returns: Keyv instance with BigMap adapter
Example:
`typescript
import { createKeyv } from '@keyv/bigmap';
// Basic usage
const keyv = createKeyv();
// Set with TTL (in milliseconds)
await keyv.set('user:123', { name: 'Alice', age: 30 }, 60000); // Expires in 60 seconds
// Get value
const user = await keyv.get('user:123');
console.log(user); // { name: 'Alice', age: 30 }
// Check if key exists
const exists = await keyv.has('user:123');
// Delete key
await keyv.delete('user:123');
// Clear all keys
await keyv.clear();
`
`typescript
import { createKeyv } from '@keyv/bigmap';
// Create with custom store size for better performance with millions of keys
const keyv = createKeyv({ storeSize: 16 });
// With custom hash function
const keyv = createKeyv({
storeSize: 8,
storeHashFunction: (key, storeSize) => {
// Custom distribution logic
return key.length % storeSize;
}
});
`
`typescript
import { createKeyv } from '@keyv/bigmap';
interface Product {
id: string;
name: string;
price: number;
}
const keyv = createKeyv
await keyv.set('product:1', {
id: '1',
name: 'Laptop',
price: 999
});
const product = await keyv.get
`
BigMap works seamlessly with the Keyv ecosystem:
`typescript
import { createKeyv } from '@keyv/bigmap';
const cache = createKeyv({ storeSize: 16 });
// Use with namespaces
const users = cache.namespace('users');
const products = cache.namespace('products');
await users.set('123', { name: 'Alice' });
await products.set('456', { name: 'Laptop' });
// Iterate over keys
for await (const [key, value] of cache.iterator()) {
console.log(key, value);
}
`
new BigMap
Creates a new BigMap instance.
Parameters:
- options (optional): Configuration optionsstoreSize
- (number): Number of internal Map instances to use. Default: 4. Must be at least 1.storeHashFunction
- (StoreHashFunction): Custom hash function for key distribution. Default: defaultHashFunction
Example:
`typescript`
const bigMap = new BigMap
const customBigMap = new BigMap
storeSize: 10,
storeHashFunction: (key, storeSize) => key.length % storeSize
});
| Property | Type | Access | Description |
|----------|------|--------|-------------|
| size | number | Read-only | Gets the total number of entries in the BigMap. |storeSize
| | number | Read/Write | Gets or sets the number of internal Map instances. Note: Setting this will clear all entries. Default: 4 |storeHashFunction
| | StoreHashFunction \| undefined | Read/Write | Gets or sets the hash function used for key distribution. |store
| | Array
Examples:
`typescript
const bigMap = new BigMap
// size property
bigMap.set('key1', 100);
console.log(bigMap.size); // 1
// storeSize property
console.log(bigMap.storeSize); // 4 (default)
bigMap.storeSize = 8; // Changes size and clears all entries
// storeHashFunction property
bigMap.storeHashFunction = (key, storeSize) => key.length % storeSize;
// store property
console.log(bigMap.store.length); // 8
`
Sets the value for a key in the map.
Parameters:
- key (K): The key to setvalue
- (V): The value to associate with the key
Returns: Map - The internal Map instance where the key was stored
Example:
`typescript`
bigMap.set('user123', { name: 'Alice' });
Gets the value associated with a key.
Parameters:
- key (K): The key to retrieve
Returns: V | undefined - The value, or undefined if not found
Example:
`typescript`
const value = bigMap.get('user123');
Checks if a key exists in the map.
Parameters:
- key (K): The key to check
Returns: boolean - True if the key exists, false otherwise
Example:
`typescript`
if (bigMap.has('user123')) {
console.log('User exists');
}
Deletes a key-value pair from the map.
Parameters:
- key (K): The key to delete
Returns: boolean - True if the entry was deleted, false if the key was not found
Example:
`typescript`
const deleted = bigMap.delete('user123');
Removes all entries from the map.
Returns: void
Example:
`typescript`
bigMap.clear();
console.log(bigMap.size); // 0
Executes a provided function once for each key-value pair.
Parameters:
- callbackfn (function): Function to execute for each entryvalue
- (V): The value of the current entrykey
- (K): The key of the current entrymap
- (Map): The BigMap instancethisArg
- (optional): Value to use as this when executing the callback
Returns: void
Example:
`typescript${key}: ${value}
bigMap.forEach((value, key) => {
console.log();
});
// With custom context
const context = { total: 0 };
bigMap.forEach(function(value) {
this.total += value;
}, context);
`
Returns an iterator of all keys in the map.
Returns: IterableIterator
Example:
`typescript`
for (const key of bigMap.keys()) {
console.log(key);
}
Returns an iterator of all values in the map.
Returns: IterableIterator
Example:
`typescript`
for (const value of bigMap.values()) {
console.log(value);
}
Returns an iterator of all key-value pairs in the map.
Returns: IterableIterator<[K, V]>
Example:
`typescript`
for (const [key, value] of bigMap.entries()) {
console.log(key, value);
}
Returns an iterator for the map (same as entries()). Enables for...of loops.
Returns: IterableIterator<[K, V]>
Example:
`typescript`
for (const [key, value] of bigMap) {
console.log(key, value);
}
Gets the internal Map instance for a specific key.
Parameters:
- key (K): The key to find the store for
Returns: Map - The internal Map instance
Example:
`typescript`
const store = bigMap.getStore('user123');
Gets the internal Map instance at a specific index.
Parameters:
- index (number): The index of the Map to retrieve (0 to storeSize - 1)
Returns: Map - The Map at the specified index
Throws: Error if index is out of bounds
Example:
`typescript`
const firstMap = bigMap.getStoreMap(0);
Initializes the internal store with empty Map instances. Called automatically during construction.
Returns: void
Type definition for custom hash functions.
`typescript`
type StoreHashFunction = (key: string, storeSize: number) => number;
Parameters:
- key (string): The key to hash (converted to string)storeSize
- (number): The number of stores (adjusted for zero-based index)
Returns: number - The index of the store to use (0 to storeSize - 1)
The default hash function using DJB2 algorithm from Hashery:
Example:
`typescript
import { defaultHashFunction } from '@keyv/bigmap';
const index = defaultHashFunction('myKey', 4);
`
DJB2 hash algorithm implementation.
Parameters:
- string (string): The string to hashmin
- (number): Minimum value. Default: 0max
- (number): Maximum value. Default: 10
Returns: number - Hash value within the specified range
Example:
`typescript
import { djb2Hash } from '@keyv/bigmap';
const hash = djb2Hash('myKey', 0, 10);
``
Please see our contributing guide.