Kestrel

A distributed, k-sortable unique ID generation system using Redis and Lua, inspired by Icicle.

alt='Coverage Status' height="18"/>

Overview

Kestrel is for teams that want globally unique, time-sortable IDs in a Redis-backed system and want those IDs to be useful later in analytics.

It’s a good fit for:

- API + data platform teams: generate IDs at write-time, then decode them in your warehouse/ETL jobs for time-based partitioning and debugging.
- Distributed systems: multiple app instances can generate IDs without database round-trips, while preserving ordering by creation time.

What you get:

- K-sortable 64-bit IDs: IDs sort by creation time (milliseconds since a custom epoch).
- Embedded metadata: each ID encodes:
- timestamp (ms)
- logical shard id
- per-millisecond sequence (0–4095)
- Decoding utilities: Kestrel.decodeId() / Kestrel.decodeIds() to extract timestamp/shard/sequence for:
- data warehousing (e.g., derive created_at, partition keys, time-window rollups)
- ETL/observability (e.g., detect hot shards, analyze write bursts, investigate gaps)

See examples/ for decode-focused scripts you can drop into warehouse/ETL workflows.

Getting started

$3

- Node.js v20+ (ES modules support required)
- Redis v7+ (standalone), or Redis Sentinel / Redis Cluster (optional modes)
- Docker + Docker Compose (for the included local Sentinel/Cluster test stacks)

``sh


Initialize the repo (if applicable)

`sh
npm install @dreamystify/kestrel
`

To build the package locally with esbuild, run:

`sh
ahoy build
`

ES Modules Support

This package is built as an ES module and requires Node.js v18+ with ES modules
support. When using this package:

- Use import statements instead of require()
- Ensure your package.json has "type": "module" or use .mjs file
extensions
- The package exports are optimized for ES module resolution

Usage

The Kestrel library supports three connection modes: standalone, Sentinel, and
Cluster. Below are examples for each mode.

$3

`javascript
import { Kestrel } from '@dreamystify/kestrel';

(async () => {
// Initialize Kestrel with standalone Redis
const kestrel = await Kestrel.initialize({
host: 'localhost',
port: 6379,
username: 'yourUsername', // if using authentication
password: 'yourPassword', // if using authentication
database: 0,
});

// Generate a single unique ID
const id = await kestrel.getId();
console.log('Generated ID:', id);

// Generate a batch of 3 IDs
const ids = await kestrel.getIds(3);
console.log('Generated IDs:', ids);
})();
`

$3

`javascript
import { Kestrel } from '@dreamystify/kestrel';

(async () => {
// Initialize Kestrel using Redis Sentinel
const kestrel = await Kestrel.initialize({
sentinels: [
{ host: 'sentinel1', port: 26379 },
{ host: 'sentinel2', port: 26379 },
],
name: 'mymaster', // name of your master instance
username: 'yourUsername', // for the master
password: 'yourPassword', // for the master
sentinelUsername: 'sentinelUser', // if your Sentinel requires authentication
sentinelPassword: 'sentinelPassword', // if your Sentinel requires authentication
database: 0,
});

const id = await kestrel.getId();
console.log('Generated ID:', id);
})();
`

$3

`javascript
import { Kestrel } from '@dreamystify/kestrel';

(async () => {
// Initialize Kestrel using Redis Cluster
const kestrel = await Kestrel.initialize({
clusterNodes: [
{ host: 'redis-cluster-node1', port: 6379 },
{ host: 'redis-cluster-node2', port: 6379 },
{ host: 'redis-cluster-node3', port: 6379 },
],
username: 'yourUsername', // for cluster authentication
password: 'yourPassword', // for cluster authentication
database: 0,
});

const id = await kestrel.getId();
console.log('Generated ID:', id);
})();
`

$3

`javascript
import { Kestrel } from '@dreamystify/kestrel';

(async () => {
// Initialize Kestrel with a custom shard configuration (for standalone mode)
const kestrel = await Kestrel.initialize({
host: 'localhost',
port: 6379,
username: 'yourUsername',
password: 'yourPassword',
database: 0,
// Optionally set a fixed shard ID via environment variable:
// KESTREL_SHARD_ID_KEY: '{kestrel}-shard-id',
// KESTREL_SHARD_ID: '1',
});

const id = await kestrel.getId();
console.log('Generated ID:', id);
})();
`

In Cluster mode, the library automatically assigns shard IDs to master nodes
based on the cluster topology. You can later query each node (using redis-cli)
to see the assigned shard IDs:

`sh
docker run -it --rm --network redis_cluster redis:7.0.2 redis-cli -a yourPassword --cluster call redis-cluster-node1:6379 GET '{kestrel}-shard-id'
`

$3

Kestrel IDs can be decoded back into their component parts, allowing you to
extract information like when the ID was created, which shard generated it,
and the sequence number.

`javascript
import { Kestrel } from '@dreamystify/kestrel';

(async () => {
const kestrel = await Kestrel.initialize({
host: 'localhost',
port: 6379,
username: 'yourUsername',
password: 'yourPassword',
});

// Generate an ID
const id = await kestrel.getId();

// Decode a single ID
const decoded = Kestrel.decodeId(id);
console.log('Timestamp (ms since Unix epoch):', decoded.timestampMs);
console.log('Timestamp (ms since custom epoch):', decoded.timestamp);
console.log('Logical Shard ID:', decoded.logicalShardId); // 0-1023
console.log('Sequence:', decoded.sequence); // 0-4095
console.log('Created At:', decoded.createdAt); // JavaScript Date object

// Decode multiple IDs
const ids = await kestrel.getIds(5);
const decodedIds = Kestrel.decodeIds(ids);
decodedIds.forEach((d, index) => {
console.log(ID ${index + 1}:);
console.log( Created: ${d.createdAt.toISOString()});
console.log( Shard: ${d.logicalShardId}, Sequence: ${d.sequence});
});
})();
`

The decodeId method accepts IDs as bigint, string, or number, making it
easy to decode IDs from various sources (databases, APIs, etc.).

Error Handling

Kestrel emits events for errors. It is designed to let your application handle
logging and error processing in a way that suits your needs. For example:

`javascript
kestrel.on('error', error => {
console.error('Kestrel error:', error.error);
});

kestrel.on('connect', () => {
console.log('Kestrel connected to Redis');
});
`

Events

| Event Constant | Event String | Description |
| ------------------------------- | ----------------------------- | ------------------------------------------------------------------------------------------------- |
| CLIENT_CREATED | clientCreated | Emitted when the Redis client instance is successfully created. |
| CONNECTED | connected | Emitted when the client has successfully connected to Redis. |
| SCRIPT_LOADED | scriptLoaded | Emitted when a Lua script is successfully loaded on a Redis node. |
| NODE_ADDED | nodeAdded | Emitted when a new node is detected in a cluster. |
| NODE_REMOVED | nodeRemoved | Emitted when a node is removed from a cluster. |
| CLUSTER_NODE_ADDED | +node | Emitted when a new cluster node is added (internal cluster topology event). |
| CLUSTER_NODE_REMOVED | -node | Emitted when a cluster node is removed (internal cluster topology event). |
| ERROR | error | Emitted when an error occurs within the Kestrel library. |
| Redis Connection Events | | |
| CONNECT | connect | Emitted when a connection is established. |
| CONNECTING | connecting | Emitted when the client is attempting to establish a connection. |
| RECONNECTING | reconnecting | Emitted when the client is attempting to reconnect after a disconnect or error. |
| DISCONNECTED | disconnected | Emitted when the client has been disconnected. |
| WAIT | wait | Emitted when the client is waiting (typically during retry/backoff). |
| READY | ready | Emitted when the client is ready to accept commands. |
| CLOSE | close | Emitted when the connection is closed. |
| END | end | Emitted when the connection has ended. |
| RECONNECTED | reconnected | Emitted when the client has successfully reconnected. |
| RECONNECTION_ATTEMPTS_REACHED | reconnectionAttemptsReached | Emitted when the maximum number of reconnection attempts is reached and no further retries occur. |

Testing

`sh


Start the testing environment

`sh


Start Redis containers (standalone + sentinel + cluster) and wait until they're ready

Kudos

The project was inspired by Icicle, just
setup for node.js.