jsonl-db

Simple JSONL-based key-value store. Uses an append-only file to store the data. With support for database dumps and compressing the db file.

!Build Status


Usage

Load the module:

``ts
import { DB } from "@alcalzone/jsonl-db";
`

Open or create a database file and use it like a Map

`ts
// Open
const db = new DB("/path/to/file");
await db.open();
// db.isOpen is now true

// and use it
db.set("key", value);
db.delete("key");
db.clear();
if (db.has("key")) {
result = db.get("key");
}
// ...forEach, keys(), entries(), values(), ...
`

$3

If corrupt data is encountered while opening the DB, the call to open() will be rejected. If this is to be expected, use the options parameter on the constructor to turn on forgiving behavior:
`ts
const db = new DB("/path/to/file", { ignoreReadErrors: true });
await db.open();
`
Warning: This may result in inconsistent data since invalid lines are silently ignored.

$3

You can optionally transform the parsed values by passing a reviver function. This allows storing non-primitive objects in the database if those can be transformed to JSON (e.g. by overwriting the toJSON method). To control the transformation values before they are saved to the database, use the serializer function. This is necessary for Maps, Sets, WeakMaps and WeakSets.
`ts
function reviver(key: string, serializedValue: any) {
// MUST return a value. If you don't want to transform serializedValue, return it.
}

function serializer(key: string, value: any) {
// MUST return a value. If you don't want to transform value, return it.
}

const db = new DB("/path/to/file", { reviver, serializer });
await db.open();
`

$3

Data written to the DB is persisted asynchronously. Be sure to call close() when you no longer need the database in order to flush all pending writes and close all files:

`ts
await db.close();
`
Now, db.isOpen is false. While the db is not open, any calls that access the data will throw an error.

$3

By default, the database immediately writes to the database file. You can throttle the write accesses using the throttleFS constructor option. Be aware that buffered data will be lost in case the process crashes.
`ts
const db = new DB("/path/to/file", { throttleFS: { / throttle options / } });
`
The following options exist:
| Option | Default | Description |
|-----------------|---------|-------------|
| intervalMs | 0 | Write to the database file no more than every intervalMs milliseconds. |
| maxBufferedCommands | +Infinity | Force a write after maxBufferedCommands have been buffered. This reduces memory consumption and data loss in case of a crash. |

To create a compressed copy of the database in /path/to/file.dump, use the dump() method. If any data is written to the db during the dump, it is appended to the dump but most likely compressed.

$3

A lockfile is used to avoid concurrent access to the DB file. Multiple options exist to control where this lockfile is created and how it is accessed:
`ts
const db = new DB("/path/to/file", { lockfile: { / lockfile options / } });
`

| Option | Default | Description |
|-----------------|---------|-------------|
| directory | - | Change where the lockfile is created, e.g. to put the lockfile into a tmpfs. By default the lockfile is created in the same directory as the DB file. If the directory does not exist, it will be created when opening the DB. |
| staleMs | 10000 | Duration after which the lock is considered stale. Minimum: 2000 |
| updateMs | staleMs/2 | The interval in which the lockfile's mtime will be updated. Range: 1000 ... staleMs/2 |
| retries | 0 | How often to retry acquiring a lock before giving up. The retries progressively wait longer with an exponential backoff strategy. |
| retryMinTimeoutMs | updateMs/2 or 1000 | The start interval used for retries. Minimum: 100 |




$3

`ts
await db.dump(); // To use the default dump filename /path/to/file.dump
await db.dump("/somewhere/else.jsonl"); // To use a different filename
`

After a while, the main db file may contain unnecessary entries. The raw number of entries can be read using the uncompressedSize property. To remove unnecessary entries, use the compress() method.

`ts
await db.compress();
`

Note: During this call, /path/to/file.dump is overwritten and then renamed, /path/to/file.bak is overwritten and then deleted. So make sure you don't have any important data in these files.

The database can automatically compress the database file under some conditions. To do so, use the autoCompress parameter of the constructor options:
`ts
const db = new DB("/path/to/file", { autoCompress: { / auto compress options / }});
`
The following options exist (all optional) and can be combined:
| Option | Default | Description |
|-----------------|---------|-------------|
| sizeFactor | +Infinity | Compress when uncompressedSize >= size * sizeFactor |
| sizeFactorMinimumSize | 0 | Configure the minimum size necessary for auto-compression based on size |
| intervalMs | +Infinity | Compress after a certain time has passed |
| intervalMinChanges | 1 | Configure the minimum count of changes for auto-compression based on time |
| onClose | false | Compress when closing the DB |
| onOpen | false | Compress after opening the DB |

$3

The DB can automatically keep track of the last time a key was written. To do so, set the enableTimestamps option to true. Every set call will then update the timestamp of a value, which can be retrieved using getTimestamp(key). To opt out of this behavior for individual writes, pass false as the 3rd argument to set.

`ts
db.set("key", "value", false); // Don't update the timestamp
`

> Note
> Recording timestamps decreases the DB throughput by ~10...30% depending on how it is used.

$3

Importing structured data into the database can be done the following way:
`ts
// pass the object directly, the import will be synchronous
db.importJSON({key: "value"});
`
Existing entries in the DB will not be deleted but will be overwritten if they exist.

Exporting the data as a JSON object is also possible:
`ts
const json = db.toJSON();
`

Changelog




$3

* Fixed CommonJS import

$3

* Fixes an issue where the application could run out of memory if a custom serializer threw an error during the automatic compression

$3

* BREAKING: Drop support for Node.js versions older than v20
* BREAKING: Removed exportJson method in favor of toJSON
* BREAKING: Renamed importJson method to importJSON. Importing from a file is no longer supported. Read the file yourself if you need this functionality.
* Filesystem access is now done using the native fs.promises instead of fs-extra
* The package is now a hybrid ESM/CJS package

$3

* Reduced CPU load while idle (#475)

$3

* Support opt-out of updating timestamps for individual set calls (#415)

$3

* Support recording timestamps for each value (#413)
* Drop support for Node.js 12 (#414)

$3

* Consider lockfiles from the future to be stale (#371)

$3

* Dependency updates

$3

* Allow setting minimum retry timeout
* Correct minimum value for the lockfile's stale timeout

$3

* Add the ability to control the lockfile's stale/update timeouts and retrying acquiring a lock

$3

* Errors while automatically compressing the DB are now caught

$3

* Errors while automatically restoring the DB from a backup or dump are now caught
* Simplified and decoupled the persistence code. Individual commands like dump and compress are now properly sequenced and should no longer conflict with each other.
* Increased throughput for primitive entries by ~2x

$3

* Individual writes are now collected in a string and written at once, increasing throughput for larger entries by ~10x.

$3

* Stringifying the individual lines now happens lazily and only when actually necessary, increasing the throughput by 30...50x.

$3

* Add the ability to dump the database to a different location

$3

* Add the ability to specify where the lockfile is created

$3

* When opening the DB, recover from crashes that happened while compressing the DB
* Ensure that the DB files are flushed to disk when closing or renaming files
* Tests now work with the real filesystem instead of mock-fs, which breaks in Node.js 16.3+
* Modernized TypeScript build output

$3

Update dependencies and drop support for Node.js 10

$3

Prevent opening one DB file in multiple instances of the DB using lockfiles

$3

Reduced the work done while opening a DB

$3

Fixed a crash that happens while compressing the DB when the .bak file exists

$3

When consuming this library without skipLibCheck, @types/fs-extra is no longer required

$3

Update dependencies

$3

Added an optional serializer function to transform non-primitive objects before writing to the DB file

$3

Fixed a timeout leak that would prevent Node.js from exiting

$3

Leading directories are now created if they don't exist

$3

Added functionality to throttle write accesses

$3

Export JsonlDBOptions from the main entry point

$3

Added auto-compress functionality

$3

Fix: The main export no longer exports JsonlDB as DB.

$3

Added an optional reviver function to transform non-primitive objects while loading the DB

$3

* Renamed the DB class to JsonlDB
* open() now skips empty lines
* open() throws an error with the line number when it encounters an invalid line. These errors can be ignored using the new constructor options argument.

$3

* Added importJson and exportJson methods

$3

* Added isOpen property

$3

* Writes that happen while compress() replaces files are now persisted

$3

* compress()` no longer overwrites the main file while the DB is being closed

$3

* Fixed some race conditions

$3

First official release