YutiaDB (YDB3)

Ultra-lightweight embedded database optimized for realtime logs, event streams, and AI datasets — with crash-safe append-only storage and fast by-id reads.

> Storage format: YDB3 = binary framing + JSON payload
> ("YDB3" + version + [u32 len][json bytes]...)

Why YutiaDB?

YutiaDB is designed for workloads where writes are massive & continuous, and you need:
- Very fast ingestion (append-only)
- Crash-safety (power loss safe; last partial record is ignored on recovery)
- Small memory overhead (streaming reads, small cache, optional pointer index)
- Easy export to JSON/JSONL for analytics / AI training

This is not a replacement for full-feature query databases (MongoDB/Postgres).
It’s a purpose-built embedded DB for append-heavy use cases.

---

Features

- ✅ Append-only realtime ingestion (framed JSON)
- ✅ Crash-safe recovery (truncated tail-safe)
- ✅ Fast findOne({_id}) via pointer lookup (O(1) per read)
- ✅ Streaming scan for analytics (no full file load)
- ✅ Tombstone deletes (_deleted: true)
- ✅ Lightweight LRU-ish cache for hot records
- ✅ Works great for: logs, telemetry, dataset storage, user DB (by-id)

---

Install (npm)

``bash
npm i yutiadb

or

bun add yutiadb

or

pnpm add yutiadb
``

---

Quick Start

`ts
import { Datastore } from "yutiadb";

const db = new Datastore({
filename: "./data/app.ydb",
autoload: true,

// performance/durability tuning
durability: "batched", // "none" | "batched" | "immediate"
batchBytes: 4 1024 1024,
autoFlushMs: 50,
fsyncEveryFlush: 1,

maxCacheEntries: 1000,
maxPendingDocs: 500_000,
} as any);

// write (realtime)
await db.insert({ type: "req", path: "/api/v1/items", ts: Date.now() });

// read by _id (fast)
const one = await db.findOne({ _id: "..." });

// scan query (stream scan)
const items = await db.find({ type: "req" });

// ensure durability (optional)
await db.flush(); // flush buffered writes
await db.drain(); // wait until all pending buffered writes are written

await db.close();
`

---

Durability Modes

* durability: "none"
Fastest. Data may be in OS buffers; power loss might lose last writes.

* durability: "batched" (recommended)
Writes are buffered and fsync happens periodically (fsyncEveryFlush).

* durability: "immediate"
Safest (fsync every flush). Slowest.

Tip for logs: use "batched" and compact/export offline if needed.

---

Storage Format (YDB3)

File layout:

* Header:

* YDB3 (4 bytes)
* version (1 byte)

* Records:

* len (UInt32LE, 4 bytes)
* payload (UTF-8 JSON bytes)

Recovery behavior:

* If a crash truncates the last record, reader stops safely at the last valid record.

---

Use Cases

$3

* request logs
* audit logs
* webhook logs
* notification logs

$3

* store training samples (JSON)
* export to JSONL later
* sequential scan for training

$3

* fast findOne({_id})
* tombstone deletes
* optional compaction

---

Limitations

* Query engine is intentionally minimal (stream scan + by-id pointer)
* Single-process embedded DB (not a network DB)
* Multi-field secondary indexing is not included (yet)

---

Roadmap (optional)

* [ ] Time-window compaction for logs (keep last N days)
* [ ] Index snapshot .idx for faster startup
* [ ] Tail / live stream tool (tail -f` for YDB)
* [ ] Export JSONL tool

---

License

MIT