🗃️ s3db.js

Transform AWS S3 into a powerful document database

Cost-effective storage • Automatic encryption • ORM-like interface • Streaming API

 

 

 

 

 

 

📖 Full Documentation

🚀 What is s3db.js?

s3db.js is a document database that transforms AWS S3 into a fully functional database using S3's metadata capabilities. Instead of traditional storage methods, it stores document data in S3's metadata fields (up to 2KB), making it highly cost-effective while providing a familiar ORM-like interface.

Perfect for:
- 🌐 Serverless applications - No database servers to manage
- 💰 Cost-conscious projects - Pay only for what you use
- 🔒 Secure applications - Built-in encryption and validation
- 📊 Analytics platforms - Efficient data streaming and processing
- 🚀 Rapid prototyping - Get started in minutes, not hours

---

✨ Key Features

$3 - ORM-like Interface - Familiar CRUD operations - Schema Validation - Automatic data validation - Schema Registry - Stable attribute mapping across schema changes - Streaming API - Handle large datasets efficiently - Event System - Real-time notifications	$3 - Field-level Encryption - Secure sensitive data - Intelligent Caching - Reduce API calls - Concurrency Control - Optimized bulk operations - Auto-generated Passwords - Secure by default - Cost Tracking - Monitor AWS expenses
$3 - Partitions - Organize data efficiently - Bulk Operations - Handle multiple records - Nested Objects - Complex data structures - Automatic Timestamps - Track changes	$3 - Custom Behaviors - Handle large documents - Hooks System - Custom business logic - Plugin Architecture - Extend functionality - Event System - Real-time notifications

---

📋 Table of Contents

- 🚀 What is s3db.js?
- ✨ Key Features
- 🚀 Quick Start
- 💾 Installation
- 🗄️ Database
- 🪵 Logging
- 📋 Resources
- ⚡ Performance & Concurrency
- 🔌 Plugins
- 🤖 MCP & Integrations
- 🔧 CLI
- 📖 Documentation

---

📚 Documentation Quick Links

> Core Concepts: Schema & Validation • Schema Registry • Clients • Fastest Validator

> Plugins: API Plugin • Identity Plugin • All Plugins

> Guides: Path-based Basic + OIDC Migration

> Integrations: MCP Server • Model Context Protocol

> Advanced: Executor Pool Benchmark • Performance Tuning • Migration Guides • TypeScript Support

---

🚀 Quick Start

Get up and running in less than 5 minutes!

$3

``bash
npm install s3db.js
`

> Need deeper telemetry? Pass taskExecutorMonitoring alongside executorPool. It merges into the pool's monitoring block, making it easy to enable verbose stats/heap tracking for any database instance without touching individual resources.

$3

`javascript
import { S3db } from "s3db.js";

const s3db = new S3db({
connectionString: "s3://ACCESS_KEY:SECRET_KEY@BUCKET_NAME/databases/myapp"
});

await s3db.connect();
console.log("🎉 Connected to S3 database!");
`

> ⚡ Performance Tip: s3db.js comes with optimized HTTP client settings by default for excellent S3 performance. The default configuration includes keep-alive enabled, balanced connection pooling, and appropriate timeouts for most applications.

> ℹ️ Note: You do not need to provide ACCESS_KEY and SECRET_KEY in the connection string if your environment already has S3 permissions (e.g., via IAM Role on EKS, EC2, Lambda, or other compatible clouds). s3db.js will use the default AWS credential provider chain, so credentials can be omitted for role-based or environment-based authentication. This also applies to S3-compatible clouds (MinIO, DigitalOcean Spaces, etc.) if they support such mechanisms.

---

$3

Schema validation powered by fastest-validator ⚡

`javascript
const users = await s3db.createResource({
name: "users",
attributes: {
name: "string|min:2|max:100",
email: "email|unique",
age: "number|integer|positive",
isActive: "boolean"
},
timestamps: true
});
`

$3

`javascript
// Insert a user
const user = await users.insert({
name: "John Doe",
email: "john@example.com",
age: 30,
isActive: true,
createdAt: new Date()
});

// Query the user
const foundUser = await users.get(user.id);
console.log(Hello, ${foundUser.name}! 👋);

// Update the user
await users.update(user.id, { age: 31 });

// List all users
const allUsers = await users.list();
console.log(Total users: ${allUsers.length});
`

That's it! You now have a fully functional document database running on AWS S3. 🎉

$3

Enhance your database with powerful plugins for production-ready features. All plugins are available from the main s3db.js package:

`javascript
import { S3db } from "s3db.js";
import { RelationPlugin, TTLPlugin, ReplicatorPlugin, CachePlugin } from "s3db.js";

const s3db = new S3db({
connectionString: "s3://ACCESS_KEY:SECRET_KEY@BUCKET_NAME/databases/myapp",
plugins: [
// Auto-cleanup expired records (no cron jobs needed!)
new TTLPlugin({
resources: {
sessions: { ttl: 86400, onExpire: 'soft-delete' } // 24h
}
}),

// ORM-like relationships with 10-100x faster queries
new RelationPlugin({
relations: {
users: {
posts: { type: 'hasMany', resource: 'posts', foreignKey: 'userId' }
}
}
}),

// Real-time replication to BigQuery, PostgreSQL, etc.
new ReplicatorPlugin({
replicators: [{
driver: 'bigquery',
config: { projectId: 'my-project', datasetId: 'analytics' },
resources: { users: 'users_table', posts: 'posts_table' }
}]
}),

// Cache frequently accessed data (memory, S3, or filesystem)
new CachePlugin({
driver: 'memory',
ttl: 300000 // 5 minutes
})
]
});
`

Learn more about available plugins and their features in the Plugin Documentation.

---

💾 Installation

$3

`bash


npm

npm install s3db.js

pnpm

pnpm add s3db.js

yarn

yarn add s3db.js
`

$3

Some features require additional dependencies to be installed manually:

#### API Plugin Dependencies

If you plan to use the API plugin, install these dependencies:

`bash


Core API dependencies (required)

npm install hono

HTTP logging (optional, recommended)

npm install pino-http

Authentication (optional)

npm install jose # For JWT auth
`

#### Replicator Dependencies

If you plan to use the replicator system with external services, install the corresponding dependencies:

`bash


For SQS replicator (AWS SQS queues)

npm install @aws-sdk/client-sqs

For BigQuery replicator (Google BigQuery)

npm install @google-cloud/bigquery

For PostgreSQL replicator (PostgreSQL databases)

npm install pg
`

Why manual installation? These are marked as peerDependencies to keep the main package lightweight (~500KB). Only install what you need!

#### 🛠️ Development Setup

Contributing to s3db.js? Use our modular installation system to install only what you need:

`bash


Clone the repo

git clone https://github.com/forattini-dev/s3db.js.git
cd s3db.js

Install base dependencies (required)

pnpm install

Choose your dev setup:

./scripts/install-deps.sh minimal # Core only (~50MB)
./scripts/install-deps.sh common # + Replicators + Plugins (~500MB)
./scripts/install-deps.sh full # Everything (~2GB)

Or install specific groups:

pnpm run install:dev:replicators # PostgreSQL, BigQuery, etc.
pnpm run install:dev:plugins # API, Identity, ML, etc.
pnpm run install:dev:puppeteer # Web scraping suite
pnpm run install:dev:cloud # AWS SDK clients
`

See DEVELOPMENT.md for detailed setup instructions and dependency groups breakdown.

$3

s3db.js includes comprehensive TypeScript definitions out of the box. Get full type safety, autocomplete, and IntelliSense support in your IDE!

#### Basic Usage (Automatic Types)

`typescript
import { Database, DatabaseConfig, Resource } from 's3db.js';

// Type-safe configuration
const config: DatabaseConfig = {
connectionString: 's3://ACCESS_KEY:SECRET@bucket/path',
logLevel: 'debug',
executorPool: { concurrency: 100 } // Default - nested under executorPool
};

const db = new Database(config);

// TypeScript knows all methods and options!
await db.createResource({
name: 'users',
attributes: {
name: 'string|required',
email: 'string|required|email',
age: 'number|min:0'
}
});

// Full autocomplete for all operations
const users: Resource = db.resources.users;
const user = await users.insert({ name: 'Alice', email: 'alice@example.com', age: 28 });
`

#### Advanced: Generate Resource Types

For even better type safety, auto-generate TypeScript interfaces from your resources:

`typescript
import { generateTypes } from 's3db.js/typescript-generator';

// Generate types after creating resources
await generateTypes(db, { outputPath: './types/database.d.ts' });
`

See the complete example in docs/examples/typescript-usage-example.ts.

---

🗄️ Database

A Database is a logical container for your resources, stored in a specific S3 bucket path. The database manages resource metadata, connections, and provides the core interface for all operations.

$3

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| connectionString | string | required | S3 connection string (see formats below) |
| httpClientOptions | object | optimized | HTTP client configuration for S3 requests |
| logLevel | boolean | false | Enable debug logging for debugging |
| parallelism | number | 100 | Concurrent operations for bulk operations (Separate Executor Pools per Database) |
| versioningEnabled | boolean | false | Enable automatic resource versioning |
| passphrase | string | 'secret' | Default passphrase for field encryption |
| plugins | array | [] | Array of plugin instances to extend functionality |

$3

s3db.js supports multiple connection string formats for different S3 providers:

`javascript
// AWS S3 (with credentials)
"s3://ACCESS_KEY:SECRET_KEY@BUCKET_NAME/databases/myapp?region=us-east-1"

// AWS S3 (IAM role - recommended for production)
"s3://BUCKET_NAME/databases/myapp?region=us-east-1"

// MinIO (self-hosted)
"http://minioadmin:minioadmin@localhost:9000/bucket/databases/myapp"

// Digital Ocean Spaces
"https://SPACES_KEY:SPACES_SECRET@nyc3.digitaloceanspaces.com/SPACE_NAME/databases/myapp"

// LocalStack (local testing)
"http://test:test@localhost:4566/mybucket/databases/myapp"

// MemoryClient (ultra-fast in-memory testing - no S3 required!)
"memory://mybucket/databases/myapp"

// Backblaze B2
"https://KEY_ID:APPLICATION_KEY@s3.us-west-002.backblazeb2.com/BUCKET/databases/myapp"

// Cloudflare R2
"https://ACCESS_KEY:SECRET_KEY@ACCOUNT_ID.r2.cloudflarestorage.com/BUCKET/databases/myapp"
`

🔑 Complete authentication examples

#### 1. Access Keys (Development)
`javascript
const s3db = new S3db({
connectionString: "s3://ACCESS_KEY:SECRET_KEY@BUCKET_NAME/databases/myapp"
});
`

#### 2. IAM Roles (Production - Recommended)
`javascript
// No credentials needed - uses IAM role permissions
const s3db = new S3db({
connectionString: "s3://BUCKET_NAME/databases/myapp"
});
`

#### 3. MinIO (Self-hosted S3)
`javascript
// MinIO running locally (note: http:// protocol and port)
const s3db = new S3db({
connectionString: "http://minioadmin:minioadmin@localhost:9000/mybucket/databases/myapp"
});
`

#### 4. Digital Ocean Spaces (SaaS)
`javascript
// Digital Ocean Spaces (NYC3 datacenter)
const s3db = new S3db({
connectionString: "https://SPACES_KEY:SPACES_SECRET@nyc3.digitaloceanspaces.com/SPACE_NAME/databases/myapp"
});
`

---

$3

For testing, s3db.js provides MemoryClient - a pure in-memory implementation that's 100-1000x faster than LocalStack and requires zero dependencies.

Why MemoryClient?
- ⚡ 100-1000x faster than LocalStack/MinIO
- 🎯 Zero dependencies - no Docker, LocalStack, or S3 needed
- 💯 100% compatible - same API as S3Client
- 🧪 Perfect for tests - instant setup and teardown
- 💾 Optional persistence - save/load snapshots to disk

Quick Start with Connection String:

`javascript
import { S3db } from 's3db.js';

// Simple - just use memory:// protocol!
const db = new S3db({
connectionString: 'memory://mybucket'
});

await db.connect();
`

Alternative - Manual Instantiation:

`javascript
import { S3db, MemoryClient } from 's3db.js';

// Create database with MemoryClient
const db = new S3db({
client: new MemoryClient({ bucket: 'test-bucket' })
});

await db.connect();

// Use exactly like S3 - same API!
const users = await db.createResource({
name: 'users',
attributes: {
name: 'string|required',
email: 'email|required'
}
});

await users.insert({ id: 'u1', name: 'John', email: 'john@test.com' });
const user = await users.get('u1');
`

Connection String Options:

`javascript
// Basic usage
"memory://mybucket"

// With key prefix (path)
"memory://mybucket/databases/myapp"

// With multiple path segments
"memory://testdb/level1/level2/level3"

// With query parameters
"memory://mybucket?region=us-west-2"
`

Advanced Features (Manual Client):

`javascript
import { S3db, MemoryClient } from 's3db.js';

// Option 1: Connection string (recommended)
const db1 = new S3db({
connectionString: 'memory://test-bucket/tests/'
});

// Option 2: Manual client configuration
const db2 = new S3db({
client: new MemoryClient({
bucket: 'test-bucket',
keyPrefix: 'tests/', // Optional prefix for all keys
enforceLimits: true, // Enforce S3 2KB metadata limit
persistPath: './test-data.json', // Optional: persist to disk
logLevel: 'silent' // Disable logging
})
});

// Snapshot/Restore (perfect for tests)
const snapshot = client.snapshot(); // Capture current state
// ... run tests that modify data ...
client.restore(snapshot); // Restore to original state

// Persistence
await client.saveToDisk(); // Save to persistPath
await client.loadFromDisk(); // Load from persistPath

// Statistics
const stats = client.getStats();
console.log(Objects: ${stats.objectCount}, Size: ${stats.totalSizeFormatted});

// Clear all data
client.clear();
`

Testing Example:

`javascript
import {
describe,
test,
beforeEach,
afterEach } from '@jest/globals';
import { S3db
} from 's3db.js';

describe('User Tests', () => {
let db, users, snapshot;

beforeEach(async () => {
// Simple connection string setup!
db = new S3db({
connectionString: 'memory://test-db/my-tests'
});
await db.connect();

users = await db.createResource({
name: 'users',
attributes: { name: 'string', email: 'email' }
});

// Save snapshot for each test
snapshot = db.client.snapshot();
});

afterEach(() => {
// Restore to clean state (faster than recreating)
db.client.restore(snapshot);
});

test('should insert user', async () => {
await users.insert({ id: 'u1', name: 'John', email: 'john@test.com' });
const user = await users.get('u1');
expect(user.name).toBe('John');
});
});
`

Performance Comparison:

| Operation | LocalStack | MemoryClient | Speedup |
|-----------|------------|--------------|---------|
| Insert 100 records | ~2000ms | ~50ms | 40x faster |
| Query 1000 records | ~5000ms | ~100ms | 50x faster |
| Full test suite | ~120s | ~2s | 60x faster |

📚 Full MemoryClient Documentation

---

$3

When you create a database, s3db.js organizes your data in a structured way within your S3 bucket:

`
bucket-name/
└── databases/
└── myapp/ # Database root (from connection string)
├── s3db.json # Database metadata & resource definitions
│
├── resource=users/ # Resource: users
│ ├── data/
│ │ ├── id=user-123 # Document (metadata in S3 metadata, optional body)
│ │ └── id=user-456
│ └── partition=byRegion/ # Partition: byRegion
│ ├── region=US/
│ │ ├── id=user-123 # Partition reference
│ │ └── id=user-789
│ └── region=EU/
│ └── id=user-456
│
├── resource=posts/ # Resource: posts
│ └── data/
│ ├── id=post-abc
│ └── id=post-def
│
├── resource=sessions/ # Resource: sessions (with TTL)
│ └── data/
│ ├── id=session-xyz
│ └── id=session-qwe
│
├── plugin=cache/ # Plugin: CachePlugin (global data)
│ ├── config # Plugin configuration
│ └── locks/
│ └── cache-cleanup # Distributed lock
│
└── resource=wallets/ # Resource: wallets
├── data/
│ └── id=wallet-123
└── plugin=eventual-consistency/ # Plugin: scoped to resource
├── balance/
│ └── transactions/
│ └── id=txn-123 # Plugin-specific data
└── locks/
└── balance-sync # Resource-scoped lock
`

Key Path Patterns:

| Type | Pattern | Example |
|------|---------|---------|
| Metadata | s3db.json | Database schema, resources, versions |
| Document | resource={name}/data/id={id} | resource=users/data/id=user-123 |
| Partition | resource={name}/partition={partition}/{field}={value}/id={id} | resource=users/partition=byRegion/region=US/id=user-123 |
| Plugin (global) | plugin={slug}/{path} | plugin=cache/config |
| Plugin (resource) | resource={name}/plugin={slug}/{path} | resource=wallets/plugin=eventual-consistency/balance/transactions/id=txn-123 |
| Lock (global) | plugin={slug}/locks/{lockName} | plugin=ttl/locks/cleanup |
| Lock (resource) | resource={name}/plugin={slug}/locks/{lockName} | resource=wallets/plugin=eventual-consistency/locks/balance-sync |

Storage Layers:

1. Documents - User data stored in resources
- Metadata: Stored in S3 object metadata (up to 2KB)
- Body: Large content stored in S3 object body (unlimited)

2. Partitions - Organized references for O(1) queries
- Hierarchical paths with field values
- References point to main document

3. Plugin Storage - Plugin-specific data
- Global: plugin={slug}/... - Shared config, caches, locks
- Resource-scoped: resource={name}/plugin={slug}/... - Per-resource data
- Supports same behaviors as resources (body-overflow, body-only, etc.)
- 3-5x faster than creating full resources
- Examples: EventualConsistency transactions, TTL expiration queues, Cache entries, Audit logs

Why This Structure?

- ✅ Flat hierarchy - No deep nesting, better S3 performance
- ✅ Self-documenting - Path tells you what data it contains
- ✅ Partition-friendly - O(1) lookups via S3 prefix queries
- ✅ Plugin isolation - Each plugin has its own namespace
- ✅ Consistent naming - resource=, partition=, plugin=, id= prefixes

$3

`javascript
import { S3db } from 's3db.js';

// Simple connection
const db = new S3db({
connectionString: 's3://ACCESS_KEY:SECRET@bucket/databases/myapp'
});

await db.connect();

// With plugins and options
const db = new S3db({
connectionString: 's3://bucket/databases/myapp',
logLevel: 'debug',
versioningEnabled: true,
executorPool: {
concurrency: 100, // Default concurrency (can increase for high-throughput)
retries: 3,
retryDelay: 1000
},
taskExecutorMonitoring: {
enabled: true,
collectMetrics: true,
sampleRate: 0.2
},
plugins: [
new CachePlugin({ ttl: 300000 }),
new MetricsPlugin()
],
httpClientOptions: {
keepAlive: true,
maxSockets: 100,
timeout: 60000
}
});

await db.connect();
`

$3

| Method | Description |
|--------|-------------|
| connect() | Initialize database connection and load metadata |
| createResource(config) | Create or update a resource |
| getResource(name, options?) | Get existing resource instance |
| resourceExists(name) | Check if resource exists |
| resources.{name} | Access resource by property |
| uploadMetadataFile() | Save metadata changes to S3 |

$3

Customize HTTP performance for your workload:

`javascript
const db = new S3db({
connectionString: '...',
httpClientOptions: {
keepAlive: true, // Enable connection reuse
keepAliveMsecs: 1000, // Keep connections alive for 1s
maxSockets: 50, // Max 50 concurrent connections
maxFreeSockets: 10, // Keep 10 free connections in pool
timeout: 60000 // 60 second timeout
}
});
`

Presets:

High Concurrency (APIs)

`javascript
httpClientOptions: {
keepAlive: true,
keepAliveMsecs: 1000,
maxSockets: 100, // Higher concurrency
maxFreeSockets: 20, // More free connections
timeout: 60000
}
`


Aggressive Performance (High-throughput)

`javascript
httpClientOptions: {
keepAlive: true,
keepAliveMsecs: 5000, // Longer keep-alive
maxSockets: 200, // High concurrency
maxFreeSockets: 50, // Large connection pool
timeout: 120000 // 2 minute timeout
}
`


Complete documentation: See above for all Database configuration options

---

🪵 Logging

s3db.js uses Pino - a blazing-fast, low-overhead JSON logger (5-10x faster than console.*). The logging system is hierarchical: Database → Plugins → Resources automatically inherit log levels, with per-component override capabilities.

$3

All components (Database, Plugins, Resources) automatically inherit the global log level:

`javascript
const db = new Database({
connectionString: 's3://bucket/db',
loggerOptions: {
level: 'warn' // ← Database, Resources, and Plugins all inherit 'warn'
}
});

await db.usePlugin(new CachePlugin(), 'cache'); // Inherits 'warn'
await db.usePlugin(new TTLPlugin(), 'ttl'); // Inherits 'warn'
`

$3

s3db.js provides two built-in format presets for different environments:

JSON Format (Production - Structured Logs):
`javascript
const db = new Database({
connectionString: 's3://bucket/db',
loggerOptions: {
level: 'info',
format: 'json' // ← Compact JSON for log aggregation
}
});

// Output: {"level":30,"time":1234567890,"msg":"User created","userId":"123"}
`

Pretty Format (Development - Human Readable):
`javascript
const db = new Database({
connectionString: 's3://bucket/db',
loggerOptions: {
level: 'debug',
format: 'pretty' // ← Colorized, readable output
}
});

// Output: [14:23:45.123] INFO: User created
// userId: "123"
`

Auto-Detection (Default):
`javascript
// Automatically chooses format based on:
// - TTY detection (terminal vs piped)
// - NODE_ENV (development vs production)
const db = new Database({
connectionString: 's3://bucket/db',
loggerOptions: {
level: 'info'
// format is auto-detected
}
});
`

$3

s3db.js errors automatically use toJSON() for structured logging:

`javascript
import { ValidationError } from 's3db.js';

const error = new ValidationError('Invalid email', {
field: 'email',
value: 'invalid@',
statusCode: 422
});

// Logs include full error context automatically
logger.error({ err: error }, 'Validation failed');

// Output includes: name, message, code, statusCode, suggestion, stack, etc.
`

$3

Fine-tune log levels for specific plugins or resources using childLevels:

`javascript
const db = new Database({
connectionString: 's3://bucket/db',
loggerOptions: {
level: 'warn', // ← Global default

childLevels: {
// Override specific plugins
'Plugin:cache': 'debug', // Cache plugin in debug mode
'Plugin:ttl': 'trace', // TTL plugin in trace mode
'Plugin:metrics': 'error', // Metrics plugin only shows errors
'Plugin:s3-queue': 'info', // S3Queue plugin in info mode

// Override specific resources
'Resource:users': 'debug', // Users resource in debug
'Resource:logs': 'silent' // Logs resource silenced
}
}
});
`

Result:
- Database → warn
- CachePlugin → debug (override)
- TTLPlugin → trace (override)
- MetricsPlugin → error (override)
- All other plugins → warn (inherited)

$3

Plugins can use completely custom loggers that don't inherit from Database:

`javascript
import { createLogger } from 's3db.js/logger';

// Create custom logger
const customLogger = createLogger({
name: 'MyApp',
level: 'trace',
// Pino options
transport: {
target: 'pino-pretty',
options: { colorize: true }
}
});

// Plugin uses custom logger instead of inheriting
const plugin = new CachePlugin({
logger: customLogger // ← Ignores inheritance
});

await db.usePlugin(plugin, 'cache');
`

$3

Change log levels on the fly for specific components:

`javascript
// Increase verbosity for debugging
db.setChildLevel('Plugin:cache', 'debug');

// Silence a noisy plugin
db.setChildLevel('Plugin:ttl', 'silent');

// Debug specific resource
db.setChildLevel('Resource:clicks', 'trace');
`

⚠️ Limitation: setChildLevel() only affects new child loggers. Loggers already created maintain their previous level.

$3

Override logging globally using environment variables:

`bash


Set log level

S3DB_LOG_LEVEL=debug node app.js

Set output format (using presets)

S3DB_LOG_FORMAT=pretty node app.js # Pretty format (colorized, human-readable)
S3DB_LOG_FORMAT=json node app.js # JSON format (structured logs for production)

Combined example

S3DB_LOG_LEVEL=debug S3DB_LOG_FORMAT=pretty node app.js
`

Legacy Support: The old S3DB_LOG_PRETTY environment variable is still supported for backward compatibility:
`bash
S3DB_LOG_PRETTY=true node app.js # Same as S3DB_LOG_FORMAT=pretty
S3DB_LOG_PRETTY=false node app.js # Same as S3DB_LOG_FORMAT=json
`

$3

| Level | Use Case | When to Use |
|-------|----------|-------------|
| silent | No logs | Tests, silent components |
| fatal | Critical errors | System unusable |
| error | Errors | Failed operations |
| warn | Warnings | Deprecations, fallbacks |
| info | Information | Default for production |
| debug | Debug | Development |
| trace | Full trace | Deep debugging |

$3

#### Production (Minimal Logs)

`javascript
const db = new Database({
connectionString: process.env.S3DB_CONNECTION,
loggerOptions: {
level: 'warn',
format: 'json', // ← Structured logs for aggregation

childLevels: {
// Info-level logging only for critical plugins
'Plugin:metrics': 'info',
'Plugin:audit': 'info'
}
}
});
`

#### Development (Verbose)

`javascript
const db = new Database({
connectionString: 'http://localhost:9000/bucket',
loggerOptions: {
level: 'debug',
format: 'pretty', // ← Human-readable, colorized

childLevels: {
// Trace the specific plugin you're debugging
'Plugin:cache': 'trace',

// Silence noisy plugins
'Plugin:metrics': 'silent'
}
}
});
`

#### Debug Specific Plugin

`javascript
const db = new Database({
connectionString: 's3://bucket/db',
loggerOptions: {
level: 'warn',
format: 'json', // ← Production format

childLevels: {
// Debug ONLY the TTL plugin
'Plugin:ttl': 'trace'
}
}
});
`

$3

Plugins: Format is Plugin:{name}

`javascript
await db.usePlugin(new CachePlugin(), 'cache');
// Child logger: 'Plugin:cache'

await db.usePlugin(new TTLPlugin(), 'my-ttl');
// Child logger: 'Plugin:my-ttl'
`

Resources: Format is Resource:{name}

`javascript
await db.createResource({ name: 'users', ... });
// Child logger: 'Resource:users'
`

$3

The API Plugin includes automatic HTTP request/response logging with smart detection:

Smart Detection:
- If pino-http is installed: Uses full-featured pino-http with all bells and whistles
- If pino-http is NOT installed: Falls back to simple built-in HTTP logging

Installation (optional, recommended):
`bash
npm install pino-http
`

Usage:
`javascript
import { APIPlugin } from 's3db.js';

const api = new APIPlugin({
port: 3000,

// Enable HTTP logging (works with or without pino-http!)
httpLogger: {
enabled: true,
autoLogging: true, // Log all requests/responses
ignorePaths: ['/health'], // Skip logging for these paths

// Custom log level based on status code
customLogLevel: (req, res, err) => {
if (err || res.statusCode >= 500) return 'error';
if (res.statusCode >= 400) return 'warn';
return 'info';
}
},

// Enable request ID tracking (recommended)
requestId: {
enabled: true,
headerName: 'X-Request-ID'
}
});
`

What you get:

| Feature | With pino-http | Without pino-http |
|---------|---------------|-------------------|
| Request logging | ✅ Full | ✅ Basic |
| Response logging | ✅ Full | ✅ Basic |
| Error logging | ✅ Full | ✅ Basic |
| Request ID | ✅ Auto | ✅ Manual |
| Custom serializers | ✅ Yes | ✅ Basic |
| Performance overhead | ⚡ Minimal | ⚡ Minimal |

No installation required! HTTP logging works out-of-the-box with basic features. Install pino-http for enhanced capabilities.

Automatic Logging Output:
`json
{
"level": 30,
"time": 1234567890,
"req": {
"id": "abc123",
"method": "POST",
"url": "/users",
"headers": { "user-agent": "...", "content-type": "application/json" }
},
"res": {
"statusCode": 201,
"headers": { "content-type": "application/json" }
},
"responseTime": 45,
"msg": "request completed"
}
`

Features:
- Request/response correlation with request IDs
- Automatic status code-based log levels
- Error serialization with toJSON()
- Path filtering (e.g., skip /health, /metrics)
- Zero configuration required

$3

1. Production: Use format: 'json' with level: 'warn' for structured logging
2. Development: Use format: 'pretty' with level: 'debug' for readability
3. Debugging: Use childLevels to isolate specific components
4. Performance: Lower levels (trace, debug) have performance impact
5. Inheritance: Components automatically inherit global level if not overridden
6. Error Logging: Custom errors automatically use toJSON() for rich context
7. CI/CD: Use format: 'json' in automated environments for parsing
8. HTTP Logging: Enable httpLogger in API Plugin for automatic request tracking

---

📋 Resources

Resources are the core abstraction in s3db.js - they define your data structure, validation rules, and behavior. Think of them as tables in traditional databases, but with much more flexibility and features.

$3

Resources provide:
- ✅ Schema validation with 30+ field types
- ✅ 5 behavior strategies for handling 2KB S3 metadata limit
- ✅ Partitioning for O(1) queries vs O(n) scans
- ✅ Hooks & middlewares for custom logic
- ✅ Events for real-time notifications
- ✅ Versioning for schema evolution
- ✅ Encryption for sensitive fields
- ✅ Streaming for large datasets

Quick example:
`javascript
const users = await db.createResource({
name: 'users',
attributes: {
email: 'email|required|unique',
password: 'secret|required',
age: 'number|min:18|max:120'
},
behavior: 'enforce-limits',
timestamps: true,
partitions: {
byAge: { fields: { age: 'number' } }
}
});

await users.insert({ email: 'john@example.com', password: 'secret123', age: 25 });
`

$3

Define your data structure with powerful validation using fastest-validator - a blazing-fast validation library with comprehensive type support:

#### Basic Types

| Type | Example | Validation Rules |
|------|---------|------------------|
| string | "name: 'string\|required'" | min, max, length, pattern, enum |
| number | "age: 'number\|min:0'" | min, max, integer, positive, negative |
| boolean | "isActive: 'boolean'" | true, false |
| email | "email: 'email\|required'" | RFC 5322 validation |
| url | "website: 'url'" | Valid URL format |
| date | "createdAt: 'date'" | ISO 8601 dates |
| array | "tags: 'array\|items:string'" | items, min, max, unique |
| object | "profile: { type: 'object', props: {...} }" | Nested validation |

#### Advanced Types (with encoding)

| Type | Savings | Example |
|------|---------|---------|
| secret | Encrypted | "password: 'secret\|required'" - AES-256-GCM |
| embedding:N | 77% | "vector: 'embedding:1536'" - Fixed-point Base62 |
| ip4 | 47% | "ipAddress: 'ip4'" - Binary Base64 |
| ip6 | 44% | "ipv6: 'ip6'" - Binary Base64 |

Encoding optimizations:
- ISO timestamps → Unix Base62 (67% savings)
- UUIDs → Binary Base64 (33% savings)
- Dictionary values → Single bytes (95% savings)

#### Schema Examples

> 📖 Validation powered by fastest-validator
> All schemas use fastest-validator's syntax with full support for shorthand notation.

`javascript
// Simple schema
{
name: 'string|required|min:2|max:100',
email: 'email|required|unique',
age: 'number|integer|min:0|max:150'
}

// Nested objects - MAGIC AUTO-DETECT! ✨ (recommended)
// Just write your object structure - s3db detects it automatically!
{
name: 'string|required',
profile: { // ← No $$type needed! Auto-detected as optional object
bio: 'string|max:500',
avatar: 'url|optional',
social: { // ← Deeply nested also works!
twitter: 'string|optional',
github: 'string|optional'
}
}
}

// Need validation control? Use $$type (when you need required/optional)
{
name: 'string|required',
profile: {
$$type: 'object|required', // ← Add required validation
bio: 'string|max:500',
avatar: 'url|optional'
}
}

// Advanced: Full control (rare cases - strict mode, etc)
{
name: 'string|required',
profile: {
type: 'object',
optional: false,
strict: true, // ← Enable strict validation
props: {
bio: 'string|max:500',
avatar: 'url|optional'
}
}
}

// Arrays with validation
{
name: 'string|required',
tags: 'array|items:string|min:1|max:10|unique',
scores: 'array|items:number|min:0|max:100'
}

// Encrypted fields
{
email: 'email|required',
password: 'secret|required',
apiKey: 'secret|required'
}
`

$3

S3 metadata has a 2KB limit. Behaviors define how to handle data that exceeds this:

| Behavior | Enforcement | Data Loss | Use Case |
|----------|-------------|-----------|----------|
| user-managed | None | Possible | Dev/Test - warnings only |
| enforce-limits | Strict | No | Production - throws errors |
| truncate-data | Truncates | Yes | Content management - smart truncation |
| body-overflow | Splits | No | Mixed data - metadata + body |
| body-only | Unlimited | No | Large docs - everything in body |

`javascript
// Enforce limits (recommended for production)
const users = await db.createResource({
name: 'users',
behavior: 'enforce-limits',
attributes: { name: 'string', bio: 'string' }
});

// Body overflow for large content
const blogs = await db.createResource({
name: 'blogs',
behavior: 'body-overflow',
attributes: { title: 'string', content: 'string' }
});

// Body-only for documents
const documents = await db.createResource({
name: 'documents',
behavior: 'body-only',
attributes: { title: 'string', content: 'string', metadata: 'object' }
});
`

$3

#### CRUD Operations

`javascript
// Create
const user = await users.insert({ name: 'John', email: 'john@example.com' });

// Read
const user = await users.get('user-123');
const all = await users.list({ limit: 10, offset: 0 });
const filtered = await users.query({ isActive: true });

// Update (3 methods with different performance)
await users.update(id, { name: 'Jane' }); // GET+PUT merge (baseline)
await users.patch(id, { name: 'Jane' }); // HEAD+COPY (40-60% faster*)
await users.replace(id, fullObject); // PUT only (30-40% faster)
// *patch() uses HEAD+COPY for metadata-only behaviors

// Delete
await users.delete('user-123');
`

#### Bulk Operations

`javascript
// Bulk insert
await users.insertMany([
{ name: 'User 1', email: 'user1@example.com' },
{ name: 'User 2', email: 'user2@example.com' }
]);

// Bulk get
const data = await users.getMany(['user-1', 'user-2', 'user-3']);

// Bulk delete
await users.deleteMany(['user-1', 'user-2']);
`

$3

Organize data for fast queries without scanning:

`javascript
const analytics = await db.createResource({
name: 'analytics',
attributes: {
userId: 'string',
event: 'string',
timestamp: 'date',
region: 'string'
},
partitions: {
// Single field
byEvent: { fields: { event: 'string' } },

// Multiple fields (composite)
byEventAndRegion: {
fields: {
event: 'string',
region: 'string'
}
},

// Nested field
byUserCountry: {
fields: {
'profile.country': 'string'
}
}
},

// Async partitions for 70-100% faster writes
asyncPartitions: true
});

// Query by partition (O(1))
const usEvents = await analytics.list({
partition: 'byEventAndRegion',
partitionValues: { event: 'click', region: 'US' }
});
`

Automatic timestamp partitions:

`javascript
const events = await db.createResource({
name: 'events',
attributes: { name: 'string', data: 'object' },
timestamps: true // Auto-creates byCreatedDate and byUpdatedDate partitions
});

const todayEvents = await events.list({
partition: 'byCreatedDate',
partitionValues: { createdAt: '2024-01-15' }
});
`

$3

Add custom logic before/after operations:

`javascript
const products = await db.createResource({
name: 'products',
attributes: { name: 'string', price: 'number', sku: 'string' },
hooks: {
// Before operations
beforeInsert: [
async (data) => {
data.sku = PROD-${Date.now()};
return data;
}
],
beforeUpdate: [
async (data) => {
data.updatedAt = new Date().toISOString();
return data;
}
],

// After operations
afterInsert: [
async (data) => {
console.log(Product ${data.name} created with SKU ${data.sku});
}
],
afterDelete: [
async (data) => {
await notifyWarehouse(data.sku);
}
]
}
});
`

Available hooks:
- beforeInsert, afterInsert
- beforeUpdate, afterUpdate
- beforeDelete, afterDelete
- beforeGet, afterGet
- beforeList, afterList

$3

Intercept and transform method calls:

`javascript
// Authentication middleware
users.useMiddleware('inserted', async (ctx, next) => {
if (!ctx.args[0].userId) {
throw new Error('Authentication required');
}
return await next();
});

// Logging middleware
users.useMiddleware('updated', async (ctx, next) => {
const start = Date.now();
const result = await next();
console.log(Update took ${Date.now() - start}ms);
return result;
});

// Validation middleware
users.useMiddleware('inserted', async (ctx, next) => {
ctx.args[0].name = ctx.args[0].name.toUpperCase();
return await next();
});
`

Supported methods:
fetched, list, inserted, updated, deleted, deleteMany, exists, getMany, count, page, listIds, getAll

$3

Listen to resource operations:

`javascript
const users = await db.createResource({
name: 'users',
attributes: { name: 'string', email: 'string' },

// Declarative event listeners
events: {
insert: (event) => {
console.log('User created:', event.id, event.name);
},

update: [
(event) => console.log('Update detected:', event.id),
(event) => {
if (event.$before.email !== event.$after.email) {
console.log('Email changed!');
}
}
],

delete: (event) => {
console.log('User deleted:', event.id);
}
}
});

// Programmatic listeners
users.on('inserted', (event) => {
sendWelcomeEmail(event.email);
});
`

Available events:
inserted, updated, deleted, insertMany, deleteMany, list, count, fetched, getMany

$3

Process large datasets efficiently:

`javascript
// Readable stream
const readableStream = await users.readable({
batchSize: 50,
concurrency: 10
});

readableStream.on('data', (user) => {
console.log('Processing:', user.name);
});

readableStream.on('end', () => {
console.log('Stream completed');
});

// Writable stream
const writableStream = await users.writable({
batchSize: 25,
concurrency: 5
});

userData.forEach(user => writableStream.write(user));
writableStream.end();
`

$3

A complex, production-ready resource showing all capabilities:

`javascript
const orders = await db.createResource({
name: 'orders',

// Schema with all features
attributes: {
// Basic fields
orderId: 'string|required|unique',
userId: 'string|required',
status: 'string|required|enum:pending,processing,completed,cancelled',
total: 'number|required|min:0',

// Encrypted sensitive data
paymentToken: 'secret|required',

// Nested objects
customer: {
type: 'object',
props: {
name: 'string|required',
email: 'email|required',
phone: 'string|optional',
address: {
type: 'object',
props: {
street: 'string|required',
city: 'string|required',
country: 'string|required|length:2',
zipCode: 'string|required'
}
}
}
},

// Arrays
items: 'array|items:object|min:1',
tags: 'array|items:string|unique|optional',

// Special types
ipAddress: 'ip4',
userAgent: 'string|max:500',

// Embeddings for AI/ML
orderEmbedding: 'embedding:384'
},

// Behavior for large orders
behavior: 'body-overflow',

// Automatic timestamps
timestamps: true,

// Versioning for schema evolution
versioningEnabled: true,

// Custom ID generation
idGenerator: () => ORD-${Date.now()}-${Math.random().toString(36).substr(2, 5)},

// Partitions for efficient queries
partitions: {
byStatus: { fields: { status: 'string' } },
byUser: { fields: { userId: 'string' } },
byCountry: { fields: { 'customer.address.country': 'string' } },
byUserAndStatus: {
fields: {
userId: 'string',
status: 'string'
}
}
},

// Async partitions for faster writes
asyncPartitions: true,

// Hooks for business logic
hooks: {
beforeInsert: [
async function(data) {
// Validate stock availability
const available = await this.validateStock(data.items);
if (!available) throw new Error('Insufficient stock');

// Calculate total
data.total = data.items.reduce((sum, item) => sum + item.price * item.quantity, 0);

return data;
},
async (data) => {
// Add metadata
data.processedAt = new Date().toISOString();
return data;
}
],

afterInsert: [
async (data) => {
// Send confirmation email
await sendOrderConfirmation(data.customer.email, data.orderId);
},
async (data) => {
// Update inventory
await updateInventory(data.items);
}
],

beforeUpdate: [
async function(data) {
// Prevent status rollback
if (data.status === 'cancelled' && this.previousStatus === 'completed') {
throw new Error('Cannot cancel completed order');
}
return data;
}
],

afterUpdate: [
async (data) => {
// Notify customer of status change
if (data.$before.status !== data.$after.status) {
await notifyStatusChange(data.customer.email, data.status);
}
}
]
},

// Events for monitoring
events: {
insert: (event) => {
console.log(Order ${event.orderId} created - Total: $${event.total});
metrics.increment('orders.created');
},

update: [
(event) => {
if (event.$before.status !== event.$after.status) {
console.log(Order ${event.orderId}: ${event.$before.status} → ${event.$after.status});
metrics.increment(orders.status.${event.$after.status});
}
}
],

delete: (event) => {
console.warn(Order ${event.orderId} deleted);
metrics.increment('orders.deleted');
}
}
});

// Add middlewares for cross-cutting concerns
orders.useMiddleware('inserted', async (ctx, next) => {
// Rate limiting
await checkRateLimit(ctx.args[0].userId);
return await next();
});

orders.useMiddleware('updated', async (ctx, next) => {
// Audit logging
const start = Date.now();
const result = await next();
await auditLog.write({
action: 'order.update',
orderId: ctx.args[0],
duration: Date.now() - start,
timestamp: new Date()
});
return result;
});
`

Complete documentation: docs/resources.md

---

⚡ Performance & Concurrency

s3db.js features Separate Executor Pools - a revolutionary architecture where each Database instance gets its own independent executor pool for maximum efficiency and zero contention.

$3

Each database instance gets its own executor pool, enabling:

- 🚀 40-50% faster at medium scale (5,000+ operations)
- 📈 13x less memory at large scale (10,000+ operations)
- ⏱️ Zero contention between concurrent databases
- 🛡️ Auto-retry with exponential backoff
- 🧠 Adaptive tuning - automatically adjusts concurrency based on performance
- Default parallelism: 100 (up from 10, optimized for S3 throughput)

$3

Executor pool is enabled by default with optimized settings:

`javascript
import { Database } from 's3db.js';const db = new Database({
connectionString: 's3://bucket/database'
// That's it! Executor pool is automatically configured with:
// - Separate pool per database (zero contention)
// - Concurrency: 100 (default)
// - Auto-retry with exponential backoff
// - Priority queue for important operations
// - Real-time metrics
})

await db.connect()
`

$3

Executor pools (and the standalone TasksRunner/TasksPool) support lightweight vs full-featured schedulers, observability exports, and adaptive concurrency:

`javascript
const db = new Database({
connectionString: 's3://bucket/database',
executorPool: {
features: { profile: 'light', emitEvents: false }, // or 'balanced'
monitoring: {
enabled: true,
reportInterval: 1000,
exporter: (snapshot) => console.log('[executor]', snapshot)
},
autoTuning: {
enabled: true,
minConcurrency: 10,
maxConcurrency: 200,
targetLatency: 250,
adjustmentInterval: 5000
}
}
})
`

Use the light profile for PromisePool-style throughput when you just need FIFO fan-out. Switch to balanced when you need retries, priority aging, rich metrics, or adaptive scaling. The same options apply to filesystem/memory clients via taskExecutorMonitoring, autoTuning, and features.profile.

$3

Customize concurrency for your specific workload:

`javascript
import { Database } from 's3db.js';const db = new Database({
connectionString: 's3://bucket/database',
executorPool: {
concurrency: 200, // Increase for high-throughput scenarios
// Or use auto-tuning:
// concurrency: 'auto', // Auto-tune based on system load
autotune: {
targetLatency: 100, // Target 100ms per operation
minConcurrency: 50, // Never go below 50
maxConcurrency: 500 // Never exceed 500
}
}
})
`

$3

`javascript
// Get queue statistics
const stats = db.client.getQueueStats()
console.log(stats)
// {
// queueSize: 0,
// activeCount: 50,
// processedCount: 15420,
// errorCount: 3,
// retryCount: 8
// }

// Get performance metrics
const metrics = db.client.getAggregateMetrics()
console.log(metrics)
// {
// count: 15420,
// avgExecution: 45,
// p50: 42,
// p95: 78,
// p99: 125
// }

// Lifecycle control
await db.client.pausePool() // Pause processing
db.client.resumePool() // Resume processing
await db.client.drainPool() // Wait for queue to empty
db.client.stopPool() // Stop and cleanup
`

$3

OperationPool emits events for monitoring and observability:

| Event | Parameters | Description |
|-------|------------|-------------|
| pool:taskStarted | (task) | Task execution started |
| pool:taskCompleted | (task, result) | Task completed successfully |
| pool:taskError | (task, error) | Task failed with error |
| pool:taskRetry | (task, attempt) | Task retry attempt (1-based) |
| pool:taskMetrics | (metrics) | Task performance metrics |
| pool:paused | () | Pool paused (waiting for active tasks) |
| pool:resumed | () | Pool resumed processing |
| pool:drained | () | All tasks completed (queue empty) |
| pool:stopped | () | Pool stopped (pending tasks cancelled) |

Example:
`javascript
db.client.on('pool:taskCompleted', (task, result) => {
console.log(✓ ${task.id}: ${task.timings.execution}ms)
})

db.client.on('pool:taskError', (task, error) => {
console.error(✗ ${task.id}:, error.message)
})
`

See src/concerns/operation-pool.js for event implementation details.

$3

Benchmark results from comprehensive testing of 108 scenarios (see docs/benchmarks/operation-pool.md and BENCHMARK-RESULTS-TABLE.md):

| Scale | Separate Pools | Promise.all | Shared Pool | Winner |
|-------|----------------|------------|------------|--------|
| 1,000 ops | 2.1ms | 1.8ms | 2.5ms | Promise.all (marginal) |
| 5,000 ops | 18ms | 28ms | 32ms | Separate Pools (+40%) |
| 10,000 ops | 35ms | 45ms | 52ms | Separate Pools (+37%) |
| Memory (10K) | 88 MB | 1,142 MB | 278 MB | Separate Pools (13x better) |

$3

✅ Automatic (no configuration needed):
- All operations benefit from Separate Pools
- Default concurrency: 100 (optimized for S3)
- Zero contention between databases
- Auto-retry with exponential backoff
- Adaptive tuning available for custom scenarios

Customize concurrency for:
- High-throughput APIs: executorPool: { concurrency: 200 }
- Data pipelines: executorPool: { concurrency: 300-500 }
- Single/low-frequency ops: executorPool: { concurrency: 10 }
- Memory-constrained: executorPool: { concurrency: 25-50 }

$3

Separate Pools comes pre-configured with production-ready defaults. Override only what you need:

`javascript
// Minimal - uses all defaults (recommended)
const db = new Database({
connectionString: 's3://bucket/database'
// executorPool uses defaults: { concurrency: 100 }
})

// Custom - override specific settings
const db = new Database({
connectionString: 's3://bucket/database',
executorPool: {
concurrency: 200, // Concurrency per database pool (default: 100)
retries: 3, // Max retry attempts
retryDelay: 1000, // Initial retry delay (ms)
timeout: 30000, // Operation timeout (ms)
retryableErrors: [ // Errors to retry (empty = all)
'NetworkingError',
'TimeoutError',
'RequestTimeout',
'ServiceUnavailable',
'SlowDown',
'RequestLimitExceeded'
],
autotune: { // Auto-tuning (optional)
enabled: true,
targetLatency: 100, // Target latency (ms)
minConcurrency: 50, // Min per database
maxConcurrency: 500, // Max per database
targetMemoryPercent: 0.7, // Target memory usage (70%)
adjustmentInterval: 5000 // Check interval (ms)
}
},
taskExecutorMonitoring: {
enabled: true,
collectMetrics: true,
sampleRate: 1,
mode: 'balanced'
}
})
`

Complete documentation: docs/benchmarks/executor-pool.md

---

🔌 Plugins

> Quick Jump: 🌐 API | 🔐 Identity | ⚡ Performance | 📊 Data | 🎮 Gaming | 🔧 DevOps | 🤖 ML/AI | 🕷️ Web Scraping

Extend s3db.js with powerful plugins. All plugins are optional and can be installed independently.

$3

APIPlugin • IdentityPlugin

APIPlugin - Transform s3db.js into production-ready REST API with OpenAPI, multi-auth (JWT/OIDC/Basic/API Key), rate limiting, and template engines.

IdentityPlugin - Complete OAuth2/OIDC server with MFA, whitelabel UI, and enterprise SSO.

$3

CachePlugin • TTLPlugin • EventualConsistencyPlugin • MetricsPlugin

CachePlugin - Memory/S3/filesystem caching with compression and automatic invalidation.

TTLPlugin - Auto-cleanup expired records with O(1) partition-based deletion.

EventualConsistencyPlugin - Eventually consistent counters and high-performance analytics.

MetricsPlugin - Performance monitoring with Prometheus export.

$3

ReplicatorPlugin • ImporterPlugin • BackupPlugin • AuditPlugin

ReplicatorPlugin - Real-time replication to BigQuery, PostgreSQL, MySQL, Turso, PlanetScale, and SQS.

ImporterPlugin - Stream processing for large JSON/CSV imports.

BackupPlugin - Automated backups to S3, filesystem, or cross-cloud.

AuditPlugin - Compliance logging for all database operations.

$3

TournamentPlugin

TournamentPlugin - Complete tournament engine supporting Single/Double Elimination, Round Robin, Swiss, and League formats with automated bracket generation.

$3

QueueConsumerPlugin • SchedulerPlugin • TfstatePlugin • CloudInventoryPlugin • CostsPlugin

QueueConsumerPlugin - Process RabbitMQ/SQS messages for event-driven architectures.

SchedulerPlugin - Cron-based job scheduling for maintenance tasks.

TfstatePlugin - Track Terraform infrastructure changes and drift detection.

CloudInventoryPlugin - Multi-cloud inventory with versioning and diff tracking.

CostsPlugin - AWS cost tracking and optimization insights.

$3

MLPlugin • VectorPlugin • FullTextPlugin • GeoPlugin

MLPlugin - Machine learning model management and inference pipelines.

VectorPlugin - Vector similarity search (cosine, euclidean) for RAG and ML applications.

FullTextPlugin - Full-text search with tokenization and indexing.

GeoPlugin - Geospatial queries and distance calculations.

$3

PuppeteerPlugin

PuppeteerPlugin - Enterprise-grade browser automation with anti-bot detection, cookie farming, proxy rotation, and intelligent pooling for web scraping at scale.

$3

RelationPlugin • StateMachinePlugin • S3QueuePlugin

RelationPlugin - ORM-like relationships with join optimization (10-100x faster queries).

StateMachinePlugin - Finite state machine workflows for business processes.

S3QueuePlugin - Distributed queue with zero race conditions using S3.

$3

`bash


Core plugins (no dependencies)

Included in s3db.js package

External dependencies (install only what you need)

pnpm add pg # PostgreSQL replication (ReplicatorPlugin)
pnpm add @google-cloud/bigquery # BigQuery replication (ReplicatorPlugin)
pnpm add @aws-sdk/client-sqs # SQS replication/consumption (ReplicatorPlugin, QueueConsumerPlugin)
pnpm add amqplib # RabbitMQ consumption (QueueConsumerPlugin)
pnpm add ejs # Template engine (APIPlugin - optional)
`

$3

`javascript
import { S3db } from 's3db.js';
import { CachePlugin, MetricsPlugin, TTLPlugin } from 's3db.js';

const db = new S3db({
connectionString: 's3://bucket/databases/myapp',
plugins: [
// Cache frequently accessed data
new CachePlugin({
driver: 'memory',
ttl: 300000, // 5 minutes
config: {
maxMemoryPercent: 0.1, // 10% of system memory
enableCompression: true
}
}),

// Track performance metrics
new MetricsPlugin({
enablePrometheus: true,
port: 9090
}),

// Auto-cleanup expired sessions
new TTLPlugin({
resources: {
sessions: { ttl: 86400, onExpire: 'soft-delete' } // 24h
}
})
]
});
`

$3

Simple plugin example:

`javascript
import { Plugin } from 's3db.js';

export class MyPlugin extends Plugin {
constructor(options = {}) {
super(options);
this.name = 'MyPlugin';
}

async initialize(database) {
console.log('Plugin initialized!');

// Wrap methods
this.wrapMethod('Resource', 'inserted', async (original, resource, args) => {
console.log(Inserting into ${resource.name});
const result = await original(...args);
console.log(Inserted: ${result.id});
return result;
});
}
}
`

Complete documentation: docs/plugins/README.md

---

🤖 MCP & Integrations

$3

S3DB includes a powerful MCP server with 28 specialized tools for database operations, debugging, and monitoring.

#### Quick Start

`bash


Claude CLI (one command)

claude mcp add s3db \
--transport stdio \
-- npx -y s3db.js s3db-mcp --transport=stdio

Standalone HTTP server

npx s3db.js s3db-mcp --transport=sse
`

#### Features

- ✅ 28 tools - CRUD, debugging, partitions, bulk ops, export/import, monitoring
- ✅ Multiple transports - SSE for web, stdio for CLI
- ✅ Auto-optimization - Cache and cost tracking enabled by default
- ✅ Partition-aware - Intelligent caching with partition support

#### Tool Categories

1. Connection (3) - dbConnect, dbDisconnect, dbStatus
2. Debugging (5) - dbInspectResource, dbGetMetadata, resourceValidate, dbHealthCheck, resourceGetRaw
3. Query (2) - resourceQuery, resourceSearch
4. Partitions (4) - resourceListPartitions, dbFindOrphanedPartitions, etc.
5. Bulk Ops (3) - resourceUpdateMany, resourceBulkUpsert, resourceDeleteAll
6. Export/Import (3) - resourceExport, resourceImport, dbBackupMetadata
7. Monitoring (4) - dbGetStats, resourceGetStats, cacheGetStats, dbClearCache

Complete documentation: docs/mcp.md

$3

s3db.js integrates seamlessly with:

- BigQuery - Real-time data replication via ReplicatorPlugin
- PostgreSQL - Sync to traditional databases via ReplicatorPlugin
- AWS SQS - Event streaming and message queues
- RabbitMQ - Message queue integration
- Prometheus - Metrics export via MetricsPlugin
- Vector Databases - Embedding field type with 77% compression

---

🔧 CLI

s3db.js includes a powerful CLI for database management and operations.

$3

`bash


Global

npm install -g s3db.js

Project

npm install s3db.js
npx s3db [command]
`

$3

`bash


List resources

s3db list

Query resources

s3db query users
s3db query users --filter '{"status":"active"}'

Insert records

s3db insert users --data '{"name":"John","email":"john@example.com"}'

Update records

s3db update users user-123 --data '{"age":31}'

Delete records

s3db delete users user-123

Export data

s3db export users --format json > users.json
s3db export users --format csv > users.csv

Import data

s3db import users < users.json

Stats

s3db stats
s3db stats users

MCP Server

s3db s3db-mcp --transport=stdio
s3db s3db-mcp --transport=sse --port=17500
`

$3

`bash
S3DB_CONNECTION_STRING=s3://bucket/databases/myapp
S3DB_CACHE_ENABLED=true
S3DB_COSTS_ENABLED=true
S3DB_VERBOSE=false
``

---

📖 Documentation

$3

- [**Resources (Complete Guid