nosql-file


A flexible, file-based data storage library for Node.js with support for YAML and JSON formats, featuring concurrent access control, multiple write modes, and both array and key-value storage options.

Features

- Dual Storage Types: Collections (array-based) and Dictionaries (key-value)
- Multiple Formats: YAML and JSON support with automatic serialization
- Write Modes: Async, fast (background), and memory-only modes
- File Locking: Built-in concurrent access control with FileLockManager
- Splited Mode: Store dictionary values in separate files for better performance with large datasets
- Metadata Support: Store metadata (version, tags, timestamps, custom fields) in separate files without polluting data
- Event Emitters: Observable writes and errors for reactive applications
- Type-Safe: Full TypeScript support with generics
- No Dependencies: Core functionality with minimal external dependencies
- Synchronous Operations: Load configuration synchronously at application startup
- Comprehensive Testing: 267 tests with 96%+ code coverage

Installation

``bash
npm install nosql-file
`

Quick Start

$3

`typescript
import { NoSqlFile } from 'nosql-file';

// Create a database instance
const db = new NoSqlFile('./data');

// Get a collection (array-based storage)
const users = await db.collection('users');

// Insert documents
await users.insert({ id: 1, name: 'Alice', age: 30 });
await users.insert({ id: 2, name: 'Bob', age: 25 });

// Query documents
const adults = users.find({ age: 30 });
console.log(adults); // [{ id: 1, name: 'Alice', age: 30 }]

// Update documents
await users.update({ id: 1 }, { age: 31 });

// Delete documents
await users.delete({ id: 2 });

// Get all documents
const all = users.getAll();
console.log(Total users: ${users.count()});

// Clear all data
await users.clear();
`

$3

`typescript
import { NoSqlFile } from 'nosql-file';

const db = new NoSqlFile('./data');

// Get a dictionary (key-value storage)
const config = await db.dictionary('config');

// Set key-value pairs
await config.set('apiUrl', 'https://api.example.com');
await config.set('timeout', 5000);
await config.set('debug', true);

// Get values
const apiUrl = config.get('apiUrl');

// Check existence
if (config.has('timeout')) {
console.log(Timeout: ${config.get('timeout')}ms);
}

// List all keys/values
const keys = config.keys();
const allValues = config.values();

// Delete a key
await config.delete('debug');

// Get all data
const all = config.getAll();

// Clear all
await config.clear();
`

API Documentation

$3

Main class for managing collections and dictionaries.

#### Constructor

`typescript
const db = new NoSqlFile(dataPath, options);
`

- dataPath: Directory where data files will be stored
- options.format: 'yaml' or 'json' (default: 'yaml')
- options.disableMetadata: boolean - Disable metadata file creation (default: false)

#### Methods

- collection(name): Promise - Get or create a collection
- dictionary(name, options): Promise - Get or create a dictionary
- options.splited: boolean - Use splited mode (default: false)
- options.disableMetadata: boolean - Override global metadata setting
- syncAll(): Promise - Sync all collections and dictionaries to disk
- discardAll(): Promise - Reload all data from disk, discarding changes
- dropCollection(name): Promise - Delete a collection and its data
- dropDictionary(name, options): Promise - Delete a dictionary and its data
- close(): Promise - Sync all pending writes before closing

$3

Generic array-based document storage.

#### Constructor

`typescript
const collection = new Collection(name, dataPath, format, fileLockManager, disableMetadata);
`

- name: Collection name (used as filename)
- dataPath: Directory path
- format: 'yaml' or 'json' (default: 'yaml')
- fileLockManager: Optional custom lock manager
- disableMetadata: Disable metadata files (default: false)

#### Methods

- insert(document, options): Promise - Add a document
- find(query): T[] - Find documents matching query (empty query returns all)
- update(query, updates, options): Promise - Update matching documents
- delete(query, options): Promise - Delete matching documents
- getAll(): T[] - Get all documents
- count(): number - Get document count
- clear(options): Promise - Delete all documents
- sync(): Promise - Manually sync to disk
- discard(): Promise - Reload from disk, discarding changes
- drop(): Promise - Delete the collection file and clear memory

#### Events

- written - Emitted after successful sync
- error - Emitted when an error occurs

$3

Key-value storage with two modes.

#### Constructor

`typescript
const dict = new Dictionary(name, dataPath, format, splited, fileLockManager, disableMetadata);
`

- name: Dictionary name
- dataPath: Directory path
- format: 'yaml' or 'json' (default: 'yaml')
- splited: Use splited mode - one file per key (default: false)
- fileLockManager: Optional custom lock manager
- disableMetadata: Disable metadata files (default: false)

#### Methods

- set(key, value, options): Promise - Set a key-value pair
- get(key): unknown - Get value by key
- delete(key, options): Promise - Delete a key
- has(key): boolean - Check if key exists
- keys(): string[] - Get all keys
- values(): unknown[] - Get all values
- getAll(): Record - Get all data
- clear(options): Promise - Delete all keys
- sync(specificKey): Promise - Manually sync to disk
- discard(): Promise - Reload from disk, discarding changes
- drop(): Promise - Delete the dictionary file/directory and clear memory

#### Events

- written - Emitted after successful sync
- error - Emitted when an error occurs

$3

All write operations (insert, update, delete, set, etc.) support an options parameter with a mode property:

`typescript
// Async mode (default) - wait for write completion
await collection.insert(doc, { mode: 'async' });

// Fast mode - fire and forget, write in background
await collection.insert(doc, { mode: 'fast' });

// Memory mode - only update in-memory, no disk write
await collection.insert(doc, { mode: 'memory' });
`

Advanced Usage

$3

Store metadata separately from your data files for version tracking, timestamps, and custom information:

`typescript
const users = await db.collection('users');

// Set metadata
await users.setMeta({
version: 1,
tags: ['users', 'authentication'],
custom: {
author: 'John Doe',
description: 'User database'
}
});

// Get metadata (includes auto-updated timestamps)
const meta = await users.getMeta();
console.log(meta.createdAt); // ISO 8601 timestamp
console.log(meta.updatedAt); // Updated on each sync
console.log(meta.version); // 1
console.log(meta.tags); // ['users', 'authentication']

// Metadata is stored in users.meta.yaml (separate from users.yaml)
`

See METADATA.md for complete metadata documentation.

$3

If you don't need metadata files, you can disable them globally or per collection/dictionary:

`typescript
// Disable metadata globally for all collections and dictionaries
const db = new NoSqlFile('./data', { disableMetadata: true });

// Collections and dictionaries won't create .meta files
const users = await db.collection('users');
await users.insert({ name: 'Alice' });
// users.meta.yaml is NOT created

// Override for a specific dictionary
const config = await db.dictionary('config', { disableMetadata: false });
// config.meta.yaml IS created (metadata enabled)

// Direct instantiation with metadata disabled
import { Collection, Dictionary } from 'nosql-file';

const logs = new Collection('logs', './data', 'yaml', undefined, true);
const cache = new Dictionary('cache', './data', 'yaml', false, undefined, true);

// Accessing metadata methods when disabled will throw an error
try {
await logs.getMeta();
} catch (error) {
console.error(error.message); // "Metadata is disabled for this data store"
}
`

When to disable metadata:
- Temporary data (logs, caches) that doesn't need tracking
- Performance-critical applications with frequent writes
- Reduced file system clutter
- When metadata features are not needed

$3

`typescript
interface User {
id: number;
name: string;
email: string;
age: number;
}

const users = await db.collection('users');

// Type-safe operations
await users.insert({
id: 1,
name: 'Alice',
email: 'alice@example.com',
age: 30
});

const result = users.find({ age: 30 });
// result is User[]
`

$3

For dictionaries with large or numerous values, use splited mode:

`typescript
// Each key stored in separate file
const cache = await db.dictionary('cache', { splited: true });

await cache.set('largeData', { / large object / });
await cache.set('config', { / another object / });

// More efficient for partial updates
await cache.set('config', newConfigValue);
`

$3

`typescript
const collection = await db.collection('users');

collection.on('written', () => {
console.log('Data successfully saved');
});

collection.on('error', (error) => {
console.error('Write error:', error);
});

await collection.insert({ name: 'Alice' });
`

$3

`typescript
const collection = await db.collection('users');

try {
await collection.insert({ name: 'Bob' });
} catch (error) {
console.error('Failed to insert:', error);
}
`

$3

NEW FEATURE: Synchronous methods for loading configuration at application startup.

Use case: Perfect for applications where you need to load JWT secrets, database URLs, and other configuration synchronously before the application starts.

Important: Synchronous methods bypass file locking. Only use them in single-threaded contexts like application startup where there's no concurrent access.

#### Collection Synchronous Methods

`typescript
import { Collection } from 'nosql-file';

const config = new Collection('config', './data');

// Load data synchronously at startup
config.loadSync();

// Read configuration (already loaded)
const jwtSecret = config.find({ key: 'JWT_SECRET' })[0]?.value;

// Write synchronously (if needed during startup)
config.insertSync({ key: 'NEW_KEY', value: 'new-value' });
config.updateSync({ key: 'PORT' }, { value: 3001 });
config.deleteSync({ key: 'OLD_KEY' });
config.clearSync();
`

#### Dictionary Synchronous Methods

`typescript
import { Dictionary } from 'nosql-file';

const config = new Dictionary('app-config', './data');

// Load configuration synchronously
config.loadSync();

// Access values immediately
const jwtSecret = config.get('JWT_SECRET') as string;
const databaseUrl = config.get('DATABASE_URL') as string;
const port = config.get('PORT') as number;

// Set values synchronously
config.setSync('API_KEY', 'abc123');
config.deleteSync('OLD_KEY');
config.clearSync();
`

#### Application Startup Integration Example

`typescript
// main.ts
import { NestFactory } from '@nestjs/core';
import { AppModule } from './app.module';
import { Dictionary } from 'nosql-file';
import * as path from 'path';

async function bootstrap() {
// Load config synchronously before app starts
const config = new Dictionary('config', path.join(__dirname, '../config'), 'yaml');
config.loadSync();

const port = config.get('PORT') as number || 3000;
const jwtSecret = config.get('JWT_SECRET') as string;

// Set environment variables from config
process.env.JWT_SECRET = jwtSecret;
process.env.PORT = port.toString();

// Start your application framework
// const app = await createApp();
// await app.listen(port);
console.log(Application is running on: http://localhost:${port});
}
bootstrap();
`

#### Available Synchronous Methods

Collection:
- loadSync() - Load documents from disk
- insertSync(document) - Insert a document
- updateSync(query, updates) - Update documents
- deleteSync(query) - Delete documents
- clearSync() - Clear all documents
- discardSync() - Reload from disk

Dictionary:
- loadSync() - Load key-value pairs from disk
- setSync(key, value) - Set a key-value pair
- deleteSync(key) - Delete a key
- clearSync() - Clear all keys
- discardSync() - Reload from disk

Remember: These methods are only safe when:
- Used at application startup (single-threaded context)
- No concurrent access to the same files
- Not mixed with async operations on the same instance

See examples/10-sync-config.ts for complete usage examples.

$3

File locking automatically handles concurrent access:

`typescript
const dict = await db.dictionary('shared');

// Multiple concurrent writes are automatically queued
Promise.all([
dict.set('key1', 'value1'),
dict.set('key2', 'value2'),
dict.set('key3', 'value3')
]);
`

Storage Structure

$3

`
./data/
users.yaml # All users in one file
config.yaml # All config in one file
`

$3

`
./data/
cache/ # Cache dictionary directory
user_1.yaml
user_2.yaml
user_3.yaml
`

Configuration

$3

- Format: YAML
- Collection: array-based storage, single file per collection
- Dictionary: key-value storage, single file per dictionary
- Write mode: async (wait for completion)
- Lock timeout: 5 seconds

$3

`typescript
import { FileLockManager } from 'nosql-file';

const lockManager = new FileLockManager();
const collection = new Collection('users', './data', 'yaml', lockManager);
`

Testing

Run the test suite:

`bash
npm test
`

Run tests with coverage:

`bash
npm test -- --coverage
`

Performance Considerations

1. Splited Mode: Use for dictionaries with large or numerous values
2. Fast Mode: Use when fire-and-forget is acceptable (background writes)
3. Memory Mode: Use for temporary changes before explicit sync
4. Batch Operations: Group multiple operations before syncing

Limitations

- File-based storage is slower than in-memory databases
- Not suitable for high-frequency write operations (thousands per second)
- Best suited for configuration, caching, and moderate data persistence
- Large datasets should use splited mode for dictionaries

Data Formats

$3

Clean, human-readable format. Default choice.

`yaml
- id: 1
name: Alice
age: 30
- id: 2
name: Bob
age: 25
`

$3

Standard JSON format. Use when YAML is not preferred.

`json
[
{ "id": 1, "name": "Alice", "age": 30 },
{ "id": 2, "name": "Bob", "age": 25 }
]
`

Best Practices

1. Always await database operations to ensure data consistency
2. Use typed collections for better type safety
3. Handle errors appropriately with try/catch
4. Call db.close() before application shutdown
5. Use splited mode for dictionaries with large values
6. Prefer async mode for critical writes, fast mode for non-critical
7. Monitor the 'error' event for production applications

License

MIT

Contributing

Contributions are welcome. Please ensure tests pass and coverage remains above 95%.

Changelog

$3

- Breaking Change: Renamed main class from Database to NoSqlFile to avoid naming conflicts
- All functionality remains identical, only the class name changed
- Update your imports: import { NoSqlFile } from 'nosql-file' instead of Database`

$3

- Initial release
- Collection and Dictionary support
- YAML and JSON formats
- File locking for concurrent access (read/write locks)
- Multiple write modes (async, fast, memory)
- Comprehensive test suite (181 tests)
- Full TypeScript support
- GitHub Actions CI/CD pipelines