Disk-first structured logger for Node.js with local ingestion to Microsoft SQL Server (MSSQL)
npm install unified-logger-sdk> 📦 npm package: https://www.npmjs.com/package/unified-logger-sdk
Unified Logger SDK is a durable, disk-first logging system for backend services that require reliable log persistence, structured logging, and database-backed querying without introducing centralized logging infrastructure or vendor lock-in.
The SDK is designed for production systems where log loss, partial writes, or tight coupling to external services is unacceptable.
⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻
Core Principles
• Disk-first logging (logs are never written directly to a database)
• Structured, schema-stable log format (NDJSON)
• Deterministic behavior under high load
• No always-on servers or collectors
• No external SaaS dependency
• Compatible with monoliths and microservices
• Works on Windows, Linux, and macOS
• Log schema is backward-compatible within a major version
⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻
Architecture Overview
Unified Logger SDK is intentionally split into two components:
1. Node.js Logging SDK (Public)
• Runs inside the application process
• Writes structured logs to local disk
• Never blocks request execution
• Minimal runtime overhead
2. Native Ingestion Engine (Embedded)
• Implemented in Go for reliability and performance
• Runs locally as a background process
• Reads log files from disk
• Batches records and retries failures
• Persists logs into Microsoft SQL Server
Application (Node.js)
|
| structured logs (NDJSON)
v
Local Disk (append-only files)
|
| background ingestion
v
Embedded Ingestor (Go)
|
v
Microsoft SQL Server
This separation guarantees log durability even if:
• The application crashes
• The database is temporarily unavailable
• The system restarts mid-write
⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻
Non-Goals
This SDK intentionally does not provide:
• Centralized SaaS log ingestion
• Real-time streaming pipelines
• Log visualization dashboards
• Distributed tracing
• Metrics aggregation
Unified Logger SDK focuses only on durable logging.
⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻
Installation
npm install unified-logger-sdk
The package includes platform-specific ingestion binaries.
No system-level dependencies are required.
⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻
Supported Database
Currently supported database:
• Microsoft SQL Server (MSSQL)
PostgreSQL and MySQL are intentionally not enabled in this release to guarantee correctness and stability.
⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻
Quick Start
const {
createLogger,
startIngestor,
buildIngestorConfig
} = require("unified-logger-sdk");
const logger = createLogger({
service: { name: "example-service", version: "1.0.0" },
logging: { logDir: "./logs" }
});
startIngestor(
buildIngestorConfig({
logDir: "./logs",
db: {
type: "mssql",
connection: process.env.MSSQL_CONNECTION_STRING
}
})
);
logger.info("application started");
This is the minimum required setup.
⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻
Basic Usage
Create a Logger
const { createLogger } = require("unified-logger-sdk");
const logger = createLogger({
service: {
name: "order-service",
version: "1.0.0"
},
logging: {
logDir: "./logs"
}
});
logger.info("service started");
logger.error("order processing failed", {
why_code: "ORDER_FAILED",
orderId: 123
});
Logs are written immediately to disk in NDJSON format.
⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻
Enable Database Ingestion (MSSQL)
const {
startIngestor,
buildIngestorConfig
} = require("unified-logger-sdk");
startIngestor(
buildIngestorConfig({
logDir: "./logs",
db: {
type: "mssql",
connection: process.env.MSSQL_CONNECTION_STRING,
table: "unified_logs"
}
})
);
Important Notes
• The ingestor runs as a local background process
• The database table is auto-created if missing
• Inserts are batched
• Transient failures are retried
• Duplicate log records are prevented
• No cron jobs or manual execution required
Start the ingestor once per application instance, typically during service startup.
⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻
Log Storage Model
Directory Structure
logs/
├── active/ # logs currently being written
├── processed/ # successfully ingested files
└── archive/ # files that failed ingestion after retries
• Files are never deleted automatically
• Log files are immutable once processed
• Failed files are preserved for investigation
⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻
Database Schema (5W Model)
The ingestor uses a fixed 5W schema:
CREATE TABLE unified_logs (
log_id UNIQUEIDENTIFIER PRIMARY KEY,
log_time DATETIME2 NOT NULL,
who_service NVARCHAR(100) NOT NULL,
what_level NVARCHAR(10) NOT NULL,
what_message NVARCHAR(1000) NOT NULL,
why_code NVARCHAR(100) NULL,
payload NVARCHAR(MAX) NULL,
ingested_at DATETIME2 NOT NULL DEFAULT SYSUTCDATETIME()
);
This schema is stable and versioned.
⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻
Operational Characteristics
• Safe to restart application or ingestor at any time
• Safe to re-run ingestion multiple times
• Logging continues even if MSSQL is down
• No in-memory buffering that risks OOM
• No long-running threads inside the Node.js process
⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻
Security Model
• No outbound network calls
• No telemetry
• No data exfiltration
• Database credentials remain local
• Suitable for restricted enterprise environments
⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻
Versioning
Semantic Versioning is enforced:
• Patch: bug fixes only
• Minor: backward-compatible features
• Major: schema or behavioral changes
⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻⸻
Status
Production-ready (MSSQL only)
Other databases may be added only after correctness is guaranteed.