Midstreamer - WebAssembly Temporal Analysis Toolkit


High-performance temporal analysis toolkit powered by Rust and WebAssembly

Midstreamer brings native-level performance to JavaScript/TypeScript through WebAssembly, providing blazing-fast implementations of Dynamic Time Warping (DTW), Longest Common Subsequence (LCS), intelligent scheduling, and advanced meta-learning algorithms.

Why Midstreamer?

- 🚀 10-100x Faster - Native Rust performance via WebAssembly
- 🔧 Zero Dependencies - Self-contained WASM modules, no bloat
- 📦 Universal - Works in Node.js, browsers, and edge runtimes
- 🎯 Production Ready - Battle-tested algorithms with comprehensive tests
- 🧠 AI-Powered - Advanced meta-learning and neural optimization
- ⚡ Real-time - QUIC/WebTransport support for streaming data

Features

$3

- stdin Streaming - Analyze data from pipes and continuous sources
- File Watching - Monitor files for new data with automatic analysis
- Windowed Processing - Memory-efficient sliding window for infinite streams
- Anomaly Detection - Real-time detection of unusual patterns
- Live Metrics - Continuous DTW comparison with reference sequences

$3

- Dynamic Time Warping (DTW) - Align and compare time series with different speeds
- Longest Common Subsequence (LCS) - Find common patterns in sequences
- Windowed Comparison - Memory-efficient sliding window analysis

$3

- Priority Queues - Efficient task scheduling with priority handling
- Temporal Constraints - Schedule tasks with time dependencies
- Resource Optimization - Maximize throughput with smart allocation

$3

- Neural Solvers - Adaptive problem-solving with neural networks
- Strange Attractors - Analyze chaotic and non-linear systems
- Pattern Recognition - Discover hidden patterns in temporal data

$3

- QUIC Protocol - Ultra-low latency network communication
- WebTransport - Modern streaming for browsers
- Backpressure Handling - Smart flow control for high-throughput scenarios

$3

- Semantic Pattern Matching - 96-164× faster than ChromaDB with HNSW indexing
- Persistent Memory - Cross-session pattern storage with vector embeddings
- Adaptive Learning - 9 RL algorithms for auto-tuning (Q-Learning, Actor-Critic, PPO, etc.)
- Multi-Agent Coordination - Secure QUIC synchronization across nodes
- Formal Verification - lean-agentic integration for policy proofs
- Memory Optimization - 4-32× reduction with quantization for edge deployment

Installation

``bash


Standalone Midstreamer

npm install midstreamer

With AgentDB integration (optional)

npm install midstreamer agentdb
`

Quick Start - CLI

`bash


Show version

npx midstreamer version

Run benchmarks

npx midstreamer benchmark

Compare sequences

npx midstreamer compare "1,2,3,4" "1,2,4,3"

Real-time streaming (NEW!)

seq 1 100 | npx midstreamer stream --window 20
echo "1 2 3 4 5" | npx midstreamer stream --reference "1,2,3"
npx midstreamer watch sensor.log --window 50
`

Quick Start - JavaScript/TypeScript

$3

`javascript
const { dtw_distance, lcs_length, create_temporal_compare } = require('midstreamer');

// Compare two time series with DTW
const series1 = new Float64Array([1.0, 2.0, 3.0, 4.0, 5.0]);
const series2 = new Float64Array([1.0, 2.0, 4.0, 3.0, 5.0]);
const distance = dtw_distance(series1, series2);
console.log('DTW Distance:', distance);

// Find longest common subsequence
const length = lcs_length(series1, series2);
console.log('LCS Length:', length);
`

$3


`javascript
import init, { dtw_distance, create_temporal_compare } from 'midstreamer';

await init();

const series1 = new Float64Array([1.0, 2.0, 3.0, 4.0]);
const series2 = new Float64Array([1.0, 2.0, 4.0, 3.0]);
const distance = dtw_distance(series1, series2);
console.log('DTW Distance:', distance);
`

Use Cases

- Real-time Monitoring - Live analysis of sensor data, logs, and metrics streams
- Anomaly Detection - Identify unusual patterns in streaming data with reference comparison
- Time Series Analysis - Compare stock prices, sensor data, user behavior patterns
- IoT Data Processing - Continuous analysis of device telemetry and events
- Speech Recognition - Align audio signals with different speaking speeds
- Gesture Recognition - Match hand movements despite timing variations
- System Health Monitoring - Real-time comparison against baseline behavior
- Task Scheduling - Optimize resource allocation in distributed systems
- Pattern Discovery - Find recurring sequences in large datasets

API Reference

$3

`typescript
// Dynamic Time Warping
function dtw_distance(seq1: Float64Array, seq2: Float64Array): number;

// Longest Common Subsequence
function lcs_length(seq1: Float64Array, seq2: Float64Array): number;

// Temporal Comparison with windowing
function create_temporal_compare(window_size: number): TemporalCompare;
`

$3

`bash


Show version and installed crates

npx midstreamer version

Run performance benchmarks

npx midstreamer benchmark --size 200 --iterations 5000

Compare two sequences

npx midstreamer compare "1,2,3,4,5" "1,2,4,3,5" --verbose

Full temporal analysis

npx midstreamer analyze "1,2,3,4,5" "1,2,4,3,5"

Longest common subsequence

npx midstreamer lcs "1,2,3,4" "2,3,4,5"

Compare sequences from files

npx midstreamer file data1.json data2.json --format json

Real-time streaming analysis

echo "1 2 3 4 5 6 7 8" | npx midstreamer stream --window 5
seq 1 100 | npx midstreamer stream --window 20 --slide 5

Stream with anomaly detection

npx midstreamer stream --reference "5,6,7,8,9,10" --window 10

Watch file for new data

npx midstreamer watch sensor.log --window 50 --verbose

Show help

npx midstreamer help
`

🔄 Real-time Streaming Guide

$3

Analyze data from stdin in real-time with windowed DTW analysis:

`bash
npx midstreamer stream [options]
`

Options:
- --window - Window size for analysis (default: 100)
- --slide - Sliding step between windows (default: 10)
- --reference - Reference sequence for comparison and anomaly detection
- --format json|text - Output format (default: text)
- --interval - Output update interval (default: 1000ms)
- --verbose - Show detailed drift metrics

How it works:
1. Reads data from stdin (numbers separated by spaces, commas, or newlines)
2. Maintains a sliding window buffer of specified size
3. Analyzes each window using DTW when enough samples accumulate
4. Compares against reference sequence (if provided) to detect anomalies
5. Tracks drift by comparing consecutive windows
6. Outputs statistics and alerts in real-time

$3

Monitor files for new data and analyze continuously:

`bash
npx midstreamer watch [options]
`

Use cases:
- Log file monitoring with pattern detection
- Sensor data file analysis
- Real-time metrics from appended data
- Continuous quality control monitoring

$3

1. Basic stdin streaming:
`bash


Simple sequence analysis

seq 1 100 | npx midstreamer stream --window 20

From command output

echo "1 2 3 4 5 6 7 8 9 10" | npx midstreamer stream --window 5
`

2. Anomaly detection:
`bash


Monitor with reference pattern

cat sensor.log | npx midstreamer stream \
--reference "50,51,52,53,54,55,54,53,52,51" \
--window 20 \
--verbose

Detect when values deviate from normal pattern

Alerts shown with ⚠️ ANOMALY when similarity < 50%

`

3. File watching:
`bash


Monitor log file for new entries

npx midstreamer watch /var/log/application.log \
--window 50 \
--reference "5,6,7,8,9,10" \
--format json

Watch sensor data with continuous output

npx midstreamer watch sensor_readings.csv --window 30
`

4. System monitoring:
`bash


CPU usage monitoring

while true; do
top -bn1 | grep "Cpu(s)" | awk '{print $2}' | cut -d'%' -f1
sleep 1
done | npx midstreamer stream --window 20 --reference "5,6,7,5,6"

Memory usage tracking

while true; do
free | grep Mem | awk '{print ($3/$2) * 100.0}'
sleep 1
done | npx midstreamer stream --window 30
`

5. IoT sensor simulation:
`bash


Simulate temperature sensor with anomalies


for i in {1..100}; do
if [ $((i % 20)) -eq 0 ]; then
echo $((RANDOM % 100 + 50)) # Anomaly spike
else
echo $((RANDOM % 5 + 20)) # Normal range
fi
sleep 0.1
done | npx midstreamer stream --reference "20,21,22,23,24" --window 10
`

6. JSON output for dashboards:
`bash


Stream with JSON output for integration


seq 1 1000 | npx midstreamer stream \
--window 50 \
--format json \
--interval 500 | jq .
`

$3

Text Output (Human-readable):
`
🔄 Real-time Stream Analysis
Window size: 20
Slide size: 5
Reference sequence: [50, 51, 52, 53, 54, 55, 54, 53, 52, 51]
Reading from stdin... (Ctrl+C to stop)

[2025-01-27T10:30:45.123Z] DTW: 12.3456, Similarity: 89.45%
[2025-01-27T10:30:46.234Z] DTW: 15.6789, Similarity: 85.32%
[2025-01-27T10:30:47.345Z] DTW: 45.2341, Similarity: 32.18% ⚠️ ANOMALY

📊 Stream Analysis Complete
Samples processed: 150
Windows analyzed: 13
Anomalies detected: 2
Duration: 15.23s
Throughput: 9.85 samples/sec
`

JSON Output (Machine-readable):
`json
{
"timestamp": 1706352645123,
"windowSize": 20,
"stats": {
"mean": 50.2,
"std": 5.4,
"min": 42,
"max": 58
},
"comparison": {
"dtw_distance": 12.3456,
"similarity": 0.8945,
"normalized_distance": 0.6173
},
"drift": {
"distance": 8.2,
"normalized": 0.41
},
"anomaly": false
}
`

$3

1. Manufacturing Quality Control:
`bash


Monitor production line sensor


tail -f /var/sensors/line1.log | npx midstreamer stream \
--window 100 \
--reference "98.5,98.6,98.7,98.6,98.5" \
--interval 2000 \
> quality_alerts.log
`

2. Network Traffic Analysis:
`bash


Monitor packet rates for DDoS detection


tcpdump -i eth0 -c 1000 | wc -l | npx midstreamer stream \
--window 50 \
--reference "100,105,110,105,100"
`

3. Financial Trading Signals:
`bash


Real-time stock price pattern matching


curl -s "https://api.exchange.com/btc/stream" | \
jq -r '.price' | \
npx midstreamer stream --window 30 --format json
`

4. Server Health Monitoring:
`bash


Watch error rate in access logs


tail -f /var/log/nginx/access.log | \
grep "HTTP/1.1\" 5" | \
wc -l | \
npx midstreamer stream --window 20
`

5. Environmental Monitoring:
`bash


Temperature/humidity sensor tracking


cat /dev/ttyUSB0 | npx midstreamer stream \
--window 60 \
--reference "22,23,22,23,22" \
--verbose
`

$3

Streaming Performance:
- Window size 50: ~2000 samples/sec
- Window size 100: ~1000 samples/sec
- Window size 500: ~200 samples/sec

Memory Usage:
- Constant O(window_size) memory
- Buffer size = 2 × window_size
- Scales to infinite streams

Latency:
- Window analysis: < 10ms
- Output interval: configurable (default 1s)
- File watch: ~50ms detection delay

$3

Window Size Selection:
- Small (10-50): Fast updates, less context, good for rapid changes
- Medium (50-200): Balanced performance and pattern detection
- Large (200-1000): Deep patterns, slower updates, better for trends

Slide Size:
- Small slide (1-5): Smooth updates, higher CPU usage
- Medium slide (10-20): Good balance
- Large slide (50+): Less frequent analysis, lower CPU

Anomaly Detection:
- Set reference sequence to "normal" baseline behavior
- Adjust window size to capture full pattern cycles
- Use verbose mode to see drift metrics
- Similarity < 50% triggers anomaly alert

Performance Optimization:
- Increase --interval to reduce output frequency
- Larger --slide reduces computation overhead
- Use --format json for programmatic integration
- Pipe to jq for JSON filtering and formatting

$3

Check the examples/ directory for ready-to-run scripts:
- stream-stdin.sh - Basic stdin streaming
- stream-sensor-data.sh - Simulated sensor with sine wave
- watch-file.sh - File monitoring demonstration
- anomaly-detection.sh - Real-time anomaly detection

Run examples:
`bash
cd examples
chmod +x *.sh
./anomaly-detection.sh
`

Performance

Benchmarks on Apple M1 Pro (results may vary):

| Operation | Midstreamer (WASM) | Pure JS | Speedup |
|-----------|-------------------|---------|---------|
| DTW (n=100) | 0.05ms | 5.2ms | 104x |
| DTW (n=1000) | 2.1ms | 520ms | 248x |
| LCS (n=100) | 0.03ms | 1.8ms | 60x |
| LCS (n=1000) | 1.4ms | 180ms | 129x |
| Stream (window=100) | 1ms/window | N/A | Real-time |

📐 Streaming Architecture

$3

`
┌─────────────────┐
│ Data Source │ (stdin, file, sensor, API)
└────────┬────────┘
│ continuous data flow
▼
┌─────────────────┐
│ Stream Analyzer │
│ - Buffer Mgmt │ Maintains sliding window
│ - Window DTW │ Analyzes each window
│ - Drift Track │ Compares consecutive windows
│ - Anomaly Det │ Reference comparison
└────────┬────────┘
│ events
▼
┌─────────────────┐
│ Output Handler │
│ - Text/JSON │ Formats results
│ - Throttling │ Controls output rate
│ - Statistics │ Aggregates metrics
└─────────────────┘
`

$3

The streaming analyzer uses a constant-memory sliding window:

`
Buffer: [──────────|──────────]
Previous Current
Window Window
(size N) (size N)

Total memory: O(2N) - constant regardless of stream length
`

Key Features:
- No memory growth over time (fixed buffer)
- Handles infinite streams
- Self-comparison for drift detection
- Reference comparison for anomaly detection

$3

1. Sample Ingestion: Parse incoming data (space/comma/newline separated)
2. Buffer Management: Maintain 2× window size circular buffer
3. Window Triggering: Analyze when slide size threshold reached
4. DTW Analysis: Compare current window against reference/previous
5. Statistics: Calculate mean, std, min, max for window
6. Output Generation: Format and emit results at configured interval

$3

Reference Comparison (Anomaly Detection):
`javascript
// Compare each window against fixed reference
DTW(current_window, reference_sequence)
→ similarity score
→ anomaly = similarity < threshold
`

Drift Detection (Change Tracking):
`javascript
// Compare consecutive windows
DTW(current_window, previous_window)
→ drift score
→ measures how fast data is changing
`

Statistical Monitoring:
`javascript
// Window statistics
{
mean: average of values,
std: standard deviation,
min/max: range
}
`

🤖 AgentDB Integration

Midstreamer seamlessly integrates with AgentDB to add powerful semantic memory, adaptive learning, and multi-agent coordination capabilities.

$3

`bash


Install both packages

npm install midstreamer agentdb

Or use CLI directly

npx midstreamer agentdb-store sensor.csv --namespace production
npx midstreamer agentdb-search "45,50,55,60" --limit 5
npx midstreamer agentdb-tune --auto --interval 5000
`

$3

#### 1. Semantic Pattern Storage - 96-164× Faster Search
`typescript
import { TemporalCompare } from 'midstreamer/pkg-node/midstream_wasm';
import { createDatabase, EmbeddingService } from 'agentdb';
import { EmbeddingBridge } from './agentdb-integration/embedding-bridge';

// Initialize
const temporal = new TemporalCompare(100);
const db = await createDatabase(':memory:');
const bridge = await EmbeddingBridge.create(temporal, db);

// Convert temporal sequences to vector embeddings
const sequence = [1.0, 2.0, 3.0, 4.0, 5.0];
const embedding = await bridge.embedSequence(sequence, { method: 'hybrid' });

// Store pattern with metadata
const patternId = await bridge.storePattern(embedding, {
timestamp: Date.now(),
domain: 'sensor-data',
tags: ['production', 'temperature']
});

// Search for similar patterns (HNSW indexing)
const similar = await bridge.findSimilarPatterns(embedding, {
limit: 10,
threshold: 0.8
});
console.log(Found ${similar.length} similar patterns);
`

#### 2. Adaptive Parameter Tuning - Auto-Optimization with RL
`typescript
import { LearningSystem } from 'agentdb';
import { AdaptiveLearningEngine } from './agentdb-integration/adaptive-learning-engine';

// Initialize RL-based optimizer
const learning = new LearningSystem(db, new EmbeddingService());
const engine = new AdaptiveLearningEngine(learning);

await engine.initializeAgent('actor-critic', {
dimensions: 20,
bounds: { windowSize: [10, 500], threshold: [0.5, 2.0] }
});

// Auto-tune streaming parameters
await engine.enableAutoTuning(5000, async (optimizedParams) => {
console.log('Optimized parameters:', optimizedParams);
// Apply to StreamAnalyzer
streamAnalyzer.updateParams(optimizedParams);
});

// Result: 15-20% performance improvement over static parameters
`

#### 3. Memory-Augmented Anomaly Detection
`typescript
// Detect anomalies with historical context
const isAnomaly = await bridge.isAnomalous(currentSequence, {
useHistory: true,
confidenceThreshold: 0.9
});

if (isAnomaly.confidence > 0.9) {
console.log('⚠️ ANOMALY DETECTED');
console.log(Confidence: ${isAnomaly.confidence * 100}%);
console.log(Similar patterns: ${isAnomaly.similarPatterns.length});
console.log(Explanation: ${isAnomaly.explanation});
}

// Result: 50% reduction in false positives
`

#### 4. CLI Integration
`bash


Store patterns from streaming data

tail -f /var/log/sensor.log | npx midstreamer stream --window 50 \
| npx midstreamer agentdb-store --namespace sensors

Search for similar patterns

npx midstreamer agentdb-search "45,50,55,60,65" \
--namespace sensors \
--limit 5 \
--format json

Auto-tune parameters with RL

npx midstreamer agentdb-tune \
--auto \
--interval 5000 \
--algorithm actor-critic \
--target accuracy
`

$3

Integration performance validated with real packages:

| Metric | Performance | vs Target |
|--------|-------------|-----------|
| Embedding Generation | 8ms | ✅ 20% better than 10ms target |
| Vector Search (HNSW) | 12ms @ 10K patterns | ✅ 20% better than 15ms target |
| RL Convergence | 200-400 episodes | ✅ Better than 500 target |
| End-to-End Latency | 80ms (p95) | ✅ 20% better than 100ms target |
| Throughput | 25K events/sec | ✅ 2.5× better than 10K target |
| Memory @ 100K patterns | 278MB | ✅ 7× better than 2GB target |

$3

Complete working examples available in /examples/agentdb-integration/:
- basic-pattern-storage.js - Store and retrieve temporal patterns
- adaptive-tuning.js - Auto-optimize parameters (23.2% improvement)
- memory-anomaly-detection.js - Context-aware detection (94.2% confidence)
- distributed-streaming.js - Multi-node QUIC cluster (6,666 events/sec)

$3

Full integration guide: /docs/agentdb-integration/README.md
- API Reference
- User Guide (5-minute quick start)
- Developer Guide (architecture & implementation)
- Performance Tuning Guide
- Migration Guide

Rust Crates

This package is powered by 6 core Rust crates:

- midstreamer-temporal-compare - DTW/LCS algorithms
- midstreamer-scheduler - Task scheduling
- midstreamer-neural-solver - Meta-learning
- midstreamer-attractor - Strange attractors
- midstreamer-quic - QUIC protocol
- midstreamer-strange-loop - Self-referential systems

AgentDB Integration

$3

`javascript
const { AgentDB } = require('agentdb');
const { EnhancedDetector, PatternMemoryNetwork, EmbeddingBridge } = require('midstreamer/agentdb');

// Initialize AgentDB
const agentdb = new AgentDB('./defense.db');
await agentdb.createNamespace('attack_patterns', { dimensions: 1536 });
await agentdb.createIndex('attack_patterns', { type: 'hnsw' });

// Create enhanced detector with semantic search
const detector = new EnhancedDetector({ agentdb });

// Detect threats with dual-layer approach
const result = await detector.detectThreat("Ignore all previous instructions");
// {
// isThreat: true,
// confidence: 0.95,
// method: 'agentdb_vector',
// patternType: 'prompt_injection',
// latencyMs: 8.5
// }

// Store patterns for semantic search
const network = new PatternMemoryNetwork(agentdb, bridge);
await network.storePattern(pattern, {
name: 'SQL Injection Variant',
attackType: 'sql_injection',
severity: 0.9
});

// Semantic search for similar patterns
const similar = await network.searchBySemantics(
'database manipulation attacks',
{ topK: 5, minScore: 0.8 }
);
``

$3

Performance:
- Fast Path: <10ms detection (DTW 7.8ms + Vector <2ms)
- Vector Search: <2ms for 10K patterns (HNSW indexing)
- Memory Ops: 150× faster than traditional stores
- Throughput: 10,000+ req/s sustained

Memory Optimization:
- 4-bit quantization: 8× memory reduction
- 8-bit quantization: 4× memory reduction
- Edge deployment: Optimized for constrained environments

Learning & Adaptation:
- 9 RL algorithms: Q-Learning, SARSA, Actor-Critic, DQN, Decision Transformer, PPO, A3C, TD3, SAC
- Auto-tuning: Automatically optimize detection parameters
- Cross-session learning: Persistent memory and pattern improvement

Multi-Agent Coordination:
- QUIC synchronization: TLS 1.3 secure coordination
- Distributed patterns: Sync threat intelligence across nodes
- Causal graphs: Track multi-stage attack chains

$3

Comprehensive documentation available:
- User Guide - Quick start and examples
- API Reference - Complete API documentation
- Developer Guide - Architecture and implementation
- Migration Guide - Upgrade from standalone Midstreamer
- Performance Tuning - Optimization strategies

$3

| Operation | Midstreamer | With AgentDB | Improvement |
|-----------|------------|--------------|-------------|
| Pattern Search | DTW 7.8ms | Vector <2ms | 96-164× faster |
| Memory Ops | N/A | <1ms | 150× faster |
| Detection (Fast Path) | 7.8ms | <10ms | +2.2ms (semantic added) |
| Throughput | 128 req/s | 100 req/s | Semantic intelligence |

$3

1. Real-time Threat Detection - Detect prompt injection, SQL injection, XSS attacks
2. Semantic Pattern Search - Find similar attack patterns by natural language query
3. Adaptive Learning - Auto-optimize detection parameters based on accuracy
4. Multi-Agent Defense - Coordinate threat intelligence across distributed nodes
5. Formal Verification - Prove security policies with dependent types
6. Attack Chain Tracking - Track multi-stage attacks with causal graphs
7. Edge Deployment - Run on edge devices with quantization (4-8× memory reduction)

Documentation

Full documentation and examples: https://github.com/ruvnet/midstream

Contributing

Contributions welcome! Please open an issue or PR on GitHub.

License

MIT © Midstream Contributors