AVALW Distributed Cache Worker - Intelligent caching with size limits and auto-cleanup
npm install @avalw/search-worker
┌─────────────────────────────────────────────────────────────────────────┐
│ AVALW DISTRIBUTED CACHE │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────┐ ┌──────────────┐ ┌─────────────┐ │
│ │ User │────────▶│ avalw.org │────────▶│ Coordinator │ │
│ │ Search │ │ Server │ │ Server │ │
│ └─────────┘ └──────────────┘ └──────┬──────┘ │
│ │ │
│ ┌─────────────────────────┼───────┐ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌──────────┐ ┌──────────┐ ┌──────┐ │
│ │ Worker 1 │ │ Worker 2 │ │ ... │ │
│ │ Cache │ │ Cache │ │ │ │
│ └──────────┘ └──────────┘ └──────┘ │
│ │
│ YOUR WORKER stores search results and serves them when requested │
│ │
└─────────────────────────────────────────────────────────────────────────┘
`
$3
`
1. USER SEARCHES 2. CHECK CACHE 3. SERVE RESULT
"spotify"
┌─────────────┐
┌──────┐ │ Coordinator │ ┌──────────┐
│ User │──search──▶ │ checks │──hit──▶ │ Worker │──result──▶ User
└──────┘ │ workers │ │ serves │
└─────────────┘ └──────────┘
│
miss │
▼
┌──────────┐
│ Server │──result──▶ User + Distribute to Workers
│ process │
└──────────┘
`
$3
`
✅ DOES ❌ DOESN'T
───────────────────────────────── ─────────────────────────────────
• Store search RESULTS • Have the search INDEX
• Serve cached results • Process raw documents
• Use 50-200MB RAM • Use significant CPU
• Use 500MB-2GB disk • Access your files
• Connect via WebSocket • Run arbitrary code
• Earn you API access • Collect your data
`
Quick Start
`bash
1. Get your token at https://worker.avalw.org
2. Run the worker:
npx @avalw/search-worker --token YOUR_TOKEN
`
That's it! Your worker will connect and start contributing.
Installation
`bash
Global install
npm install -g @avalw/search-worker
Or use npx (no install needed)
npx @avalw/search-worker --token YOUR_TOKEN
`
What's New in v2.1.0
`
┌────────────────────────────────────────────────────────────────┐
│ INTELLIGENT CACHE SYSTEM │
├────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Size Limits │ │ Time-Based │ │ Popularity │ │
│ │ 100MB - 2GB │ │ Eviction │ │ Scoring │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
│ │
│ • Cache up to 2GB of search results │
│ • Auto-remove queries unused for 7+ days │
│ • Popular queries stay cached longer │
│ • Background cleanup every hour │
│ │
└────────────────────────────────────────────────────────────────┘
`
Command Line Options
`bash
Basic usage (500MB cache, 7 day retention)
npx @avalw/search-worker --token abc123
Custom cache size (100-2048 MB)
npx @avalw/search-worker --token abc123 --cache-size 1024
Custom retention (1-30 days)
npx @avalw/search-worker --token abc123 --max-age 14
Full customization
npx @avalw/search-worker \
--token abc123 \
--cache-size 1024 \
--max-age 14
`
$3
`
┌──────────────────┬─────────────────┬───────────────────────────────────┐
│ Option │ Default │ Description │
├──────────────────┼─────────────────┼───────────────────────────────────┤
│ -t, --token │ $AVALW_TOKEN │ Your worker authentication token │
│ -s, --cache-size │ 500 │ Max cache size in MB (100-2048) │
│ -a, --max-age │ 7 │ Days to keep unused queries (1-30)│
│ -h, --help │ │ Show help │
│ -v, --version │ │ Show version │
└──────────────────┴─────────────────┴───────────────────────────────────┘
`
$3
`bash
export AVALW_WORKER_TOKEN=your-token-here
export AVALW_CACHE_SIZE_MB=1024
export AVALW_MAX_AGE_DAYS=14
npx @avalw/search-worker
`
Programmatic Usage
`javascript
const { createWorker } = require('@avalw/search-worker');
const worker = createWorker({
workerToken: 'your-token-here',
// Cache configuration
cacheSizeMB: 1024, // 1GB cache
maxAgeDays: 14, // 14 day retention
// Event handlers
onConnected: (w) => console.log('Connected to coordinator'),
onDisconnected: (w) => console.log('Disconnected'),
onCacheHit: (key) => console.log('Served:', key),
onCacheStore: (key) => console.log('Cached:', key),
onError: (err) => console.error('Error:', err)
});
// Connect
worker.connect();
// Get stats
const stats = worker.getStats();
console.log(Cache: ${stats.cache.sizeFormatted} / ${stats.cache.maxSizeFormatted});
console.log(Hit rate: ${stats.hitRate}%);
// Runtime configuration
worker.setCacheSize(2048); // Change to 2GB
worker.setCacheMaxAge(30); // Change to 30 days
// Cleanup
worker.disconnect();
`
API Tiers & Rewards
Your contribution is measured in Core-Hours:
`
┌─────────────────────────────────────────────────────────────────────┐
│ CORE-HOURS FORMULA │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ Core-Hours = Cores × (CPU Speed ÷ 3.0 GHz) × Usage% × Hours │
│ │
│ Example: 8 cores × (3.5 GHz ÷ 3.0 GHz) × 80% × 1 hour │
│ = 8 × 1.17 × 0.8 × 1 = 7.5 Core-Hours │
│ │
└─────────────────────────────────────────────────────────────────────┘
`
$3
`
┌─────────────┬────────────┬───────────┬─────────────────┐
│ Tier │ Core-Hours │ Min Cores │ API Requests │
├─────────────┼────────────┼───────────┼─────────────────┤
│ Starter │ 3 │ 3 │ 1,000/day │
│ Basic │ 12 │ 4 │ 5,000/day │
│ Pro │ 36 │ 6 │ 20,000/day │
│ Business │ 72 │ 8 │ 50,000/day │
│ Enterprise │ 240 │ 16 │ 500,000/day │
└─────────────┴────────────┴───────────┴─────────────────┘
`
Intelligent Cache Details
$3
`
┌─────────────────────────────────────────────────────────────┐
│ EVICTION PRIORITY │
├─────────────────────────────────────────────────────────────┤
│ │
│ 1. EXPIRED TTL │
│ └── Items past their time-to-live (5 min default) │
│ │
│ 2. MAX AGE EXCEEDED │
│ └── Items not accessed for N days (7 days default) │
│ │
│ 3. LOWEST POPULARITY │
│ └── Score = log2(accessCount) × recency × stability │
│ │
│ Priority: 1 > 2 > 3 (expired items removed first) │
│ │
└─────────────────────────────────────────────────────────────┘
`
$3
`
Score = log2(accessCount + 1) × recencyFactor × stabilityFactor
┌─────────────────┬─────────────────────────────────────────┐
│ Factor │ Description │
├─────────────────┼─────────────────────────────────────────┤
│ accessCount │ Number of times the query was accessed │
│ recencyFactor │ 1.0 → 0.0 over 7 days since last access │
│ stabilityFactor │ Higher for consistent access patterns │
└─────────────────┴─────────────────────────────────────────┘
Example: Query accessed 100 times, last accessed 1 day ago
Score = log2(101) × 0.86 × 1.2 = 6.65 × 0.86 × 1.2 = 6.86
`
Resource Usage
`
┌──────────────────────────────────────────────────────────┐
│ TYPICAL RESOURCE USAGE │
├──────────────────────────────────────────────────────────┤
│ │
│ CPU: 1-5% [▓░░░░░░░░░░░░░░░░░░░] minimal │
│ RAM: 50-200MB [▓▓▓▓░░░░░░░░░░░░░░░░] light │
│ Disk: 500MB-2GB [▓▓▓▓▓▓▓▓░░░░░░░░░░░░] configurable │
│ Network: Low [▓▓░░░░░░░░░░░░░░░░░░] WebSocket │
│ │
└──────────────────────────────────────────────────────────┘
`
Privacy & Security
`
┌────────────────────────────────────────────────────────────────┐
│ PRIVACY GUARANTEES │
├────────────────────────────────────────────────────────────────┤
│ │
│ ✓ Workers cache RESULTS only, not the search index │
│ ✓ No IP addresses are logged │
│ ✓ Workers don't see who requested a search │
│ ✓ All communication encrypted (WSS/HTTPS) │
│ ✓ No access to your local files │
│ ✓ Open source - audit the code yourself │
│ │
└────────────────────────────────────────────────────────────────┘
`
Running as a Service
$3
`bash
Create service file
sudo nano /etc/systemd/system/avalw-worker.service
`
`ini
[Unit]
Description=AVALW Search Worker
After=network.target
[Service]
Type=simple
User=nobody
Environment=AVALW_WORKER_TOKEN=your-token-here
Environment=AVALW_CACHE_SIZE_MB=1024
Environment=AVALW_MAX_AGE_DAYS=14
ExecStart=/usr/bin/npx @avalw/search-worker
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
`
`bash
Enable and start
sudo systemctl daemon-reload
sudo systemctl enable avalw-worker
sudo systemctl start avalw-worker
Check status
sudo systemctl status avalw-worker
`
$3
`dockerfile
FROM node:18-alpine
RUN npm install -g @avalw/search-worker
ENV AVALW_WORKER_TOKEN=your-token-here
ENV AVALW_CACHE_SIZE_MB=1024
ENV AVALW_MAX_AGE_DAYS=14
CMD ["avalw-worker"]
`
`bash
docker build -t avalw-worker .
docker run -d --name avalw avalw-worker
`
$3
`bash
pm2 start avalw-worker --name "avalw" -- --token YOUR_TOKEN --cache-size 1024
pm2 save
pm2 startup
`
$3
`batch
@echo off
set AVALW_WORKER_TOKEN=your-token-here
set AVALW_CACHE_SIZE_MB=1024
npx @avalw/search-worker
`
Troubleshooting
`
┌─────────────────────────────────────────────────────────────────┐
│ COMMON ISSUES │
├─────────────────────────────────────────────────────────────────┤
│ │
│ "Connection refused" │
│ └── Check internet connection, firewall settings │
│ │
│ "Invalid token" │
│ └── Get a new token at https://worker.avalw.org │
│ │
│ "Cache full, high eviction rate" │
│ └── Increase --cache-size or decrease --max-age │
│ │
│ "Memory usage too high" │
│ └── Decrease --cache-size (minimum 100MB) │
│ │
│ "Worker keeps disconnecting" │
│ └── Check network stability, worker will auto-reconnect │
│ │
└─────────────────────────────────────────────────────────────────┘
`
System Requirements
`
┌────────────────┬────────────────────────────────────┐
│ Component │ Requirement │
├────────────────┼────────────────────────────────────┤
│ Node.js │ 14.0.0 or higher │
│ CPU │ 3+ cores for Starter tier │
│ RAM │ 512MB+ free │
│ Disk │ 500MB - 2GB for cache │
│ Network │ Stable internet, WebSocket support │
│ OS │ Windows, macOS, Linux │
└────────────────┴────────────────────────────────────┘
`
Support
- Dashboard: worker.avalw.org
- Documentation: worker.avalw.org/protocol
- Email: contact@avalw.com
- Search Engine: avalw.org
License
MIT License - Free to use, modify, and distribute.
---
`
Made with transparency in mind by AVALW
https://avalw.com
``