Tiered storage library with S3, disk, and memory caching
npm install tiered-storageCascading cache that flows hot → warm → cold. Memory, disk, S3—or bring your own.
- Cascading writes - data flows down through all tiers
- Bubbling reads - check hot first, fall back to warm, then cold
- Pluggable backends - memory, disk, S3, or implement your own
- Selective placement - skip tiers for big files that don't need memory caching
- Prefix invalidation - invalidate('user:') nukes all user keys
- Optional compression - transparent gzip
``bash`
npm install tiered-storage
`typescript
import { TieredStorage, MemoryStorageTier, DiskStorageTier, S3StorageTier } from 'tiered-storage'
const storage = new TieredStorage({
tiers: {
hot: new MemoryStorageTier({ maxSizeBytes: 100 1024 1024 }),
warm: new DiskStorageTier({ directory: './cache' }),
cold: new S3StorageTier({ bucket: 'my-bucket', region: 'us-east-1' }),
},
placementRules: [
{ pattern: '**/index.html', tiers: ['hot', 'warm', 'cold'] },
{ pattern: '*/.{jpg,png,gif,mp4}', tiers: ['warm', 'cold'] },
{ pattern: '**', tiers: ['warm', 'cold'] },
],
})
// just set - rules decide where it goes
await storage.set('site:abc/index.html', indexHtml) // → hot + warm + cold
await storage.set('site:abc/hero.png', imageData) // → warm + cold
await storage.set('site:abc/video.mp4', videoData) // → warm + cold
// reads bubble up from wherever it lives
const page = await storage.getWithMetadata('site:abc/index.html')
console.log(page.source) // 'hot'
const video = await storage.getWithMetadata('site:abc/video.mp4')
console.log(video.source) // 'warm'
// nuke entire site
await storage.invalidate('site:abc/')
`
Hot tier stays small and fast. Warm tier has everything. Cold tier is the source of truth.
`
┌─────────────────────────────────────────────┐
│ Cold (S3) - source of truth, all data │
│ ↑ │
│ Warm (disk) - everything hot has + more │
│ ↑ │
│ Hot (memory) - just the hottest stuff │
└─────────────────────────────────────────────┘
Writes cascade down. Reads bubble up.
Items leave upper tiers through eviction or TTL expiration:
`typescript
const storage = new TieredStorage({
tiers: {
// hot: LRU eviction when size/count limits hit
hot: new MemoryStorageTier({
maxSizeBytes: 100 1024 1024,
maxItems: 500,
}),
// warm: evicts when maxSizeBytes hit, policy controls which items go
warm: new DiskStorageTier({
directory: './cache',
maxSizeBytes: 10 1024 1024 * 1024,
evictionPolicy: 'lru', // 'lru' | 'fifo' | 'size'
}),
// cold: never evicts, keeps everything
cold: new S3StorageTier({ bucket: 'my-bucket', region: 'us-east-1' }),
},
defaultTTL: 14 24 60 60 1000, // TTL checked on read
})
`
A file that hasn't been accessed eventually gets evicted from hot (LRU), then warm (size limit + policy). Next request fetches from cold and promotes it back up.
Define once which keys go where, instead of passing skipTiers on every set():
`typescript
const storage = new TieredStorage({
tiers: {
hot: new MemoryStorageTier({ maxSizeBytes: 50 1024 1024 }),
warm: new DiskStorageTier({ directory: './cache' }),
cold: new S3StorageTier({ bucket: 'my-bucket', region: 'us-east-1' }),
},
placementRules: [
// index.html goes everywhere for instant serving
{ pattern: '**/index.html', tiers: ['hot', 'warm', 'cold'] },
// images and video skip hot
{ pattern: '*/.{jpg,png,gif,webp,mp4}', tiers: ['warm', 'cold'] },
// assets directory skips hot
{ pattern: 'assets/**', tiers: ['warm', 'cold'] },
// everything else: warm + cold only
{ pattern: '**', tiers: ['warm', 'cold'] },
],
})
// just call set() - rules handle placement
await storage.set('site:abc/index.html', html) // → hot + warm + cold
await storage.set('site:abc/hero.png', image) // → warm + cold
await storage.set('site:abc/assets/font.woff', font) // → warm + cold
await storage.set('site:abc/about.html', html) // → warm + cold
`
Rules are evaluated in order. First match wins. Cold is always included.
Get data. Returns null if missing or expired.
Get data plus which tier served it.
Store data. Options:
`typescript`
{
ttl: 86400000, // custom TTL
skipTiers: ['hot'], // skip specific tiers
metadata: { ... }, // custom metadata
}
Delete from all tiers.
Delete all keys matching prefix. Returns count.
Renew TTL.
Async iterator over keys.
Stats across all tiers.
Warm up hot tier from warm tier. Run on startup.
Warm up warm tier from cold tier.
`typescript`
new MemoryStorageTier({
maxSizeBytes: 100 1024 1024,
maxItems: 1000,
})
LRU eviction. Fast. Single process only.
`typescript`
new DiskStorageTier({
directory: './cache',
maxSizeBytes: 10 1024 1024 * 1024,
evictionPolicy: 'lru', // or 'fifo', 'size'
})
Files on disk with .meta sidecars.
`typescript`
new S3StorageTier({
bucket: 'data',
metadataBucket: 'metadata', // recommended!
region: 'us-east-1',
})
Works with AWS S3, Cloudflare R2, MinIO. Use a separate metadata bucket—otherwise updating access counts requires copying entire objects.
Implement StorageTier:
`typescript
interface StorageTier {
get(key: string): Promise
set(key: string, data: Uint8Array, metadata: StorageMetadata): Promise
delete(key: string): Promise
exists(key: string): Promise
listKeys(prefix?: string): AsyncIterableIterator
deleteMany(keys: string[]): Promise
getMetadata(key: string): Promise
setMetadata(key: string, metadata: StorageMetadata): Promise
getStats(): Promise
clear(): Promise
// Optional: combine get + getMetadata for better performance
getWithMetadata?(key: string): Promise<{ data: Uint8Array; metadata: StorageMetadata } | null>
}
`
The optional getWithMetadata method returns both data and metadata in a single call. Implement it if your backend can fetch both efficiently (e.g., parallel I/O, single query). Falls back to separate get() + getMetadata() calls if not implemented.
`bash``
cp .env.example .env # add S3 creds
bun run serve
Visit http://localhost:3000 to see it work. Check http://localhost:3000/admin/stats for live cache stats.
MIT