S3Strata

Object storage abstraction layer with dual-bucket tiered storage and visibility control for Bun/TypeScript.

Features

- Dual-Bucket Architecture: Abstracts HOT (SSD) and COLD (HDD) storage tiers
- Multi-Endpoint Support: Use different S3 providers/regions for each tier (e.g., AWS S3 for HOT, Backblaze B2 for COLD)
- Visibility Control: PUBLIC and PRIVATE file access with path-based separation
- Smart URL Generation: Direct URLs for public files, presigned URLs for private files
- Hot Storage Duration: Set time-based expiration for files in HOT storage with automatic archival to COLD
- Type-safe: Full TypeScript support
- Database Agnostic: Pluggable storage adapter interface - bring your own ORM/database
- Lateral Movement: Easy migration between storage tiers and visibility levels
- Automatic Archival: Housekeeping method to periodically move expired HOT files to COLD storage

Installation

``bash
npm install s3strata


or


bun add s3strata
`

Architecture

$3

- HOT: Fast SSD storage for frequently accessed files
- COLD: Cost-effective HDD storage for archival/infrequent access

$3

- PUBLIC: Publicly accessible files with /public prefix
- Returns direct URLs without presigning
- Example: https://s3.example.com/hot-bucket/public/images/uuid.jpg
- PRIVATE: Protected files with /private prefix
- Returns presigned URLs (default 4-hour expiration)
- Example: Presigned URL to /hot-bucket/private/submissions/uuid.pdf

$3

Files in HOT storage can be flagged with an expiration time (hot_until). When the expiration time is reached, files can be automatically archived to COLD storage using the housekeeping method.

- Permanent HOT: Files with hot_until = null stay in HOT storage indefinitely
- Temporary HOT: Files with hot_until = are marked for archival after that date
- Immediate archival: Set duration = 0 to mark for immediate archival

$3


`
{bucket}/{visibility}/{custom-path}
├── hot-bucket/
│ ├── public/ # Publicly accessible HOT files
│ └── private/ # Protected HOT files
└── cold-bucket/
├── public/ # Publicly accessible COLD files
└── private/ # Protected COLD files
`

Quick Start

$3

S3Strata is database-agnostic. You need to implement the StorageAdapter interface to connect to your database:

`typescript
import type { StorageAdapter, PhysicalFile, StorageTier } from 's3strata';

class MyStorageAdapter implements StorageAdapter {
async create(data: {
storage_tier: StorageTier;
filename: string;
path: string;
hot_until: Date | null;
}): Promise {
// Implement: Create a new file record in your database
}

async findById(id: string | number): Promise {
// Implement: Find a file by ID
}

async update(
id: string | number,
data: {
storage_tier?: StorageTier;
path?: string;
hot_until?: Date | null;
}
): Promise {
// Implement: Update a file record
}

async delete(id: string | number): Promise {
// Implement: Delete a file record
}

async findExpiredHotFiles(): Promise {
// Implement: Find HOT files where hot_until <= now
}
}
`

See examples/prisma-adapter.ts for a complete Prisma implementation example.

$3

S3Strata supports two configuration modes:

Option A: Shared Endpoint (same S3 provider for both tiers)

`typescript
import { FileManager, StorageTier, FileVisibility } from 's3strata';
import type { S3StrataConfig } from 's3strata';

// Configure S3 connection
const config: S3StrataConfig = {
endpoint: 's3.amazonaws.com',
port: 443,
useSSL: true,
accessKey: 'YOUR_ACCESS_KEY',
secretKey: 'YOUR_SECRET_KEY',
hotBucket: 'my-hot-bucket',
coldBucket: 'my-cold-bucket',
// Advanced options (optional)
advanced: {
defaultPresignedUrlExpiration: 7200, // 2 hours (default: 14400 = 4 hours)
maxFileSize: 500 1024 1024, // 500MB (default: Infinity = no limit)
defaultStorageTier: StorageTier.HOT, // default: StorageTier.HOT
defaultVisibility: FileVisibility.PRIVATE, // default: FileVisibility.PRIVATE
},
};

// Create your storage adapter instance
const adapter = new MyStorageAdapter(/ ... /);

// Create FileManager instance
const fileManager = new FileManager(config, adapter);
`

Option B: Separate Endpoints (different S3 providers for each tier)

Perfect for cost optimization - use expensive fast storage for HOT, cheap archival for COLD:

`typescript
const config: S3StrataConfig = {
// HOT storage on AWS S3 (fast, expensive)
hot: {
endpoint: 's3.us-east-1.amazonaws.com',
accessKey: 'AWS_ACCESS_KEY',
secretKey: 'AWS_SECRET_KEY',
bucket: 'my-hot-bucket',
},
// COLD storage on Backblaze B2 (slower, 77% cheaper)
cold: {
endpoint: 's3.us-west-004.backblazeb2.com',
accessKey: 'B2_ACCESS_KEY',
secretKey: 'B2_SECRET_KEY',
bucket: 'my-archive-bucket',
},
};

const fileManager = new FileManager(config, adapter);
`

See MULTI_ENDPOINT.md for detailed multi-endpoint configuration guide.

$3

S3Strata allows you to customize default behaviors through the advanced configuration object:

`typescript
const config: S3StrataConfig = {
// ... your S3 configuration ...
advanced: {
// Default expiration time for presigned URLs in seconds
// Default: 14400 (4 hours)
defaultPresignedUrlExpiration: 3600, // 1 hour

// Maximum allowed file size in bytes
// Default: Infinity (no limit)
maxFileSize: 100 1024 1024, // 100MB

// Default storage tier for new files
// Default: StorageTier.HOT
defaultStorageTier: StorageTier.HOT,

// Default visibility for new files
// Default: FileVisibility.PRIVATE
defaultVisibility: FileVisibility.PRIVATE,
},
};
`

When to use advanced options:
- Set maxFileSize to enforce upload limits and prevent abuse
- Customize defaultPresignedUrlExpiration based on your security requirements
- Change defaultStorageTier if you want new uploads to go to COLD storage by default
- Set defaultVisibility to PUBLIC if most of your files should be publicly accessible

All advanced options are optional and have sensible defaults. You can override defaults on a per-operation basis (e.g., specify expiresIn when calling getUrl()).

$3

`typescript
// Upload a private file to HOT storage
const file = await fileManager.upload(buffer, {
tier: StorageTier.HOT,
visibility: FileVisibility.PRIVATE,
filename: 'document.pdf',
});

// Get URL (returns presigned URL for private files)
const url = await fileManager.getUrl(file, {
expiresIn: 14400, // 4 hours (default)
});

// Make file public (moves from /private to /public)
const updatedFile = await fileManager.setVisibility(file, {
visibility: FileVisibility.PUBLIC,
});

// Get URL (now returns direct public URL)
const publicUrl = await fileManager.getUrl(updatedFile);

// Move to COLD storage (migrates to HDD bucket)
const coldFile = await fileManager.setTier(updatedFile, {
tier: StorageTier.COLD,
});

// Delete file
await fileManager.delete(coldFile);
`

API Reference

$3

#### upload(data: Buffer | Blob, options?: UploadOptions): Promise

Upload a file to object storage.

Options:
- tier?: StorageTier - Storage tier (HOT or COLD), defaults to HOT
- visibility?: FileVisibility - File visibility (PUBLIC or PRIVATE), defaults to PRIVATE
- filename?: string - Custom filename (optional, auto-generated if not provided)
- pathSuffix?: string - Custom path suffix (optional, auto-generated if not provided)
- hotDuration?: number - Duration in seconds to keep file in HOT storage (only for HOT tier)

Example:
`typescript
const file = await fileManager.upload(buffer, {
tier: StorageTier.HOT,
visibility: FileVisibility.PRIVATE,
filename: 'report.pdf',
hotDuration: 7 24 60 * 60, // 7 days
});
`

#### getUrl(file: PhysicalFile, options?: GetUrlOptions): Promise

Generate URL for a file. Returns direct URL for PUBLIC files, presigned URL for PRIVATE files.

Options:
- expiresIn?: number - Presign duration in seconds (default: 14400 = 4 hours), only used for PRIVATE files

Example:
`typescript
const url = await fileManager.getUrl(file, { expiresIn: 3600 }); // 1 hour
`

#### setVisibility(file: PhysicalFile, options: SetVisibilityOptions): Promise

Change file visibility (PUBLIC ↔ PRIVATE). Moves file between /public and /private prefixes.

Options:
- visibility: FileVisibility - New visibility level
- moveFile?: boolean - Whether to move the file immediately (default: true)

Example:
`typescript
const publicFile = await fileManager.setVisibility(file, {
visibility: FileVisibility.PUBLIC,
});
`

#### setTier(file: PhysicalFile, options: SetTierOptions): Promise

Move file between storage tiers (HOT ↔ COLD).

Options:
- tier: StorageTier - New storage tier
- moveFile?: boolean - Whether to move the file immediately (default: true)
- hotDuration?: number - Duration in seconds to keep file in HOT storage (only when tier=HOT)

Example:
`typescript
const coldFile = await fileManager.setTier(file, {
tier: StorageTier.COLD,
});
`

#### setHotDuration(file: PhysicalFile, options: SetHotDurationOptions): Promise

Set hot storage duration for a HOT tier file.

Options:
- duration: number | null - Duration in seconds (null = permanent, 0 = immediate archival)

Example:
`typescript
// Set 30-day expiration
const updated = await fileManager.setHotDuration(file, {
duration: 30 24 60 * 60,
});

// Make permanent (remove expiration)
const permanent = await fileManager.setHotDuration(file, {
duration: null,
});

// Mark for immediate archival
const immediate = await fileManager.setHotDuration(file, {
duration: 0,
});
`

#### delete(file: PhysicalFile): Promise

Delete a file from both storage and database.

Example:
`typescript
await fileManager.delete(file);
`

#### getById(id: string | number): Promise

Get file record from database by ID.

Example:
`typescript
const file = await fileManager.getById(123);
`

#### exists(file: PhysicalFile): Promise

Check if file exists in object storage.

Example:
`typescript
const exists = await fileManager.exists(file);
`

#### archiveExpiredHotFiles(): Promise

Housekeeping method to move expired HOT files to COLD storage. Returns the number of files moved.

Example:
`typescript
// Run periodically via cron job
const movedCount = await fileManager.archiveExpiredHotFiles();
console.log(Archived ${movedCount} files from HOT to COLD);
`

#### listFiles(): Promise

List all file records from the database.

Example:
`typescript
const allFiles = await fileManager.listFiles();
console.log(allFiles);
`

Storage Adapter Examples

$3

`typescript
import { PrismaClient } from '@prisma/client';
import { PrismaStorageAdapter } from 's3strata';

const prisma = new PrismaClient();
const adapter = new PrismaStorageAdapter(prisma);
const fileManager = new FileManager(config, adapter);
`

Prisma Schema:
`prisma
model PhysicalFile {
id Int @id @default(autoincrement())
storage_tier StorageTier
filename String
path String
hot_until DateTime?
created_at DateTime @default(now())
updated_at DateTime @updatedAt
}

enum StorageTier {
HOT
COLD
}
`

$3

`typescript
import type { StorageAdapter, PhysicalFile, StorageTier } from 's3strata';
import { Database } from 'some-database-library';

class CustomStorageAdapter implements StorageAdapter {
constructor(private db: Database) {}

async create(data: {
storage_tier: StorageTier;
filename: string;
path: string;
hot_until: Date | null;
}): Promise {
const result = await this.db.insert('physical_files', data);
return result;
}

async findById(id: string | number): Promise {
return await this.db.findOne('physical_files', { id });
}

async update(
id: string | number,
data: {
storage_tier?: StorageTier;
path?: string;
hot_until?: Date | null;
}
): Promise {
return await this.db.update('physical_files', { id }, data);
}

async delete(id: string | number): Promise {
await this.db.delete('physical_files', { id });
}

async findExpiredHotFiles(): Promise {
return await this.db.findMany('physical_files', {
storage_tier: 'HOT',
hot_until: { lte: new Date() },
});
}
}
`

Complete Usage Example

`typescript
import { FileManager, StorageTier, FileVisibility, PrismaStorageAdapter } from 's3strata';
import { PrismaClient } from '@prisma/client';

// Initialize
const prisma = new PrismaClient();
const adapter = new PrismaStorageAdapter(prisma);
const config = {
endpoint: 's3.amazonaws.com',
accessKey: process.env.S3_ACCESS_KEY!,
secretKey: process.env.S3_SECRET_KEY!,
hotBucket: 'my-hot-bucket',
coldBucket: 'my-cold-bucket',
};
const fileManager = new FileManager(config, adapter);

// Upload a file with 7-day hot storage
const file = await fileManager.upload(fileBuffer, {
tier: StorageTier.HOT,
visibility: FileVisibility.PRIVATE,
filename: 'user-upload.pdf',
hotDuration: 7 24 60 * 60, // 7 days
});

// Get presigned URL (4 hour expiration)
const url = await fileManager.getUrl(file);

// After processing, make it public
const publicFile = await fileManager.setVisibility(file, {
visibility: FileVisibility.PUBLIC,
});

// Get public URL
const publicUrl = await fileManager.getUrl(publicFile);

// Move to cold storage for long-term archival
const archivedFile = await fileManager.setTier(publicFile, {
tier: StorageTier.COLD,
});

// Run housekeeping (e.g., in a cron job)
setInterval(async () => {
const count = await fileManager.archiveExpiredHotFiles();
console.log(Archived ${count} expired files);
}, 60 60 1000); // Every hour
`

Environment Variables

You can use environment variables for configuration:

`bash
S3_ENDPOINT=s3.amazonaws.com
S3_PORT=443
S3_USE_SSL=true
S3_ACCESS_KEY=your-access-key
S3_SECRET_KEY=your-secret-key
S3_BUCKET_HOT=hot-bucket
S3_BUCKET_COLD=cold-bucket
`

TypeScript Support

S3Strata is written in TypeScript and provides full type definitions.

`typescript
import type {
PhysicalFile,
StorageAdapter,
S3StrataConfig,
S3StrataAdvancedOptions,
UploadOptions,
GetUrlOptions,
SetVisibilityOptions,
SetTierOptions,
} from 's3strata';
``

License

MIT

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.