Universal storage manager for handling file operations across S3, GCS, R2, DigitalOcean Spaces, Firebase, and GitHub Actions
npm install @yofix/storageUniversal storage manager for handling file operations across multiple cloud providers. Upload and download files to GitHub Actions Artifacts, Firebase Storage, AWS S3, Google Cloud Storage, Cloudflare R2, DigitalOcean Spaces, or local filesystem with a unified API.
- 7 Storage Providers: GitHub Actions Artifacts, Firebase Storage, AWS S3, Google Cloud Storage, Cloudflare R2, DigitalOcean Spaces, and local filesystem
- Unified API: Same interface across all providers
- Connection Pooling: Singleton registry with credential-keyed caching for connection reuse
- Upload & Download: Full bidirectional file operations
- Glob Patterns: Upload multiple files using glob patterns
- Progress Tracking: Real-time upload progress callbacks
- Type Safe: Full TypeScript support with comprehensive types
- Quick Helpers: quickUpload() and quickDownload() for simple operations
- OAuth Support: Google Cloud Storage supports OAuth refresh tokens
- S3-Compatible: R2 and DO Spaces use S3 protocol with multipart uploads
- GitHub Action: Ready-to-use as a GitHub Action
``bash`
npm install @yofix/storage
`typescript
import { uploadFiles, downloadFiles, quickUpload, quickDownload } from '@yofix/storage'
// Upload files
const result = await uploadFiles({
storage: { provider: 's3', config: { region: 'us-east-1', bucket: 'my-bucket' } },
files: ['dist/*/', 'package.json']
})
// Download files
const downloaded = await downloadFiles({
storage: { provider: 's3', config: { region: 'us-east-1', bucket: 'my-bucket' } },
files: ['config.json', 'data/users.json']
})
// Quick upload (data directly)
const path = await quickUpload(
{ provider: 'gcs', config: { bucket: 'my-bucket' } },
{ path: 'data.json', data: Buffer.from(JSON.stringify(data)) }
)
// Quick download
const buffer = await quickDownload(
{ provider: 'gcs', config: { bucket: 'my-bucket' } },
{ path: 'data.json' }
)
`
`typescript
import { uploadFiles } from '@yofix/storage'
const result = await uploadFiles({
storage: {
provider: 'local',
config: {
directory: './uploads',
createIfNotExists: true,
basePath: 'files'
}
},
files: ['package.json', 'dist/*/'],
verbose: true
})
`
`typescript${progress.filesUploaded}/${progress.totalFiles}
const result = await uploadFiles({
storage: {
provider: 's3',
config: {
region: 'us-east-1',
bucket: 'your-bucket',
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
basePath: 'uploads',
acl: 'public-read'
}
},
files: ['dist/*/'],
onProgress: (progress) => {
console.log()`
}
})
`typescript
// Using service account
const result = await uploadFiles({
storage: {
provider: 'gcs',
config: {
bucket: 'your-bucket',
projectId: 'your-project',
keyFilename: '/path/to/service-account.json'
}
},
files: ['images/*/.{jpg,png}']
})
// Using OAuth refresh token (user-based auth)
const result = await uploadFiles({
storage: {
provider: 'gcs',
config: {
bucket: 'your-bucket',
refreshToken: 'user-refresh-token',
clientId: 'oauth-client-id',
clientSecret: 'oauth-client-secret'
}
},
files: ['uploads/*/']
})
`
GCS Authentication Priority:
1. Access Token (direct token)
2. OAuth Refresh Token (auto-refresh)
3. Service Account Credentials (JSON)
4. Service Account Key File
5. Application Default Credentials
`typescript`
const result = await uploadFiles({
storage: {
provider: 'firebase',
config: {
bucket: 'your-project.appspot.com',
credentials: JSON.parse(process.env.FIREBASE_CREDENTIALS),
basePath: 'uploads'
}
},
files: ['images/*/.{jpg,png}']
})
`typescript`
const result = await uploadFiles({
storage: {
provider: 'r2',
config: {
accountId: 'your-account-id',
bucket: 'your-bucket',
accessKeyId: process.env.R2_ACCESS_KEY_ID,
secretAccessKey: process.env.R2_SECRET_ACCESS_KEY,
basePath: 'uploads',
publicUrl: 'https://cdn.example.com' // Optional custom domain
}
},
files: ['dist/*/']
})
`typescript`
const result = await uploadFiles({
storage: {
provider: 'do-spaces',
config: {
region: 'nyc3', // nyc3, sfo3, ams3, sgp1, fra1, syd1
bucket: 'your-space',
accessKeyId: process.env.DO_SPACES_KEY,
secretAccessKey: process.env.DO_SPACES_SECRET,
basePath: 'uploads',
acl: 'public-read'
}
},
files: ['static/*/']
})
`typescript`
// Only works within GitHub Actions environment
const result = await uploadFiles({
storage: {
provider: 'github',
config: {
artifactName: 'build-artifacts',
retentionDays: 30,
basePath: 'dist'
}
},
files: ['dist//', 'coverage//']
})
`typescript
import { downloadFiles, quickDownload } from '@yofix/storage'
// Download multiple files
const result = await downloadFiles({
storage: {
provider: 's3',
config: { region: 'us-east-1', bucket: 'my-bucket' }
},
files: ['config.json', 'data/users.json'],
verbose: true
})
// Access downloaded content
result.files.forEach(file => {
console.log(Downloaded: ${file.remotePath})Size: ${file.size} bytes
console.log()Content: ${file.buffer.toString()}
console.log()
})
// Quick download single file
const buffer = await quickDownload(
{ provider: 's3', config: { region: 'us-east-1', bucket: 'my-bucket' } },
{ path: 'config.json' }
)
`
By default, providers use singleton mode with connection reuse for better performance:
`typescript
import { StorageManager } from '@yofix/storage'
// Multiple uploads reuse the same S3 client
await uploadFiles({ storage: s3Config, files: batch1 }) // Creates client
await uploadFiles({ storage: s3Config, files: batch2 }) // Reuses client!
// Configure registry behavior
StorageManager.configure({
singleton: true, // Enable/disable singleton mode (default: true)
autoCleanup: true, // Auto-cleanup idle connections
idleTTL: 5 60 1000, // 5 minutes idle timeout
cleanupInterval: 60 * 1000 // Check every minute
})
// Disable singleton mode (original per-operation behavior)
StorageManager.configure({ singleton: false })
// Manual cleanup
await StorageManager.cleanup() // Cleanup all providers
await StorageManager.cleanupProvider(s3Config) // Cleanup specific provider
await StorageManager.cleanupIdle(60000) // Cleanup idle > 1 minute
// Get connection statistics
const stats = StorageManager.getStats()
// { totalProviders: 2, activeProviders: 1, idleProviders: 1, providers: [...] }
`
`typescript
import { StorageManager } from '@yofix/storage'
const manager = new StorageManager({
storage: { provider: 'local', config: { directory: './uploads' } },
files: ['*.json'],
verbose: true
})
// Upload
const uploadResult = await manager.upload()
// Download
const downloadResult = await manager.download(['config.json', 'data.json'])
// File operations
const exists = await manager.fileExists('package.json')
const files = await manager.listFiles('uploads/')
const objects = await manager.listObjects({ prefix: 'images/', maxKeys: 100 })
const size = await manager.getFileSize('large-file.zip')
const metadata = await manager.getFileMetadata('document.pdf')
await manager.deleteFile('old-file.txt')
// Direct data upload
const path = await manager.uploadData({
path: 'data.json',
data: Buffer.from(JSON.stringify({ key: 'value' })),
contentType: 'application/json'
})
// Generate signed URLs (S3, GCS, Firebase, R2, DO Spaces)
const downloadUrl = await manager.getSignedUrl('documents/report.pdf', {
action: 'read',
expires: 3600 // 1 hour in seconds
})
const uploadUrl = await manager.getSignedUrl('uploads/new-file.pdf', {
action: 'write',
expires: new Date(Date.now() + 15 60 1000), // 15 minutes
contentType: 'application/pdf'
})
`
`typescript`
interface GitHubConfig {
artifactName?: string // Default: 'storage-artifacts'
retentionDays?: number // 1-90, Default: 90
basePath?: string
}
`typescript`
interface FirebaseConfig {
projectId?: string
credentials?: string | object // Service account JSON (supports base64)
bucket: string // Required
basePath?: string
}
`typescript`
interface S3Config {
region: string // Required
bucket: string // Required
accessKeyId?: string
secretAccessKey?: string
basePath?: string
acl?: string // 'public-read', 'private', etc.
endpoint?: string // For S3-compatible services
}
`typescript`
interface GCSConfig {
bucket: string // Required
projectId?: string
keyFilename?: string // Service account key file path
credentials?: string | object
accessToken?: string // OAuth access token
refreshToken?: string // OAuth refresh token
clientId?: string // Required with refreshToken
clientSecret?: string // Required with refreshToken
basePath?: string
}
`typescript`
interface R2Config {
accountId: string // Required
bucket: string // Required
accessKeyId: string // Required
secretAccessKey: string // Required
basePath?: string
publicUrl?: string // Custom domain for public buckets
}
`typescript`
interface DOSpacesConfig {
region: string // Required: nyc3, sfo3, ams3, sgp1, fra1, syd1
bucket: string // Required
accessKeyId: string // Required
secretAccessKey: string // Required
basePath?: string
acl?: string // 'public-read', 'private', etc.
}
`typescript`
interface LocalConfig {
directory: string // Required
createIfNotExists?: boolean // Default: true
basePath?: string
}
Generate pre-signed URLs for temporary access to files without exposing credentials. Useful for:
- Direct browser uploads/downloads
- Sharing temporary access links
- Secure file transfers
`typescript
import { StorageManager } from '@yofix/storage'
const manager = new StorageManager({
storage: { provider: 's3', config: { region: 'us-east-1', bucket: 'my-bucket' } }
})
// Generate a download URL (expires in 1 hour)
const downloadUrl = await manager.getSignedUrl('reports/annual.pdf', {
action: 'read',
expires: 3600 // seconds
})
// Generate an upload URL (expires in 15 minutes)
const uploadUrl = await manager.getSignedUrl('uploads/user-file.pdf', {
action: 'write',
expires: new Date(Date.now() + 15 60 1000),
contentType: 'application/pdf' // Required for write on some providers
})
// Use with fetch for direct upload
await fetch(uploadUrl, {
method: 'PUT',
headers: { 'Content-Type': 'application/pdf' },
body: fileBuffer
})
`
Supported Providers: S3, GCS, Firebase, R2, DO Spaces
Options:
| Option | Type | Description |
|--------|------|-------------|
| action | 'read' \| 'write' | Download or upload access |expires
| | Date \| number | Expiration as Date or seconds from now |contentType
| | string | MIME type (required for write on some providers) |
| Feature | GitHub | Firebase | S3 | GCS | R2 | DO Spaces | Local |
|---------|:------:|:--------:|:--:|:---:|:--:|:---------:|:-----:|
| Upload File | Yes | Yes | Yes | Yes | Yes | Yes | Yes |
| Upload Data | - | Yes | Yes | Yes | Yes | Yes | Yes |
| Download | - | Yes | Yes | Yes | Yes | Yes | Yes |
| File Exists | - | Yes | Yes | Yes | Yes | Yes | Yes |
| Delete File | - | Yes | Yes | Yes | Yes | Yes | Yes |
| List Files | - | Yes | Yes | Yes | Yes | Yes | Yes |
| Get Metadata | - | Yes | Yes | Yes | Yes | Yes | Yes |
| OAuth Support | - | - | - | Yes | - | - | - |
| Multipart Upload | - | - | Yes | - | Yes | Yes | - |
| Signed URLs | - | Yes | Yes | Yes | Yes | Yes | - |
| CDN Support | - | - | - | - | - | Yes | - |
| Base Path | Yes | Yes | Yes | Yes | Yes | Yes | Yes |
`yaml
name: Upload Artifacts
on:
push:
branches: [main]
jobs:
upload:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Build
run: npm run build
- name: Upload to GitHub Artifacts
uses: yofix/storage@main
with:
provider: github
files: 'dist//,coverage//'
artifact-name: 'build-artifacts'
retention-days: 30
verbose: true
- name: Upload to S3
uses: yofix/storage@main
with:
provider: s3
files: 'dist/*/'
s3-region: us-east-1
s3-bucket: my-bucket
s3-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
s3-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
s3-base-path: builds/${{ github.sha }}
s3-acl: public-read
verbose: true
`
Full TypeScript support with comprehensive type definitions:
`typescript
import type {
// Core
StorageProvider,
StorageOptions,
StorageResult,
StorageError,
ProviderConfig,
// Files
FileToUpload,
DataToUpload,
UploadedFile,
UploadProgress,
FileToDownload,
DownloadedFile,
DownloadOptions,
DownloadResult,
// Metadata
FileMetadata,
StorageObject,
ListOptions,
SignedUrlOptions,
// Provider configs
GitHubConfig,
FirebaseConfig,
S3Config,
GCSConfig,
R2Config,
DOSpacesConfig,
LocalConfig,
// Provider interface
IStorageProvider,
// Registry
RegistryOptions
} from '@yofix/storage'
`
`typescript
const result = await uploadFiles({...})
if (!result.success) {
result.errors?.forEach(error => {
console.error([${error.code}] ${error.message})File: ${error.file}
if (error.file) {
console.error()`
}
})
}
`bashAWS S3
export AWS_ACCESS_KEY_ID=your-access-key
export AWS_SECRET_ACCESS_KEY=your-secret-key
MIT
Contributions are welcome! Please open an issue or submit a pull request on GitHub.