Google Cloud Storage data store for Uploadista
Google Cloud Storage data store for Uploadista - Store files in Google Cloud.
Provides GCS-based file storage with resumable uploads, cross-platform support (Node.js and browsers), and comprehensive error handling. Supports both legacy Node.js SDK and new REST-based implementation for edge environments.
- Cross-Platform - Works in Node.js, browsers, and edge environments
- Dual Implementations - Optimized Node.js and REST-based APIs
- Resumable Uploads - Automatic resume on connection failures
- Streaming Support - Efficient memory usage with stream-based uploads
- Full Observability - Metrics, logging, and distributed tracing
- Metadata Support - Attach custom metadata to objects
- Automatic Retry - Intelligent retry logic with exponential backoff
- TypeScript - Full type safety with comprehensive JSDoc
``bash`
npm install @uploadista/data-store-gcs @google-cloud/storage @uploadista/coreor
pnpm add @uploadista/data-store-gcs @google-cloud/storage @uploadista/core
- Node.js 18+ (for Node.js implementation)
- Google Cloud project with Cloud Storage bucket
- Service account JSON key or Application Default Credentials
- TypeScript 5.0+ (optional but recommended)
`typescript
import { gcsStoreNodejs } from "@uploadista/data-store-gcs";
import { createUploadServerLayer } from "@uploadista/server";
import { memoryKvStore } from "@uploadista/kv-store-memory";
import { webSocketEventEmitter } from "@uploadista/event-emitter-websocket";
// Create GCS store (Node.js optimized)
const gcsStore = gcsStoreNodejs({
bucketName: "my-uploads",
kvStore: memoryKvStore,
keyFilename: process.env.GOOGLE_APPLICATION_CREDENTIALS,
});
// Use in upload server
const uploadLayer = createUploadServerLayer({
dataStore: gcsStore,
kvStore: memoryKvStore,
eventEmitter: webSocketEventEmitter,
});
`
`typescript
import { gcsStoreRest } from "@uploadista/data-store-gcs";
// Create GCS store (works in browsers and edge)
const gcsStore = gcsStoreRest({
bucketName: "my-uploads",
kvStore: memoryKvStore,
credentials: JSON.parse(process.env.GCS_CREDENTIALS!),
});
// Works in browsers, Cloudflare Workers, etc.
`
`bashOption 1: Service account key file (Node.js)
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account-key.json
Configuration
$3
`typescript
type GCSStoreOptions = {
// Required
bucketName: string; // GCS bucket name
kvStore: KvStore; // Metadata store // Authentication - choose one:
keyFilename?: string; // Path to service account JSON key
credentials?: object; // Service account object (for embed)
};
`$3
`typescript
import {
gcsStore, // Legacy Node.js (uses @google-cloud/storage)
gcsStoreNodejs, // New Node.js (service-based, recommended)
gcsStoreRest, // REST-based (cross-platform)
} from "@uploadista/data-store-gcs";// Recommended: Use service-based implementations
const nodeStore = gcsStoreNodejs(options); // Node.js optimized
const restStore = gcsStoreRest(options); // Cross-platform
`Google Cloud Setup Guide
$3
`bash
gsutil mb -l us-central1 gs://my-uploads-prod
`$3
`bash
Create service account
gcloud iam service-accounts create uploadista-service \
--display-name "Uploadista Service"Grant Storage admin role
gcloud projects add-iam-policy-binding my-project \
--member serviceAccount:uploadista-service@my-project.iam.gserviceaccount.com \
--role roles/storage.adminCreate and download key
gcloud iam service-accounts keys create key.json \
--iam-account uploadista-service@my-project.iam.gserviceaccount.com
`$3
`bash
Make bucket public (optional)
gsutil acl ch -u AllUsers:R gs://my-uploads-prodOr set via IAM (recommended)
gcloud storage buckets set-iam-policy gs://my-uploads-prod policy.json
`policy.json:
`json
{
"bindings": [
{
"members": ["serviceAccount:uploadista-service@my-project.iam.gserviceaccount.com"],
"role": "roles/storage.objectCreator"
}
]
}
`$3
`bash
gcloud services enable storage-api
gcloud services enable storage-component.googleapis.com
`$3
`bash
gsutil cors set cors.json gs://my-uploads-prod
`cors.json:
`json
[
{
"origin": ["https://myapp.com"],
"method": ["GET", "PUT", "POST"],
"responseHeader": ["Content-Type"],
"maxAgeSeconds": 3600
}
]
`Complete Server Example
`typescript
import Fastify from "fastify";
import WebSocket from "@fastify/websocket";
import JwT from "@fastify/jwt";
import { createFastifyUploadistaAdapter } from "@uploadista/adapters-fastify";
import { gcsStoreNodejs } from "@uploadista/data-store-gcs";
import { redisKvStore } from "@uploadista/kv-store-redis";
import { webSocketEventEmitter } from "@uploadista/event-emitter-websocket";
import { memoryEventBroadcaster } from "@uploadista/event-broadcaster-memory";const fastify = Fastify({ logger: true });
await fastify.register(JwT, { secret: process.env.JWT_SECRET! });
await fastify.register(WebSocket);
// Configure GCS (Node.js)
const gcsStore = gcsStoreNodejs({
bucketName: process.env.GCS_BUCKET!,
kvStore: redisKvStore,
keyFilename: process.env.GOOGLE_APPLICATION_CREDENTIALS,
});
// Create adapter
const adapter = await createFastifyUploadistaAdapter({
baseUrl: "uploadista",
dataStore: gcsStore,
kvStore: redisKvStore,
eventEmitter: webSocketEventEmitter,
eventBroadcaster: memoryEventBroadcaster,
flows: createFlowsEffect,
authMiddleware: async (req, reply) => {
try {
await req.jwtVerify();
return {
clientId: (req.user as any).sub,
permissions: ["upload:create"],
};
} catch {
return null;
}
},
});
// Routes
fastify.all(
/${adapter.baseUrl}/*, (req, res) => adapter.handler(req, res));
fastify.get("/ws", { websocket: true }, (socket, req) => {
adapter.websocketHandler(socket, req);
});await fastify.listen({ port: 3000 });
`Cross-Platform Example (Browsers)
`typescript
import { gcsStoreRest } from "@uploadista/data-store-gcs";// In browser or edge environment
const gcsStore = gcsStoreRest({
bucketName: "my-uploads-prod",
kvStore: memoryKvStore,
credentials: {
type: "service_account",
project_id: "my-project",
private_key_id: "...",
private_key: "...",
client_email: "...",
client_id: "...",
auth_uri: "https://accounts.google.com/o/oauth2/auth",
token_uri: "https://oauth2.googleapis.com/token",
auth_provider_x509_cert_url: "https://www.googleapis.com/oauth2/v1/certs",
},
});
// Works in browsers, Cloudflare Workers, etc.
`Performance Tuning
$3
`typescript
const gcsStore = gcsStoreNodejs({
bucketName: "my-uploads-prod",
kvStore: redisKvStore,
keyFilename: process.env.GOOGLE_APPLICATION_CREDENTIALS,
// Node.js SDK handles parallelization automatically
});
`$3
`typescript
const gcsStore = gcsStoreRest({
bucketName: "my-uploads-prod",
kvStore: redisKvStore,
credentials: require("./service-account-key.json"),
});
`Environment Configuration
$3
`env
GCS Configuration
GCS_BUCKET=my-uploads-prod
GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account-key.jsonOr embed credentials
GCS_CREDENTIALS='{"type":"service_account","project_id":"my-project",...}'
`$3
`bash
Download from Google Cloud Console
gcloud iam service-accounts keys create key.json \
--iam-account=uploadista@my-project.iam.gserviceaccount.comExport as environment variable
export GOOGLE_APPLICATION_CREDENTIALS="$(pwd)/key.json"Or use directly
const gcsStore = gcsStoreNodejs({
bucketName: "my-uploads",
kvStore,
keyFilename: "./service-account-key.json",
});
`Migration from Legacy Implementation
The store provides backward compatibility:
`typescript
import {
gcsStore, // Legacy - still works
gcsStoreNodejs, // Recommended new Node.js
} from "@uploadista/data-store-gcs";// Legacy code still works
const legacyStore = gcsStore(options);
// But use new implementation for better features
const modernStore = gcsStoreNodejs(options);
`Error Handling
Common GCS errors and solutions:
| Error | Cause | Solution |
|-------|-------|----------|
| PERMISSION_DENIED | Insufficient IAM permissions | Check service account roles |
| NOT_FOUND | Bucket doesn't exist | Verify bucket name and project |
| INVALID_ARGUMENT | Invalid bucket name | Use lowercase, 3-63 chars |
| DEADLINE_EXCEEDED | Upload timeout | Increase timeout, reduce file size |
| UNAUTHENTICATED | Missing credentials | Set GOOGLE_APPLICATION_CREDENTIALS |
Monitoring & Observability
`typescript
// Metrics automatically tracked:
// - gcs.upload.started
// - gcs.upload.progress
// - gcs.upload.completed
// - gcs.upload.failed
// - gcs.metadata.operations
`Deployment Examples
$3
`dockerfile
FROM node:20-alpine
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
COPY dist ./dist
COPY service-account-key.json ./key.jsonENV NODE_ENV=production
ENV GOOGLE_APPLICATION_CREDENTIALS=/app/key.json
EXPOSE 3000
CMD ["node", "dist/server.js"]
`$3
`bash
Create service account
gcloud iam service-accounts create uploadistaGrant permissions
gcloud projects add-iam-policy-binding PROJECT_ID \
--member serviceAccount:uploadista@PROJECT_ID.iam.gserviceaccount.com \
--role roles/storage.objectAdminDeploy
gcloud run deploy uploadista-server \
--source . \
--platform managed \
--memory 2Gi \
--service-account uploadista@PROJECT_ID.iam.gserviceaccount.com \
--set-env-vars GCS_BUCKET=my-uploads-prod
`$3
app.yaml:
`yaml
runtime: nodejs20
env: flexenv_variables:
GCS_BUCKET: "my-uploads-prod"
automatic_scaling:
min_instances: 1
max_instances: 10
`Related Packages
- @uploadista/data-store-s3 - AWS S3 storage
- @uploadista/data-store-azure - Azure Blob Storage
- @uploadista/data-store-filesystem - Local filesystem
- @uploadista/server - Core server utilities
- @uploadista/kv-store-redis - Redis KV store
- @uploadista/core - Core engine
TypeScript Support
Full TypeScript support with comprehensive types:
`typescript
import type { GCSStoreOptions } from "@uploadista/data-store-gcs";
import { gcsStoreNodejs, gcsStoreRest } from "@uploadista/data-store-gcs";
`Troubleshooting
$3
`bash
Check service account permissions
gcloud projects get-iam-policy my-project \
--flatten="bindings[].members" \
--filter="bindings.members:serviceAccount:uploadista@*"Grant Storage Object Creator role
gcloud projects add-iam-policy-binding my-project \
--member serviceAccount:uploadista@my-project.iam.gserviceaccount.com \
--role roles/storage.objectCreator
`$3
`bash
List buckets
gsutil lsCheck bucket name spelling
gsutil ls gs://my-uploads-prod/
`$3
- Use
gcsStoreNodejs in Node.js (more optimized)
- Check network bandwidth to GCS
- Verify service account has appropriate permissions
- Monitor GCS quota usage$3
`bash
Test credentials
gcloud auth activate-service-account \
--key-file=service-account-key.jsonList buckets with service account
gsutil -i serviceAccount@project.iam.gserviceaccount.com ls
``MIT