@lumeweb/pinner

A TypeScript library for uploading files to IPFS and managing pinning operations with support for multiple upload protocols, custom storage backends, and flexible configuration.

Features

- Multiple Upload Methods: TUS resumable uploads, XHR uploads, and direct CAR file uploads
- Directory Support: Upload multiple files as a directory to IPFS
- Pin Management: Add, list, remove, and check status of pinned content
- Custom Blockstore: Flexible storage backend using unstorage (IndexedDB, filesystem, Redis, etc.)
- Adapters: Built-in Pinata adapter with extensible adapter pattern
- Encoders: Support for CSV, JSON, text, base64, and URL encoding
- Progress Tracking: Real-time upload progress and operation polling
- Cross-Platform: Works in both browser and Node.js environments
- TypeScript: Fully typed with comprehensive type definitions

Installation

``bash
pnpm add @lumeweb/pinner
`

Quick Start

`typescript
import { Pinner } from "@lumeweb/pinner";

// Initialize with JWT token
const pinner = new Pinner({
jwt: "your-jwt-token",
endpoint: "https://ipfs.pinner.xyz",
gateway: "https://dweb.link"
});

// Upload a file
const file = new File(["Hello, IPFS!"], "hello.txt", { type: "text/plain" });
const operation = await pinner.upload(file);

// Wait for completion
const result = await operation.result;
console.log("CID:", result.cid);
console.log("URL:", result.url);

// List pins
const pins = await pinner.listPins();
console.log("Pinned content:", pins);
`

Configuration

The Pinner class accepts a PinnerConfig object:

`typescript
interface PinnerConfig {
// Required
jwt: string;

// Optional
endpoint?: string; // Default: "https://ipfs.pinner.xyz"
gateway?: string; // Default: "https://dweb.link"
allowedFileTypes?: string[]; // MIME types allowed for upload
fetch?: typeof fetch; // Custom fetch implementation
datastore?: Datastore; // Custom datastore for Helia
storage?: Storage; // Custom unstorage instance
datastoreName?: string; // Base name for storage (default: "pinner-helia-data")
}
`

Upload Methods

$3

`typescript
// Upload with default options
const operation = await pinner.upload(file);

// Upload with custom options
const operation = await pinner.upload(file, {
metadata: { name: "My File" },
timeout: 30000
});

// Wait for result
const result = await operation.result;
`

$3

Convenience method for simple use cases:

`typescript
const result = await pinner.uploadAndWait(file);
console.log("CID:", result.cid);
`

$3

`typescript
const files = [
new File(["content1"], "file1.txt"),
new File(["content2"], "file2.txt")
];

const operation = await pinner.uploadDirectory(files);
const result = await operation.result;
`

$3

Upload pre-generated CAR files without preprocessing:

`typescript
const carFile = new File([carData], "content.car", { type: "application/vnd.ipld.car" });
const operation = await pinner.uploadCar(carFile);
const result = await operation.result;
`

$3

Use the builder API for more control:

`typescript
// Upload JSON
const operation = await pinner.upload.json({ foo: "bar" });

// Upload text
const operation = await pinner.upload.text("Hello, world!");

// Upload CSV
const operation = await pinner.upload.csv([
{ name: "Alice", age: 30 },
{ name: "Bob", age: 25 }
]);
`

Progress Tracking

Upload operations return an UploadOperation with progress tracking:

`typescript
const operation = await pinner.upload(file);

// Listen to progress events
operation.on("progress", (progress) => {
console.log(Progress: ${progress.progress}%);
console.log(Speed: ${progress.speed} bytes/sec);
});

// Listen to completion
operation.on("complete", (result) => {
console.log("Upload complete:", result.cid);
});

// Listen to errors
operation.on("error", (error) => {
console.error("Upload failed:", error);
});

// Or await the result directly
const result = await operation.result;
`

Pin Management

$3

`typescript
import { CID } from "multiformats/cid";

// Pin existing content
const cid = CID.parse("Qm...");
await pinner.pinByHash(cid);

// With options
await pinner.pinByHash(cid, {
name: "My Pin",
metadata: { key: "value" }
});
`

$3

`typescript
// List all pins
const pins = await pinner.listPins();

// List with filters
const pins = await pinner.listPins({
status: "pinned",
limit: 10,
offset: 0
});
`

$3

`typescript
const pin = await pinner.getPinStatus(cid);
console.log("Pin status:", pin.status);
console.log("Created:", pin.created);
`

$3

`typescript
const isPinned = await pinner.isPinned(cid);
if (isPinned) {
console.log("Content is pinned");
}
`

$3

`typescript
await pinner.setPinMetadata(cid, {
name: "Updated Name",
description: "Updated description"
});
`

$3

`typescript
await pinner.unpin(cid);
`

Operation Polling

Wait for operations to complete with custom polling options:

`typescript
// Poll with default settings
const result = await pinner.waitForOperation(operationId);

// Poll with custom options
const result = await pinner.waitForOperation(operationId, {
interval: 1000, // Check every 1 second
timeout: 60000, // Timeout after 60 seconds
settledStates: ["completed", "failed"]
});
`

Custom Blockstore

Configure a custom storage backend using unstorage:

`typescript
import { Pinner, createBlockstore, setDriverFactory } from "@lumeweb/pinner";
import { createStorage } from "unstorage";
import redisDriver from "unstorage/drivers/redis";

// Create Redis storage
const storage = createStorage({
driver: redisDriver({
host: "localhost",
port: 6379,
base: "pinner:"
})
});

// Initialize with custom storage
const pinner = new Pinner({
jwt: "your-token",
storage
});
`

$3

`typescript
import { createBlockstore } from "@lumeweb/pinner";

// Auto-configure (browser: IndexedDB, Node.js: filesystem)
const blockstore = createBlockstore();

// Custom driver
const blockstore = createBlockstore({
driver: localStorageDriver({ base: "my-app:" })
});

// Pre-configured storage
const blockstore = createBlockstore({
storage: myStorageInstance
});

// Disable auto-configuration (uses memory)
const blockstore = createBlockstore({
autoConfigure: false
});
`

See blockstore/README.md for detailed blockstore documentation.

Adapters

$3

The Pinata adapters provide Pinata SDK API compatibility for the Pinner client, allowing applications written for the Pinata SDK to work with Lume's IPFS pinning infrastructure with minimal code changes.

Attribution: These adapters include TypeScript type definitions and API interfaces adapted from the Pinata SDK for compatibility purposes. The original Pinata SDK is available at:
- Pinata SDK 2.x: https://github.com/PinataCloud/pinata/commit/cdc0c06116aaadaf7c4b287a2673cd23b6ba1125
- Pinata SDK 1.x: https://github.com/PinataCloud/pinata/commit/c141177ff3036e46fa7b95fcc68c159b58817836

The adapters provide Pinata SDK API compatibility but route all operations through Lume's IPFS pinning infrastructure. They do NOT use Pinata's servers or services.

#### Available Adapters

- V2 Adapter (pinataAdapter): Compatible with Pinata SDK 2.x API (recommended, latest)
- Legacy Adapter (pinataLegacyAdapter): Compatible with Pinata SDK 1.x API

See adapters/README.md for comprehensive documentation including migration guides, feature support tables, and detailed examples.

#### Setup

`typescript
import { Pinner, pinataAdapter, pinataLegacyAdapter } from "@lumeweb/pinner";

// Initialize Pinner
const pinner = new Pinner({
jwt: "your-jwt-token",
endpoint: "https://your-pinning-service-endpoint.com"
});

// Create Pinata V2 adapter (recommended)
const pinata = pinataAdapter(pinner);

// Or use the legacy adapter
const pinataLegacy = pinataLegacyAdapter(pinner);
`

#### Upload Methods

The Pinata adapter provides multiple upload methods with a fluent builder pattern:

##### Upload a File

`typescript
// Simple file upload
const result = await pinata.upload.public.file(file).execute();
console.log("CID:", result.IpfsHash);
console.log("Size:", result.PinSize);

// Upload with metadata
const result = await pinata.upload.public.file(file)
.name("My File")
.keyvalues({ key: "value" })
.execute();
`

##### Upload Multiple Files (Directory)

`typescript
const files = [
new File(["content1"], "file1.txt"),
new File(["content2"], "file2.txt")
];

const result = await pinata.upload.public.fileArray(files)
.name("My Directory")
.keyvalues({ type: "directory" })
.execute();
`

##### Upload JSON Data

`typescript
const data = { foo: "bar", number: 42 };

const result = await pinata.upload.public.json(data)
.name("data.json")
.keyvalues({ format: "json" })
.execute();
`

##### Upload Base64 String

`typescript
const base64String = "SGVsbG8sIHdvcmxkIQ==";

const result = await pinata.upload.public.base64(base64String)
.name("base64-file.txt")
.execute();
`

##### Upload from URL

`typescript
const result = await pinata.upload.public.url("https://example.com/data.json")
.name("downloaded-file.json")
.execute();
`

##### Pin by CID

`typescript
// Pin existing content
await pinata.upload.public.cid("Qm...").execute();

// Pin with metadata
await pinata.upload.public.cid("Qm...")
.name("Existing Content")
.keyvalues({ source: "external" })
.execute();
`

#### Pin Management

##### Pin by Hash

`typescript
await pinata.pinByHash("Qm...", {
name: "My Pin",
keyvalues: { key: "value" }
});
`

##### Unpin Content

`typescript
await pinata.unpin("Qm...");
`

##### Get Pin Status

`typescript
const pin = await pinata.getPinStatus("Qm...");
console.log("Pin ID:", pin.id);
console.log("IPFS Hash:", pin.ipfsPinHash);
console.log("Size:", pin.size);
console.log("Date Pinned:", pin.datePinned);
console.log("Metadata:", pin.metadata);
`

##### Check if Pinned

`typescript
const isPinned = await pinata.isPinned("Qm...");
if (isPinned) {
console.log("Content is pinned");
}
`

##### Update Pin Metadata

`typescript
await pinata.setPinMetadata("Qm...", {
name: "Updated Name",
key: "value"
});
`

#### Files Management

##### List Files

`typescript
// List all files
const files = await pinata.files.public.list().execute();

// List with pagination
const files = await pinata.files.public.list()
.limit(10)
.pageToken("next-page-token")
.execute();

files.forEach(file => {
console.log("ID:", file.id);
console.log("CID:", file.cid);
console.log("Size:", file.size);
console.log("Name:", file.name);
console.log("Created:", file.createdAt);
});
`

##### Get File by ID

`typescript
const file = await pinata.files.public.get("Qm...");
console.log("File details:", file);
`

#### Pinata SDK Compatibility

The adapter follows the Pinata SDK's API conventions, making migration straightforward:

Pinata SDK:
`typescript
import pinataSDK from '@pinata/sdk';
const pinata = pinataSDK('apiKey', 'apiSecret');

const result = await pinata.pinFileToIPFS(file, {
pinataMetadata: { name: 'My File' },
pinataOptions: { cidVersion: 1 }
});
`

Pinner with Pinata Adapter:
`typescript
import { Pinner, pinataAdapter } from '@lumeweb/pinner';

const pinner = new Pinner({ jwt: 'your-jwt-token' });
const pinata = pinataAdapter(pinner);

const result = await pinata.upload.public.file(file)
.name('My File')
.execute();
`

#### Result Format

Upload operations return a PinataUploadResult:

`typescript
interface PinataUploadResult {
IpfsHash: string; // IPFS CID
PinSize: number; // Size in bytes
Timestamp: string; // ISO timestamp
isDuplicate: boolean; // Whether content was already pinned
}
`

#### Error Handling

The adapter provides specific errors:

`typescript
import { PinataAdapterError } from "@lumeweb/pinner/adapters/pinata";

try {
const result = await pinata.upload.file(file).execute();
} catch (error) {
if (error instanceof PinataAdapterError) {
switch (error.code) {
case "UPLOAD_FAILED":
console.error("Upload failed:", error.message);
break;
case "EMPTY_FILE_ARRAY":
console.error("Cannot upload empty file array");
break;
case "INVALID_CID":
console.error("Invalid CID:", error.message);
break;
}
}
}
`

Encoders

The library provides several encoders for different data formats:

`typescript
// JSON encoder
const operation = await pinner.upload.json({ data: "value" });

// CSV encoder
const operation = await pinner.upload.csv([
{ column1: "value1", column2: "value2" }
]);

// Text encoder
const operation = await pinner.upload.text("Plain text content");

// Base64 encoder
const operation = await pinner.upload.base64("SGVsbG8sIHdvcmxkIQ==");

// URL encoder
const operation = await pinner.upload.url("https://example.com/data");
`

Error Handling

The library provides specific error types:

`typescript
import {
PinnerError,
ConfigurationError,
AuthenticationError,
UploadError,
NetworkError,
ValidationError,
EmptyFileError,
TimeoutError,
PinError,
NotFoundError,
RateLimitError
} from "@lumeweb/pinner";

try {
const result = await pinner.uploadAndWait(file);
} catch (error) {
if (error instanceof AuthenticationError) {
console.error("Authentication failed:", error.message);
} else if (error instanceof NetworkError) {
console.error("Network error:", error.message);
} else if (error instanceof ValidationError) {
console.error("Validation error:", error.message);
} else {
console.error("Unknown error:", error);
}
}
`

$3

`typescript
import { isRetryable, isAuthenticationError } from "@lumeweb/pinner";

if (isRetryable(error)) {
// Retry the operation
}

if (isAuthenticationError(error)) {
// Re-authenticate
}
`

Stream Utilities

`typescript
import {
streamToBlob,
calculateStreamSize,
asyncGeneratorToReadableStream,
readableStreamToAsyncIterable
} from "@lumeweb/pinner";

// Convert stream to blob
const blob = await streamToBlob(stream);

// Calculate stream size
const size = await calculateStreamSize(stream);

// Convert async generator to readable stream
const readableStream = asyncGeneratorToReadableStream(asyncGenerator);

// Convert readable stream to async iterable
const asyncIterable = readableStreamToAsyncIterable(readableStream);
`

Testing

The library includes comprehensive tests:

`bash


Run all tests

pnpm test

Run tests with coverage

pnpm coverage

Run specific test project

vitest run --project node-upload-unit
vitest run --project browser-upload-integration
`

$3

- Unit Tests: Test individual components with mocks
- Integration Tests: Test real-world scenarios without mocks
- Browser Tests: Run tests in Chromium using Playwright
- Node Tests: Run tests in Node.js environment

Build

`bash


Build the library

pnpm build

Type checking

pnpm lint
`

The library outputs:
- ESM: dist/esm/
- CJS: dist/cjs/
- Type definitions: dist/esm/*.d.ts

Package Exports

`typescript
// Main exports
import { Pinner } from "@lumeweb/pinner";

// Upload types and utilities
import {
UploadManager,
UploadResult,
UploadOptions,
UploadProgress
} from "@lumeweb/pinner";

// Pin types
import type {
RemotePins,
RemotePin,
RemoteAddOptions,
RemoteLsOptions
} from "@lumeweb/pinner";

// Blockstore
import {
createBlockstore,
createDatastore,
setDriverFactory
} from "@lumeweb/pinner";

// Adapters
import { pinataAdapter, pinataLegacyAdapter } from "@lumeweb/pinner";

// Blockstore module
import { UnstorageBlockstore } from "@lumeweb/pinner/blockstore";
``

License

MIT

Contributing

Contributions are welcome! Please ensure all tests pass and follow the existing code style.