Serialization codecs for Procwire binary protocol.
npm install @procwire/codecsSerialization codecs for Procwire binary protocol.
- rawCodec - Buffer passthrough (zero serialization overhead)
- rawChunksCodec - Zero-copy Buffer[] for large payloads
- msgpackCodec - Compact binary with Date/Buffer extension types
- arrowCodec - Columnar data for ML embeddings and analytics
| Codec | Input Type | Output Type | Zero-Copy | Use Case |
| ---------------- | -------------- | ----------- | ---------- | ----------------------------- |
| rawCodec | Buffer | Buffer | No | Pre-serialized binary data |
| rawChunksCodec | Buffer[] | Buffer[] | Yes | Large files, streaming |
| msgpackCodec | object | object | No | Small/medium objects, configs |
| arrowCodec | Table/object | Table | Yes (read) | ML embeddings, DB results |
``bash`
npm install @procwire/codecs
Requirements: Node.js >= 22
Peer dependencies: @procwire/protocol
`typescript
import { rawCodec, rawChunksCodec, msgpackCodec, arrowCodec } from "@procwire/codecs";
// For small/medium payloads - returns Buffer
const data = rawCodec.deserialize(payload);
// For large payloads - returns Buffer[] (ZERO-COPY!)
const chunks = rawChunksCodec.deserializeChunks(payloadChunks);
// For objects with Date/Buffer support
const obj = msgpackCodec.deserialize(payload);
// For columnar data (ML embeddings, query results)
const table = arrowCodec.deserialize(payload);
`
All codecs implement the Codec interface:
`typescript`
interface Codec
serialize(data: TInput): Buffer;
deserialize(buffer: Buffer): TOutput;
deserializeChunks?(chunks: readonly Buffer[]): TOutput; // Optional zero-copy
readonly name: string;
}
Pass-through codec for pre-serialized binary data.
`typescript
import { rawCodec } from "@procwire/codecs";
const buffer = rawCodec.serialize(myBuffer); // Returns same buffer
const data = rawCodec.deserialize(receivedBuffer); // Returns same buffer
`
When to use: Binary files, images, audio, pre-serialized data.
Zero-copy codec that preserves buffer chunks without merging.
`typescript
import { rawChunksCodec } from "@procwire/codecs";
const chunks = rawChunksCodec.deserializeChunks(payloadChunks);
// chunks is Buffer[] - same references, no copy!
`
When to use: Large file transfers, streaming where you want to avoid memory copies.
MessagePack codec with extension types for Buffer and Date.
`typescript
import { msgpackCodec } from "@procwire/codecs";
const data = {
name: "test",
buffer: Buffer.from("hello"),
date: new Date(),
};
const serialized = msgpackCodec.serialize(data);
const deserialized = msgpackCodec.deserialize(serialized);
// deserialized.buffer is Buffer
// deserialized.date is Date
`
Extension types:
- Type 1: Buffer - preserved as Buffer on deserializationDate
- Type 2: - preserved as Date on deserialization
When to use: JavaScript objects, configs, progress events, error responses.
Apache Arrow IPC codec for columnar data.
`typescript
import { arrowCodec } from "@procwire/codecs";
import { tableFromArrays } from "apache-arrow";
// From simple object
const data = arrowCodec.serialize({
embeddings: new Float32Array([0.1, 0.2, 0.3]),
ids: [1, 2, 3],
});
// From Arrow Table
const table = tableFromArrays({
column1: [1, 2, 3],
column2: ["a", "b", "c"],
});
const serialized = arrowCodec.serialize(table);
// Deserialize always returns Table (zero-copy read)
const result = arrowCodec.deserialize(serialized);
`
When to use: ML embeddings, database query results, batch data, numeric arrays.
Helper that automatically chooses the most efficient deserialization path.
`typescript
import { codecDeserialize } from "@procwire/codecs";
// Prefers deserializeChunks if available (zero-copy)
// Falls back to deserialize(Buffer.concat(chunks))
const data = codecDeserialize(myCodec, frame);
`
Implement the Codec interface for custom serialization:
`typescript
import type { Codec } from "@procwire/codecs";
const jsonCodec: Codec
name: "json",
serialize: (data) => Buffer.from(JSON.stringify(data)),
deserialize: (buf) => JSON.parse(buf.toString()),
};
// With zero-copy support
const customCodec: Codec
name: "custom",
serialize: (data) => / ... /,
deserialize: (buf) => / ... /,
deserializeChunks: (chunks) => / ... /, // Optional
};
`
`typescript``
type RawCodecType = Codec
type RawChunksCodecType = Codec
type ObjectCodecType
MIT