@flowblade/source-duckdb


Duckdb datasource adapter based on @duckdb/node-api

Install

``bash
yarn add @flowblade/source-duckdb @duckdb/node-api
`

$3

`typescript
import { DuckdbDatasource, sql } from "@flowblade/source-duckdb";

// See setup below
import { ds } from "./config.ts";

const params = {
min: 10,
max: 99,
name: "test",
createdAt: new Date().toISOString(),
};

type Row = { id: number; name: "test"; createdAt: Date };

const rawSql = sql

WITH products(productId, createdAt)
AS MATERIALIZED (
FROM RANGE(1,100) SELECT
range::INT,
TIMESTAMPTZ '2025-01-01 12:30:00.123456789+01:00'
)

SELECT productId,
${params.name} as name,
createdAt

FROM products
WHERE productId BETWEEN ${params.min}::INTEGER AND ${params.max}::INTEGER
AND createdAt < ${params.createdAt}::TIMESTAMPTZ
;

const result = await ds.query(rawSql);

const { data, meta, error } = result;

if (data) {
// Typed as Row[]
console.log(data);
}
if (error) {
// Typed as QError
console.log(error);
}

// Optionally: map over the data to transform it
const { data: mappedData } = result.map((row) => {
return {
id: row.productId,
key: key-${row.productId},
};
});
`

$3

`typescript
import os from "node:os";

import { type DuckDBConnection, DuckDBInstance } from "@duckdb/node-api";
import { DuckdbDatasource } from "@flowblade/source-duckdb";

// Create a connection to a DuckDB instance
const createConnection = async (maxThreads = 4): Promise => {
const availableThreads = os.availableParallelism();
const maxParallelism = Math.min(maxThreads, availableThreads - 1);
const threads = availableThreads > 1 ? maxParallelism : undefined;

const instance = await DuckDBInstance.create(":memory:", {
// Choose between READ_ONLY or READ_WRITE
// Note that in READ_WRITE mode concurrency is limited to 1
// See: https://duckdb.org/docs/connect/concurrency.html
access_mode: "READ_WRITE",
max_memory: "64MB",
// Using more threads may require additional memory
...(threads ? { threads: threads.toString(10) } : {}),
});
return await instance.connect();
};

const duckdb = await createConnection();

// Create a duckdb datasource
export const ds = new DuckdbDatasource({ connection: duckdb });
``

Compatibility

| Level | CI | Description |
| ------------ | --- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Node | ✅ | CI for 20.x, 22.x & 24.x |
| Cloudflare | ✅ | Ensured with @cloudflare/vitest-pool-workers (see wrangler.toml |
| Browserslist | ✅ | > 95% on 01/2025. Chrome 96+, Firefox 90+, Edge 19+, ios 15+, Safari 15+ and Opera 77+ |
| Typescript | ✅ | TS 5.0 + / are-the-type-wrong checks on CI. |
| ES2022 | ✅ | Dist files checked with es-check |
| Performance | ✅ | Monitored with codspeed.io |

Contributors

Contributions are welcome. Have a look to the CONTRIBUTING document.

Sponsors

Sponsor>), coffee>),
or star – All is spent for quality time with loved ones. Thanks ! 🙏❤️

$3

JetBrains	Embie.be

License

MIT © Sébastien Vanvelthem and contributors.