SQLite FTS5 full-text search indexing for Outfitter
npm install @outfitter/indexSQLite FTS5 full-text search indexing with WAL mode and Result-based error handling.
``bash`
bun add @outfitter/index
`typescript
import { createIndex } from "@outfitter/index";
// Create an index
const index = createIndex({ path: "./data/search.db" });
// Add documents
await index.add({
id: "doc-1",
content: "Hello world, this is searchable content",
metadata: { title: "Greeting", tags: ["hello", "world"] },
});
// Search with FTS5 syntax
const results = await index.search({ query: "hello" });
if (results.isOk()) {
for (const result of results.value) {
console.log(result.id, result.score, result.highlights);
}
}
// Cleanup
index.close();
`
- FTS5 Full-Text Search — BM25 ranking with snippet highlights
- WAL Mode — Better concurrency for read-heavy workloads
- Typed Metadata — Generic type parameter for document metadata
- Result-Based API — All operations return Result
- Tokenizer Options — unicode61, porter (stemming), or trigram
- Batch Operations — Efficient bulk document insertion
- Version Migration — Built-in schema migration support
Creates an FTS5 full-text search index.
`typescript
interface IndexOptions {
path: string; // Path to SQLite database file
tableName?: string; // FTS5 table name (default: "documents")
tokenizer?: TokenizerType; // Tokenizer (default: "unicode61")
tool?: string; // Tool identifier for metadata
toolVersion?: string; // Tool version for metadata
migrations?: IndexMigrationRegistry; // Optional migration registry
}
const index = createIndex
path: "./data/index.db",
tableName: "notes_fts",
tokenizer: "porter",
});
`
| Tokenizer | Use Case |
|-----------|----------|
| unicode61 | Default, Unicode-aware word tokenization |porter
| | English text with stemming (finds "running" when searching "run") |trigram
| | Substring matching, typo tolerance |
`typescript
interface Index
// Add single document (replaces if ID exists)
add(doc: IndexDocument): Promise
// Add multiple documents in a transaction
addMany(docs: IndexDocument[]): Promise
// Search with FTS5 query syntax
search(query: SearchQuery): Promise
// Remove document by ID
remove(id: string): Promise
// Clear all documents
clear(): Promise
// Close database connection
close(): void;
}
`
`typescript
interface IndexDocument {
id: string; // Unique document ID
content: string; // Searchable text
metadata?: Record
}
await index.add({
id: "note-123",
content: "Meeting notes from standup",
metadata: {
title: "Standup Notes",
date: "2024-01-15",
tags: ["meeting", "standup"],
},
});
`
`typescript
interface SearchQuery {
query: string; // FTS5 query string
limit?: number; // Max results (default: 25)
offset?: number; // Skip results for pagination (default: 0)
}
// Simple search
const results = await index.search({ query: "typescript" });
// Phrase search with pagination
const paged = await index.search({
query: '"error handling"',
limit: 10,
offset: 20,
});
`
FTS5 supports powerful query syntax:
| Syntax | Example | Description |
|--------|---------|-------------|
| Terms | typescript bun | Match all terms (implicit AND) |"error handling"
| Phrase | | Exact phrase match |ts OR typescript
| OR | | Match either term |typescript NOT javascript
| NOT | | Exclude term |type*
| Prefix | | Prefix matching |(react OR vue) AND typescript
| Grouping | | Complex queries |
`typescript
interface SearchResult
id: string; // Document ID
content: string; // Full document content
score: number; // BM25 relevance (negative; closer to 0 = better match)
metadata?: T; // Document metadata
highlights?: string[]; // Matching snippets with tags
}
const results = await index.search({ query: "hello world" });
if (results.isOk()) {
for (const result of results.value) {
console.log(${result.id}: ${result.highlights?.[0]});`
// "doc-1: Hello world, this is..."
}
}
For bulk indexing, use addMany for transactional efficiency:
`typescript
const documents = [
{ id: "1", content: "First document" },
{ id: "2", content: "Second document" },
{ id: "3", content: "Third document" },
];
const result = await index.addMany(documents);
if (result.isErr()) {
// Transaction rolled back, no documents added
console.error(result.error.message);
}
`
Indexes track their schema version. Provide a migration registry for upgrades:
`typescript
import { createIndex, createMigrationRegistry } from "@outfitter/index";
const migrations = createMigrationRegistry();
migrations.register(1, 2, (ctx) => {
ctx.db.run("ALTER TABLE documents ADD COLUMN category TEXT");
return Result.ok(undefined);
});
const index = createIndex({
path: "./data/index.db",
migrations,
});
`
`typescript
interface IndexMigrationRegistry {
register(
fromVersion: number,
toVersion: number,
migrate: (ctx: IndexMigrationContext) => Result
): void;
migrate(
ctx: IndexMigrationContext,
fromVersion: number,
toVersion: number
): Result
}
interface IndexMigrationContext {
db: Database; // bun:sqlite Database instance
}
`
Indexes store metadata for tracking provenance:
`typescript`
interface IndexMetadata {
version: number; // Schema version
created: string; // ISO timestamp
tool: string; // Creating tool identifier
toolVersion: string; // Creating tool version
}
The current index format version is exported for compatibility checks:
`typescript
import { INDEX_VERSION } from "@outfitter/index";
console.log(Using index format version ${INDEX_VERSION});`
All operations return Result:
`typescript
const result = await index.add(doc);
if (result.isErr()) {
console.error("Failed to add document:", result.error.message);
// result.error.cause contains the underlying error
}
`
Common error scenarios:
- Index closed after close() called
- Invalid table name or tokenizer
- SQLite errors (disk full, permissions)
- Version mismatch without migrations
1. Use WAL mode — Enabled by default for better read concurrency
2. Batch inserts — Use addMany for bulk operationsporter
3. Choose tokenizer wisely — for English, unicode61 for general useclose()` to release resources
4. Limit results — Use pagination for large result sets
5. Close when done — Call
- @outfitter/contracts — Result types and StorageError
- @outfitter/file-ops — Path utilities and workspace detection