@sf-bot/rag-core

Core RAG (Retrieval-Augmented Generation) schema with pgvector support for document embeddings.

Overview

This package provides the foundational database schema for storing and managing document embeddings using PostgreSQL and pgvector. It supports:

- Multiple embedding models per collection
- Semantic chunking strategies
- JSON/CSV import and export for migrations

Schema

$3

| Table | Description |
|-------|-------------|
| rag.embedding_model | Embedding model configurations (OpenAI, Cohere, local, etc.) |
| rag.collection | Document collections with chunking configuration |
| rag.collection_model | Links collections to embedding models (many-to-many) |
| rag.document | Source documents with content and metadata |
| rag.chunk | Document segments/chunks for embedding |
| rag.embedding | Vector embeddings (pgvector) linked to chunks |

$3

The chunk_config JSONB column on rag.collection supports:

``json
{
"strategy": "semantic",
"max_tokens": 512,
"overlap_tokens": 50
}
`

Supported strategies: semantic, fixed, sentence, paragraph

Installation

`bash
pgpm deploy @sf-bot/rag-core
``

Requirements

- PostgreSQL 17+
- pgvector extension

License

SF License