A flexible text and data chunking library for vector databases and LLMs
npm install vectors-chunkersA flexible text and data chunking library for vector databases and LLMs. Similar to LangChain's RecursiveCharacterTextSplitter but with additional features for structured data and context preservation.
* bash
npm install @yourusername/vector-chunker
``typescript
import { chunk } from 'vector-chunker';
// Text chunking
const textChunks = chunk('Your long text...', {
chunkSize: 1000,
format: 'text',
splitOn: 'sentence',
preserveContext: true,
overlap: 200
});
// Structured data chunking
const dataChunks = chunk(yourDataArray, {
chunkSize: 4000,
preserveContext: true
});
``
- Text chunking with multiple splitting strategies (character, word, sentence, paragraph)
- Structured data chunking with size control
- Context preservation between chunks
- Configurable overlap
- Metadata and relationship tracking
- Full TypeScript support
See API Documentation for detailed information.
MIT