Chunkdown is a tree-based markdown text splitter to create semantically meaningful chunks for RAG applications.
Unlike traditional splitters that use simple character or regex-based methods, this library leverages markdown's hierarchical structure for optimal chunking.
Chunkdown is built around a few core ideas that guide its design:

#### Markdown as Hierarchical Tree

A properly structured markdown document forms a hierarchical tree where headings define sections containing various nodes (paragraphs, lists, tables, etc.). We parse markdown into an Abstract Syntax Tree (AST) and transform it into a hierarchical structure where sections contain their related content. This enables intelligent chunking that keeps semantically related information together.

Hierarchical Markdown Abstract Syntax Tree

#### Content Length vs. Markdown Length

Markdown uses additional characters for formatting (bold, italic, link, etc.) that increase the total character count without necessarily changing the semantic meaning. When calculating chunk size, we count actual text content rather than raw markdown characters. This ensures consistent semantic density across chunks regardless of formatting.

> [!NOTE]
> In a future version, it will be possible to opt-out of this behavior and use raw markdown length to calculate the chunk size.

For example, the following text from Wikipedia has 804 raw characters, however, what the user actually sees rendered on the screen are only 202 characters:

The llama (/ˈlɑːmə/; Spanish pronunciation: [\[ˈʎama\]](https://en.wikipedia.org/wiki/Help:IPA/Spanish "Help:IPA/Spanish") or [\[ˈʝama\]](https://en.wikipedia.org/wiki/Help:IPA/Spanish "Help:IPA/Spanish")) (Lama glama) is a domesticated South American camelid, widely used as a meat and pack animal by Andean cultures since the pre-Columbian era.

Comparison of chunk size 100: Chunkdown (left) / LangChain Markdown Splitter (right)

#### Words as Atomic Unit

Words are the smallest meaningful unit of information for embedding purposes. While tokenizers may split words further, for practical RAG applications, breaking words mid-way creates meaningless chunks. Therefore, words are treated as indivisible atoms that cannot be split.

Comparison of chunk size 1: Chunkdown (left) / LangChain Markdown Splitter (right)

#### Never Break Semantics

Semantic elements like links, images, inline code, and certain formatting elements should ideally always remain intact. Breaking a long link like structured data generation into

[Comparison of chunk size 100: Chunkdown (left" class="text-primary hover:underline" target="_blank" rel="noopener noreferrer">structured and data generation / LangChain Markdown Splitter (right)](https://chunks.zirkelc.dev/?text=QUkgU0RLIENvcmUgaGFzIHZhcmlvdXMgZnVuY3Rpb25zIGRlc2lnbmVkIGZvciBbdGV4dCBnZW5lcmF0aW9uXSguL2dlbmVyYXRpbmctdGV4dCksIFtzdHJ1Y3R1cmVkIGRhdGEgZ2VuZXJhdGlvbl0oLi9nZW5lcmF0aW5nLXN0cnVjdHVyZWQtZGF0YSksIGFuZCBbdG9vbCB1c2FnZV0oLi90b29scy1hbmQtdG9vbC1jYWxsaW5nKS4KVGhlc2UgZnVuY3Rpb25zIHRha2UgYSBzdGFuZGFyZGl6ZWQgYXBwcm9hY2ggdG8gc2V0dGluZyB1cCBbcHJvbXB0c10oLi9wcm9tcHRzKSBhbmQgW3NldHRpbmdzXSguL3NldHRpbmdzKSwgbWFraW5nIGl0IGVhc2llciB0byB3b3JrIHdpdGggZGlmZmVyZW50IG1vZGVscy4%3D&chunkSize=100§ionOrder=input%2Cchunks%2Cpanel-1764331988127&maxOverflow=1&comparison=%255B%257B%2522id%2522%253A%2522panel-1764331988127%2522%252C%2522library%2522%253A%2522langchain%2522%252C%2522chunkdownAlgorithm%2522%253A%2522markdown%2522%252C%2522langchainAlgorithm%2522%253A%2522markdown%2522%252C%2522mastraAlgorithm%2522%253A%2522recursive%2522%252C%2522chunkSize%2522%253A100%252C%2522maxOverflowRatio%2522%253A1.5%257D%255D)

#### Allow Controlled Overflow

Preserving a complete semantic unit like a section, paragraph, sentence, etc., is often more important than adhering to a strict chunk size. The splitter allows a controlled overflow (via maxOverflowRatio) of the chunk size if it avoids splitting a complete unit, e.g. a list item.

Comparison of chunk size 200 with 1.5x overflow ratio: Chunkdown (left) / LangChain Markdown Splitter (right)

$3

``bash
npm install chunkdown
#
pnpm add chunkdown
#
bun add chunkdown
`

Usage

`typescript
import { chunkdown } from 'chunkdown';

const splitter = chunkdown({
chunkSize: 500, // Target chunk size based on content length
maxOverflowRatio: 1.5 // Allow up to 50% overflow
});

const text =


AI SDK Core

Large Language Models (LLMs) are advanced programs that can understand, create, and engage with human language on a large scale.
They are trained on vast amounts of written material to recognize patterns in language and predict what might come next in a given piece of text.

AI SDK Core simplifies working with LLMs by offering a standardized way of integrating them into your app - so you can focus on building great AI applications for your users, not waste time on technical details.

For example, here's how you can generate text with various models using the AI SDK:

AI SDK Core Functions

AI SDK Core has various functions designed for text generation, structured data generation, and tool usage.
These functions take a standardized approach to setting up prompts and settings, making it easier to work with different models.

- generateText: Generates text and tool calls.
This function is ideal for non-interactive use cases such as automation tasks where you need to write text (e.g. drafting email or summarizing web pages) and for agents that use tools.
- streamText: Stream text and tool calls.
You can use the streamText function for interactive use cases such as chat bots and content streaming.
- generateObject: Generates a typed, structured object that matches a Zod schema.
You can use this function to force the language model to return structured data, e.g. for information extraction, synthetic data generation, or classification tasks.
- streamObject: Stream a structured object that matches a Zod schema.
You can use this function to stream generated UIs.

API Reference

Please check out the AI SDK Core API Reference for more details on each function.
;

const { chunks } = splitter.split(text);
// Each chunk contains:
// - text: the chunk content as markdown string
// - breadcrumbs: array of ancestor headings for context
`

#### Breadcrumbs

Each chunk includes breadcrumbs, an array of ancestor headings that provide hierarchical context. This is useful for RAG applications to provide additional context when embedding chunks.

> [!NOTE]
> If the heading is part of the chunk itself, it is not included in the breadcrumbs.

`typescript
import { chunkdown } from 'chunkdown';

const text =
Large Language Models (LLMs) are advanced programs that understand and generate human language.

AI SDK Core

The AI SDK simplifies working with LLMs by offering a standardized API.

Text Generation

Generate text using various models.

$3

Use generateObject to get typed responses matching a schema.
;

const splitter = chunkdown({ chunkSize: 100 });
const { chunks } = splitter.split(text);

// chunks[0]:
// text: "Large Language Models (LLMs) are advanced programs that understand and generate human language.",
// breadcrumbs: [] // orphaned content before any heading

// chunks[1]:
// text: "# AI SDK Core\n\nThe AI SDK simplifies working with LLMs by offering a standardized API.",
// breadcrumbs: [] // H1 is in chunk, no ancestors

// chunks[2]:
// text: "## Text Generation\n\nGenerate text using various models.",
// breadcrumbs: [{ text: "AI SDK Core", depth: 1 }] // ancestor only

// chunks[3]:
// text: "### Structured Output\n\nUse generateObject to get typed responses matching a schema.",
// breadcrumbs: [
// { text: "AI SDK Core", depth: 1 },
// { text: "Text Generation", depth: 2 }
// ] // ancestors only (H3 is in chunk)
`

#### Links and Images

By default, links and images are never split to avoid breaking their semantic meaning.

`typescript
import { chunkdown } from 'chunkdown';

const text = Please check out the AI SDK Core API Reference for more details on each function.;

// By default, never split links and images
const splitter = chunkdown({
chunkSize: 50,
});

const { chunks } = splitter.split(text);
// chunks[0].text: "Please check out the AI SDK Core API Reference"
// chunks[1].text: "for more details on each function."

// Allow splitting links
const splitter2 = chunkdown({
chunkSize: 50,
rules: {
formatting: { split: 'allow-split' }
}
});

const { chunks: chunks2 } = splitter2.split(text);
// chunks2[0].text: "Please check out the [AI SDK Core API"
// chunks2[1].text: "Reference](/docs/reference/ai-sdk-core) for more details on each function."
`

##### Reference-Style

By default, Chunkdown normalizes reference-style links and images to inline style. This prevents issues when reference definitions end up in different chunks than their usage.

`typescript
import { chunkdown } from 'chunkdown';

const text =
Check out the [documentation][docs] and [API reference][api].

[docs]: https://example.com/docs
[api]: https://example.com/api
;

// By default, normalize to inline style
const splitter = chunkdown({
chunkSize: 100,
});

const { chunks } = splitter.split(text);
// Result:
// chunks[0].text: "Check out the documentation and API reference."

// Preserve original style
const splitter2 = chunkdown({
chunkSize: 100,
rules: {
link: { style: 'preserve' }
}
});

const { chunks: chunks2 } = splitter2.split(text);
// Result:
// chunks2[0].text: "Check out the [documentation][docs] and [API reference][api]."
// chunks2[1].text: "[docs]: https://example.com/docs"
// chunks2[2].text: "[api]: https://example.com/api"
`

#### Formatting

Unlike links and images, formatting elements like bold, italic, and ~~strikethrough~~ will be splitted if needed.

`typescript
import { chunkdown } from 'chunkdown';

const text = This is a very long bold text that contains many words and exceeds the chunk size in the middle.;

// By default, allow splitting formatting
const splitter = chunkdown({
chunkSize: 30,
});

const { chunks } = splitter.split(text);
// chunks[0].text: "This is **a very long"
// chunks[1].text: "bold text that contains many"
// chunks[2].text: "words and exceeds the"
// chunks[3].text: "chunk size** in the middle."

// Never split formatting
const splitter2 = chunkdown({
chunkSize: 30,
rules: {
formatting: { split: 'never-split' }
}
});

const { chunks: chunks2 } = splitter2.split(text);
// chunks2[0].text: "This is"
// chunks2[1].text: "a very long bold text that contains many words and exceeds the chunk size"
// chunks2[2].text: "in the middle."
`

#### Tables

When a table is split into multiple chunks, Chunkdown automatically preserves context by including the header row in each chunk. This ensures that data rows don't lose their meaning when separated from the original header.

> [!NOTE]
> The header row size is not counted when calculating chunk sizes. Only data row content is measured against the chunkSize limit.

`typescript
import { chunkdown } from 'chunkdown';

const splitter = chunkdown({
chunkSize: 100,
maxOverflowRatio: 1.0
});

const text =
| Name | Age | Country | Occupation | Email |
|----------|-----|---------|-------------------|------------------------|
| Alice | 30 | USA | Software Engineer | alice@example.com |
| Bob | 25 | UK | Designer | bob@example.com |
| Charlie | 35 | Canada | Product Manager | charlie@example.com |
| David | 40 | France | Data Scientist | david@example.com |
;

const { chunks } = splitter.split(text);
// chunks[0]:
// | Name | Age | Country | Occupation | Email |
// | - | - | - | - | - |
// | Alice | 30 | USA | Software Engineer | |
// | Bob | 25 | UK | Designer | |

// chunks[1]:
// | Name | Age | Country | Occupation | Email |
// | - | - | - | - | - |
// | Charlie | 35 | Canada | Product Manager | |
// | David | 40 | France | Data Scientist | |
`

Tables are serialized into markdown using the GFM table extension with the tablePipeAlign option set to false. That means the table delimeter row between header row and data rows will not be filled with additional dashes and whitespaces to align the columns vertically, saving many useless characters when embedding the chunks.

#### Normalization

Certain markdown elements such as formatting and lists have multiple representations. Chunkdown normalizes these element to ensure a uniform output regardless of input style variations.

`typescript
import { chunkdown } from 'chunkdown';

const text =
formatting:
__bold__
_italic_

---

lists:
- list item 1
- list item 2
;

const splitter = chunkdown({
chunkSize: 100,
});

const { chunks } = splitter.split(text);
// Markdown variations are normalized to:
// - __bold__ → bold
// - _italic_ → italic
// - "---" → "*" (thematic break)
// - list item 1 → list item 1 (starting with "")
`

#### Transformation

Transform functions allow you to modify or filter nodes during preprocessing. This is useful for cleaning up content before chunking, such as truncating long URLs or removing unwanted elements.

##### Truncating Long URLs

Prevent chunk bloat from excessively long URLs:

`typescript
const splitter = chunkdown({
chunkSize: 100,
rules: {
link: {
transform: (node) => {
// Truncate URLs longer than 100 characters
if (node.url.length > 50) {
return {
...node,
url: node.url.substring(0, 47) + '...'
};
}
return undefined;
}
}
}
});

const text = Check out our website.;
const { chunks } = splitter.split(text);
// chunks[0].text: "Check out our website."
`

##### Removing Data URLs

Data URLs in images (i.e. base64-encoded images) can be extremely long and create noise in chunks without meaningful content:

`typescript
import { chunkdown } from 'chunkdown';

const text =


Article

!Screenshot

Check our website for more info.

!Logo
;

const splitter = chunkdown({
chunkSize: 500,
rules: {
image: {
transform: (node) => {
// Remove images with data URLs
if (node.url.startsWith('data:')) {
return null; // Remove the entire image node
}
return undefined; // Keep regular images
}
}
}
});

const text =


Article

!Screenshot

Check our website for more info.
;
const { chunks } = splitter.split(text);
// chunks[0].text:
// # Article
//
// Check our website for more info.
`

API Reference

$3

Creates a new markdown splitter instance.

#### Returns

An object with the following methods and properties:

Methods:
- split(text: string): SplitterResult: Splits markdown text into chunks

Properties (readonly):
- chunkSize: number: The configured target chunk size
- maxOverflowRatio: number: The configured maximum overflow ratio
- maxRawSize: number | undefined: The configured maximum raw size (if set)

#### Options

##### chunkSize: number

The target content size for each chunk, counting only content characters, not raw markdown.

##### maxOverflowRatio?: number (optional)

The maximum overflow ratio for preserving semantic units:
- 1.0: strict chunk size, no overflow allowed (default)
- >1.0: allow overflow of up to chunkSize * maxOverflowRatio

##### rules?: Partial (optional)

Configure splitting behavior for specific markdown node types. Rules allow fine-grained control over when and how different markdown elements can be split during chunking.

Supported node types:
- link: Link elements text
- image: Image elements !alt
- table: Table elements
- list: List elements (ordered and unordered)
- blockquote: Blockquote elements
- code: Fenced code blocks `code`
- inlineCode: Inline code code
- formatting: Formatting elements (combines strong, emphasis, delete)
- strong: Bold text bold (overrides formatting if specified)
- emphasis: Italic text italic (overrides formatting if specified)
- delete: Strikethrough text ~~deleted~~ (overrides formatting if specified)

> [!NOTE]
> The formatting rule applies to all formatting elements (strong, emphasis, delete) unless you override them individually.

##### Split

Each node type can have a split rule:

- 'never-split' | { rule: 'never-split' }: Never split this element, regardless of size
- 'allow-split' | { rule: 'allow-split' }: Allow splitting this element if it exceeds the chunk size
- { rule: 'size-split', size: number }: Only split this element if its content size exceeds the specified size

##### Style

Links and images support an additional style property to control reference-style normalization:

- 'inline': Convert reference-style to inline style
- 'preserve': Keep original reference style

##### Transform

Each node type can have a transform function to modify or filter nodes:

`typescript
type NodeTransform = (node: T, context: TransformContext) => T | null | undefined;
`

- Return modified node: Replace the original with transformed version
- Return null: Remove the node from the tree
- Return undefined: Keep the node unchanged

The transform receives a context with parent, index, and root information. Transforms are applied during preprocessing, after reference-style normalization but before chunking.

##### Examples

`typescript
import { chunkdown, defaultNodeRules } from 'chunkdown';

// Never split links
chunkdown({
chunkSize: 500,
rules: {
link: { split: 'never-split' }
}
});

// Split lists only if they exceed 200 characters
chunkdown({
chunkSize: 500,
rules: {
list: { split: { rule: 'size-split', size: 200 } }
}
});

// Never split formatting by default, but allow splitting bold text
chunkdown({
chunkSize: 500,
rules: {
formatting: { split: 'never-split' }, // Applies to strong, emphasis, delete
strong: { split: 'allow-split' } // Override: allow splitting bold text
}
});

// Extend default rules
chunkdown({
chunkSize: 500,
rules: {
...defaultNodeRules, // Include defaults for other elements
link: { split: 'never-split' },
table: { split: { rule: 'allow-split' } },
list: { split: { rule: 'size-split', size: 150 } },
blockquote: { split: { rule: 'size-split', size: 300 } }
}
});

// Normalize links to inline-style, preserve images in reference-style
chunkdown({
chunkSize: 500,
rules: {
link: { style: 'inline' },
image: { style: 'preserve' }
}
});

// Remove data URLs and truncate long links
chunkdown({
chunkSize: 500,
rules: {
image: {
transform: (node) => {
// Remove images with data URLs
if (node.url.startsWith('data:')) {
return null;
}
return undefined;
}
},
link: {
transform: (node) => {
// Truncate long URLs
if (node.url.length > 100) {
return { ...node, url: node.url.substring(0, 100) + '...' };
}
return undefined;
}
}
}
});
`

Default rules:

By default, links and images are set to never split and normalize to inline style:

`typescript
const defaultNodeRules: NodeRules = {
link: {
split: 'never-split',
style: 'inline'
},
image: {
split: 'never-split',
style: 'inline'
},
};
`

When you provide custom rules, they override the defaults. Use the spread operator ...defaultNodeRules to explicitly include defaults if you want to override only specific elements.

`typescript
import { chunkdown, defaultNodeRules } from 'chunkdown';

chunkdown({
chunkSize: 500,
rules: {
...defaultNodeRules, // Include defaults for other elements
link: { split: 'never-split' },
table: { split: { rule: 'allow-split' } },
list: { split: { rule: 'size-split', size: 150 } },
blockquote: { split: { rule: 'size-split', size: 300 } }
}
});
`

##### maxRawSize?: number (optional)

The maximum raw size for each chunk, counting all characters including markdown formatting.

Certain markdown elements, such as links and images with long URLs, can have disproportionately long raw sizes compared to their actual content size.
For example, the following text has a content size of 21 but a raw size of 117 chars due to the long URL:

`markdown
This is a link with short text.
`

This is usually not a problem, but if a text contains a lot of such elements (e.g. scraped from a website with many links and images), the resulting chunks can become very large in raw size, even if their content size is within the allowed limits.
When the text is then embedded by a model, the large raw size could exceed the model's token limit, causing errors.
For example, OpenAI's latest embedding model text-embedding-3-large has a maximum limit of 8192 tokens, which roughly translates to about 32,000 characters (rules of thumb: 1 token ≈ 4 characters).

> [!NOTE]
> It is recommended to set this option to the upper limit of your embedding model.

The maxRawSize option acts as a safety net that enforces a hard limit on the total number of characters allowed in each chunk.
It is guaranteed that no chunk will exceed this limit, even if it means splitting semantic units that would otherwise be preserved.

Visualization

The chunk visualizer hosted at chunks.zirkelc.dev provides an interactive way to see how text is split into chunks:

Future Improvements

$3