text segmentation into sentences
npm install @diplodoc/sentenizer
Rule-based NLP library for sentence segmentation with Russian language support. Splits text into sentences using hand-crafted rules that handle Russian-specific cases like abbreviations, initials, and punctuation.
- Rule-based segmentation — Hand-crafted rules optimized for Russian text
- Russian-specific handling — Correctly handles abbreviations (e.g., "и т. д.", "т. п."), initials (e.g., "И. В. Иванов"), quotations, and brackets
- Functional programming — Uses Ramda for clean, composable code
- High accuracy — Tested on various Russian text samples
- Lightweight — No external NLP libraries, fast and self-contained
``bash`
npm install @diplodoc/sentenizer
`javascript
const {sentenize} = require('@diplodoc/sentenizer');
const text = 'Он купил фрукты - яблоки, бананы, и т. д. все были очень рады угощению. Вот такой он добродушный наш родственник И. В. Иванов.';
const sentences = sentenize(text);
// sentences:
// [
// 'Он купил фрукты - яблоки, бананы, и т. д. все были очень рады угощению.',
// 'Вот такой он добродушный наш родственник И. В. Иванов.'
// ]
`
`javascript
import {sentenize} from '@diplodoc/sentenizer';
const sentences = sentenize('Первое предложение. Второе предложение!');
// ['Первое предложение.', ' Второе предложение!']
`
Splits text into sentences.
Parameters:
- text (string) — Text to segment
Returns:
- string[] — Array of segmented sentences
Type signature:
`
sentenize :: string -> string[]
The library uses a rule-based approach with two types of conditions:
Join Conditions (chunks should be merged):
- Abbreviations (e.g., "и т. д.", "т. п.")
- Initials (e.g., "И. В. Иванов")
- Quotations and brackets
- Lowercase continuation
- Delimiters and spaces
Break Conditions (chunks should be split):
- Hard breaks (double newlines)
- Uppercase start after sentence end
This package is used by @diplodoc/translation` for text segmentation during translation.
See AGENTS.md for detailed development guidelines.
MIT