oniguruma-parser 🌿

[![npm version][npm-version-src]][npm-version-href]
[![npm downloads][npm-downloads-src]][npm-downloads-href]
[![bundle][bundle-src]][bundle-href]

A TypeScript library for parsing, validating, traversing, transforming, and optimizing Oniguruma regular expressions.

> [!NOTE]
> Oniguruma is a regular expression engine written in C that's used in Ruby (via a fork named Onigmo), PHP (mb_ereg, etc.), TextMate grammars (used by VS Code, Shiki, etc.), and many other tools.

This library has been battle-tested by Oniguruma-To-ES and tm-grammars, which are used by Shiki to process tens of thousands of real-world Oniguruma regexes.

📜 Contents

- Install and use
- Convert a pattern to an AST
- Traverse and transform an AST
- Convert an AST to a pattern
- Optimize regexes
- Known differences
- Oniguruma version

🕹️ Install and use

``sh
npm install oniguruma-parser
`

`js
import {toOnigurumaAst} from 'oniguruma-parser';
`

The following modules are available in addition to the root 'oniguruma-parser' export:

- Parser module: Includes numerous functions and types for constructing and working with OnigurumaAst nodes. Also includes the parse function, wrapped by toOnigurumaAst.
- Traverser module: Traverse and transform an OnigurumaAst.
- Generator module: Convert an OnigurumaAst to pattern and flags strings.
- Optimizer module: Minify and improve the performance of Oniguruma regexes.

🌿 Convert a pattern to an AST

To parse an Oniguruma pattern (with optional flags and compile-time options) and return an AST, call toOnigurumaAst, which uses the following type definition:

`ts
function toOnigurumaAst(
pattern: string,
options?: {
flags?: string;
rules?: {
captureGroup?: boolean;
singleline?: boolean;
};
}
): OnigurumaAst;
`

For example:

`js
import {toOnigurumaAst} from 'oniguruma-parser';

const ast = toOnigurumaAst('A.*');
console.log(ast);
/* →
{ type: 'Regex',
body: [
{ type: 'Alternative',
body: [
{ type: 'Character',
value: 65,
},
{ type: 'Quantifier',
kind: 'greedy',
min: 0,
max: Infinity,
body: {
type: 'CharacterSet',
kind: 'dot',
},
},
],
},
],
flags: {
type: 'Flags',
ignoreCase: false,
dotAll: false,
extended: false,
digitIsAscii: false,
posixIsAscii: false,
spaceIsAscii: false,
wordIsAscii: false,
textSegmentMode: null,
},
}
*/
`

An error is thrown if the provided pattern or flags aren't valid in Oniguruma.

> Note: toOnigurumaAst is a wrapper around the parser module's parse function that makes it easier to use by automatically providing the appropriate Unicode property validation data.

🌀 Traverse and transform an AST

See details and examples in the traverser module's readme.

↩️ Convert an AST to a pattern

See details and examples in the generator module's readme.

🪄 Optimize regexes

This library includes one of the few implementations (for any regex flavor) of a "regex optimizer" that can minify and improve the performance and readability of regexes prior to use.

Example:

`
(?x) (?:\!{1,}) (\b(?:ark|arm|art)\b) [[^0-9A-Fa-f]\P{^Nd}\p{ Letter }]
`

Becomes:

`
!+\b(ar[kmt])\b[\H\d\p{L}]
`

Optimized regexes always match exactly the same strings.

See more details and examples in the optimizer module's readme.

> [!TIP]
> 🧪 Try the optimizer demo.

🆚 Known differences

Known differences will be resolved in future versions.

$3

The following rarely-used features throw errors since they aren't yet supported:

- Rarely-used character specifiers: Non-A-Za-z with \cx \C-x, meta \M-x \M-\C-x, octal code points \o{…}, and octal encoded bytes ≥ \200.
- Code point sequences: \x{H H …} \o{O O …}.
- Absence expressions (?~|…|…), stoppers (?~|…), and clearers (?~|).
- Conditionals: (?(…)…), etc.
- Non-built-in callouts: (?{…}), etc.
- Numbered forward backreferences (incl. relative \k<+N>) and backreferences with recursion level (\k, etc.).
- Flags D P S W y{g} y{w} within pattern modifiers, and whole-pattern modifiers C I L.

Despite these gaps, more than 99.99% of real-world Oniguruma regexes are supported, based on a sample of ~55k regexes used in TextMate grammars (conditionals were used in three regexes, and other unsupported features weren't used at all). Some of the Oniguruma features above are so exotic that they aren't used in any public code on GitHub.

$3

#### Unenclosed four-digit backreferences

Although any number of digits are supported for enclosed \k<…>/\k'…' backreferences (assuming the backreference refers to a valid capturing group), unenclosed backreferences currently support only up to three digits (\999). In other words, \1000 is handled as \100 followed by 0 even if 1,000+ captures appear to the left.

> Note: An apparent bug in vscode-oniguruma (v2.0.1 tested) prevents any regex with more than 999 captures from working. They fail to match anything, with no error.

#### Erroring on patterns that trigger Oniguruma bugs

This library intentionally doesn't reproduce bugs, and it currently throws errors for several edge cases that trigger Oniguruma bugs and undefined behavior.

🔢 Oniguruma version

All versions of this library to date have followed the rules of Oniguruma 6.9.10 (released 2025-01-01), which uses Unicode 16.0.0.

At least since Oniguruma 6.0.0 (released 2016-05-09), regex syntax changes in new versions have been backward compatible. Some versions added new syntax that was previously an error (such as new Unicode property names), and in a few cases, edge case parsing bugs were fixed.

> Oniguruma 6.9.8 (released 2022-04-29) is an important baseline for JavaScript projects, since that's the version used by vscode-oniguruma 1.7.0 to the latest 2.0.1. It's therefore used in recent versions of various projects, including VS Code and Shiki. However, the regex syntax differences between Oniguruma 6.9.8 and 6.9.10 are so minor that this is a non-issue.

🧩 Contributing

Contributions are welcome. See the guide to help you get started.

🏷️ About

Created by Steven Levithan and contributors.

If you want to support this project, I'd love your help by contributing improvements (guide), sharing it with others, or sponsoring ongoing development.

MIT License.

[npm-version-src]: https://img.shields.io/npm/v/oniguruma-parser?color=78C372
[npm-version-href]: https://npmjs.com/package/oniguruma-parser
[npm-downloads-src]: https://img.shields.io/npm/dm/oniguruma-parser?color=78C372
[npm-downloads-href]: https://npmjs.com/package/oniguruma-parser
[bundle-src]: https://img.shields.io/bundlejs/size/oniguruma-parser?color=78C372&label=minzip
[bundle-href]: https://bundlejs.com/?q=oniguruma-parser&treeshake=[*]