fsm-tokenizer


finite state machine tokenizer

Contents

- What is this?
- When should I use this?
- Install
- Use
- API
- Types
- Contribute

What is this?

This package is a tokenization utility. The underlying tokenizer is implemented as a state machine that produces tokens
with positional information, as well as any other fields attached when the token was opened. Tokens are turned into
events, with the tokenizer calling any defined resolvers in the process.

When should I use this?

This package can be used to tokenize a [file][vfile], Uint8Array, string, or a list containing these values,
like command-line arguments. Developers familiar with [micromark][] will find that tokenizers and constructs (objects
used to define how to parse text) in this package work similarly.

Install

This package is [ESM only][esm].

In Node.js with [yarn][]:

``sh
yarn add @flex-development/fsm-tokenizer
`



See Git - Protocols | Yarn
 for details regarding installing from Git.



In Deno with [esm.sh][esmsh]:

`ts
import { tokenize } from 'https://esm.sh/@flex-development/fsm-tokenizer'
`

In browsers with [esm.sh][esmsh]:

`html

`

Use

TODO: use

API

TODO: api

> Please refer to the source code for documentation.

Types

This package is fully typed with [TypeScript][].

Contribute

See CONTRIBUTING.md`.

This project has a code of conduct. By interacting with this repository, organization, or
community you agree to abide by its terms.

[esm]: https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3ecc99d99c

[esmsh]: https://esm.sh

[micromark]: https://github.com/micromark/micromark

[typescript]: https://www.typescriptlang.org

[vfile]: https://github.com/vfile/vfile

[yarn]: https://yarnpkg.com