Showing 1-20 of 50 packages
tiktoken is a fast [BPE](https://en.wikipedia.org/wiki/Byte_pair_encoding) tokeniser for use with OpenAI's models.
Dynamic file contents manipulation in node.js utilising inline token strings.
Taiwanese Hokkien Transliterator and Tokeniser
A simple string tokeniser
Parse simple expressions, in a language of your own description
JS/WASM bindings for tiktoken
Fast CBOR with a focus on strictness
JS/WASM bindings for tiktoken
RFC1459 and IRCv3 protocol tokeniser
JavaScript port of tiktoken
search and replace tokens from a source file, writing to a new file
SQL tokeniser & parser
Hunspell compatible spell checker
HTML tokeniser that supports streaming interfaces. Converts HTML into tokens and provides options to transforma and convert those tokens back into a HTML string.
tiktoken is a fast [BPE](https://en.wikipedia.org/wiki/Byte_pair_encoding) tokeniser for use with OpenAI's models.
A simple tokenizer for encoding/decoding text into numeric tokens.
Tokenizes a text from left to right and returns the list of tokens with their types.
Calculates the word frequency of a text document.
JavaScript port of tiktoken
parser Edge to convert text markup to lexer tokens