Showing 1-20 of 14,791 packages
Tokenize paragraphs into sentences, and smaller tokens.
Minimal japanese sentence tokenizer written in 100% pure TypeScript.
English word and sentence tokenizer, for natural language processing.
A port of NLTK's Punkt sentence tokenizer to JS.
Tokenize CSS
A promise based streaming tokenizer
Tokenized zip support
TypeScript definition for strtok3 token
Multilingual tokenizer that automatically tags each token with its type
A tokenzier for Sass' SCSS syntax
Simple HTML Tokenizer is a lightweight JavaScript library that can be used to tokenize the kind of HTML normally found in templates.
A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models
Split text into sentences with Sentence Boundary Detection (SBD).
Parses and stringifies CSS selectors
Algorithms to help you parse CSS from an array of tokens.
JS tokenizer for LLaMA-based LLMs
Tokenizes a string that represents a regular expression.
Solve CSS math expressions
r/w stream of glsl tokens
Tiny JavaScript tokenizer.