Showing 1-20 of 210,100 packages
A promise based streaming tokenizer
Tokenized zip support
Tokenize CSS
TypeScript definition for strtok3 token
A tokenzier for Sass' SCSS syntax
Common token types for decoding and encoding numeric and string values
A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models
Simple HTML Tokenizer is a lightweight JavaScript library that can be used to tokenize the kind of HTML normally found in templates.
Parses and stringifies CSS selectors
Algorithms to help you parse CSS from an array of tokens.
Tokenizes a string that represents a regular expression.
Solve CSS math expressions
Claude tokenizer
r/w stream of glsl tokens
Tiny JavaScript tokenizer.
ProseMirror Markdown integration
tokenizer of source code for jscpd
JavaScript implementation of Japanese morphological analyzer
Tokenize a shell string into argv array
Multilingual tokenizer that automatically tags each token with its type