Showing 1-20 of 1,521 packages
A wide purpose tokenizer for node.js which looks like a stream
Tiny JavaScript tokenizer.
A promise based streaming tokenizer
Claude tokenizer
Lexer / tokenizer
tokenizer of source code for jscpd
Optimised tokenizer/lexer generator! 🐄 Much performance. Moo!
A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models
Multilingual tokenizer that automatically tags each token with its type
JS tokenizer for LLaMA-based LLMs
Tokenizer for processing admonitions
JSON AST parser, tokenizer, printer, traverser.
Simple HTML Tokenizer is a lightweight JavaScript library that can be used to tokenize the kind of HTML normally found in templates.
Amazon S3 tokenizer
CSS / Extended CSS tokenizer
gemma3 tokenizer for NodeJS/Browser
Range-request tokenizer adapter
Tokenizer for processing fenced math block (formulas)
Tokenizer utilities for Tokenlens.
[](https://github.com/botisan-ai/gpt3-tokenizer/actions/workflows/main.yml) [](https://www.npmjs.com/