Fast and versatile tokenizer for language models, supporting BPE, Unigram and WordPiece tokenization
npm install kitoken
README.md