Results for "tiktoken"

JS/WASM bindings for tiktoken

JavaScript port of tiktoken

JS/WASM bindings for tiktoken

_(inspired by [tiktoken-node](https://github.com/ceifa/tiktoken-node))_

tiktoken is a fast [BPE](https://en.wikipedia.org/wiki/Byte_pair_encoding) tokeniser for use with OpenAI's models.

This is the **x86_64-unknown-linux-gnu** binary for `tiktoken-rs-node`

This is the **x86_64-unknown-linux-gnu** binary for `@anysphere/tiktoken-node`

JavaScript port of tiktoken

Tiktoken but thicker; A bundled tokenizer with added helper functions

This is the **aarch64-apple-darwin** binary for `tiktoken-rs-node`

A faster than tiktoken tokenizer with first-class support for Vercel's AI SDK.

We use our own fork for now because we are making changes that may not be useful to everyone. For example, we add support for special tokens, and we also add support for running tokenization asynchronously with the computation happening on a different thr

JavaScript port of tiktoken

This is the **aarch64-unknown-linux-gnu** binary for `@anysphere/tiktoken-node`

This is the **aarch64-apple-darwin** binary for `@anysphere/tiktoken-node`

This is the **x86_64-pc-windows-msvc** binary for `@anysphere/tiktoken-node`

Universal AI memory layer with seamless auto-setup for Claude Code, Cursor, Windsurf, and GitHub Copilot. Intelligent Learning, token savings metrics (tiktoken), and more!

This is the **x86_64-apple-darwin** binary for `@anysphere/tiktoken-node`

A lightweight tokenizer for OpenAI's GPT model series. Uses OpenAI's tiktoken python package

JavaScript port of tiktoken