_(inspired by [tiktoken-node](https://github.com/ceifa/tiktoken-node))_
npm install tiktoken-rs-node_(inspired by tiktoken-node)_
napi Rust bindings for tiktoken-rs, featuring zero-copy encode and no WASM.
``bash`
npm install tiktoken-rs-node
`ts
import { getEncoding } from "tiktoken-rs-node";
// If you're doing this in a long-running process, make sure to do it once per-startup.
const encoding = getEncoding("gpt2");
const tokens = encoding.encode("Hello, world!");
const decodedString = encoding.decode(tokens);
`
- o200k_basecl100k_base
- p50k_edit
- p50k_base
- r50k_base
- gpt2
-
`bash``
yarn test # Run all tests
yarn benchmark # Run smoke-test bench marks
yarn benchmark:comprehensive # Run a benchmark on every encoding w/ various text lengths
yarn test:all # Run all tests and benchmarks
- [ ] Convert from UTF-8 to UTF-16 on the fly during decode so that the string can be zero-copy UTF-16 for V8.
- [ ] Investigate modifying the BPE algorithm to optimize for a counting fast path (without storing every token in memory).