⏳ js-tiktoken

tiktoken is a BPE tokeniser for use with
OpenAI's models. This is a pure JS port of the original tiktoken library.

Install the library from NPM:

``
npm install @dingyi222666/js-tiktoken
`

This fork remove the default encoding files, you need to load the encoding files by yourself.

Usage

You can only load the ranks you need, which will significantly reduce the bundle size:

`typescript
import { Tiktoken } from "js-tiktoken/lite";
import o200k_base from "js-tiktoken/ranks/o200k_base";

const enc = new Tiktoken(o200k_base);
assert(enc.decode(enc.encode("hello world")) === "hello world");
`

Alternatively, encodings can be loaded dynamically from our CDN hosted on Cloudflare Pages.

`typescript
import { Tiktoken } from "js-tiktoken/lite";

const res = await fetch(https://tiktoken.pages.dev/js/o200k_base.json);
const o200k_base = await res.json();

const enc = new Tiktoken(o200k_base);
assert(enc.decode(enc.encode("hello world")) === "hello world");
`

Full usage

If you need all the OpenAI tokenizers, you can import the entire library:

> [!CAUTION]
> This will include all the OpenAI tokenizers, which may significantly increase the bundle size. See

Lite

You can only load the ranks you need, which will significantly reduce the bundle size:

`typescript
import { Tiktoken } from "js-tiktoken/lite";
import o200k_base from "js-tiktoken/ranks/o200k_base";

const enc = new Tiktoken(o200k_base);
assert(enc.decode(enc.encode("hello world")) === "hello world");
`

Alternatively, encodings can be loaded dynamically from our CDN hosted on Cloudflare Pages.

`typescript
import { Tiktoken } from "js-tiktoken/lite";

const res = await fetch(https://tiktoken.pages.dev/js/o200k_base.json);
const o200k_base = await res.json();

const enc = new Tiktoken(o200k_base);
assert(enc.decode(enc.encode("hello world")) === "hello world");
`

Full usage

If you need all the OpenAI tokenizers, you can import the entire library:

> [!CAUTION]
> This will include all the OpenAI tokenizers, which may significantly increase the bundle size. See

`typescript
import assert from "node:assert";
import { getEncoding, encodingForModel } from "js-tiktoken";

const enc = getEncoding("gpt2");
assert(enc.decode(enc.encode("hello world")) === "hello world");
``