What is this?

GPT Token Utilities is a small library for encoding and decoding text to and from the tokenized format used by OpenAI's GPT models.

Why should I use this?

$3

Our token encoder/decoder is optimized for a balance of speed and ease of use. No external dependencies are required.

$3

Everything is written in TypeScript and includes type definitions with full documentation. This library is isomorphic and can be used in both Node and the browser!

$3

GPT Token Utils balances a small footprint with a full-featured API.
It's also tree-shakeable, so you can import only the functions you need.

Installation

$3

``bash
yarn add gpt-token-utils


or


npm install --save gpt-token-utils
`

Usage

Encoding and Decoding Text

The encode and decode exports are the main functions you'll use to work with GPT tokens.

`js
import { encode, decode } from 'gpt-token-utils'

// Encode a string to a list of tokens...
const tokens = encode('Humans are strange creatures, and ever so curious too!')

// You've got a list of tokens!
console.log(tokens) // [32661, 504, 389, 6283, 8109, 11, 290, 1683, 523, 11040, 1165, 0]

// How many tokens are there?
console.log(tokens.length) // 6

// Can we decode it back to text?
console.log(decode(tokens)) // "Humans are strange creatures...."
`

If you're working with Codex-based models optimized for strings of code, you can also use the encodeCodex and decodeCodex exports.

`js
import { encodeCodex, decodeCodex } from 'gpt-token-utils'

const codeExample = [
'function deeplyNested () {',
' return {',
' the: {',
' quick: {',
' brown: {',
' fox: {',
' jumps: {',
' over: {',
' the: {',
' lazy: {',
' dog: {',
' }',
' }',
' }',
' }',
' }',
' }',
' }',
' }',
' }',
' }',
'}',
].join('\n')

const tokens = encodeCodex(codeExample)
`

$3

By default, GPT Token Utils includes a sizable vocabulary and encoder. Alternatively, you can pass in your own to customize the encoding/decoding process.

`js
import {BytePairEncoder} from 'gpt-token-utils/BytePairEncoder'

const tokenEncoder = new BytePairEncoder({...})
`

`js
import {BytePairDecoder} from 'gpt-token-utils/BytePairDecoder'

const tokenDecoder = new TokenDecoder({...})
``

License

GPT Token Utils is licensed under the MIT License. If you've got something cool to share that's built with this library, let us know at @SisterSoftware! We would love to see it!