@retune.so/tokenizer

v0.4.0

GPT-3 encoder/decoder for Typescript based project

0/weekUpdated 1 years agoMITUnpacked: 3.3 MB

Published by dhananjoyb

npm install @retune.so/tokenizer

npm

GPT-3-Encoder

GPT-3 Encoder/Decoder for Typescript based project

About

GPT-2 and GPT-3 use byte pair encoding to turn text into a series of integers to feed into the model. This is a javascript implementation of OpenAI's original python encoder/decoder which can be found here

@retune.so/tokenizer - npm explorer