@lenml/tokenizers

a lightweight no-dependency fork from transformers.js (only tokenizers)

- github: https://github.com/lenML/tokenizers
- tokenizers-arena: https://lenml.github.io/tokenizers-arena/#/side-by-side

Usage

install

$3

``
npm install @lenml/tokenizers
`

$3

`html


`

load tokenizer

$3

`ts
import { TokenizerLoader } from "@lenml/tokenizers";
const tokenizer = TokenizerLoader.fromPreTrained({
tokenizerJSON: {
/ ... /
},
tokenizerConfig: {
/ ... /
},
});
`

$3

`ts
import { TokenizerLoader } from "@lenml/tokenizers";
const sourceUrls = {
tokenizerJSON:
"https://huggingface.co/HuggingFaceH4/zephyr-7b-gemma-v0.1/resolve/main/tokenizer.json?download=true",
tokenizerConfig:
"https://huggingface.co/HuggingFaceH4/zephyr-7b-gemma-v0.1/resolve/main/tokenizer_config.json?download=true",
};
const tokenizer = await TokenizerLoader.fromPreTrainedUrls(sourceUrls);
// or from fetch
const tokenizer = TokenizerLoader.fromPreTrained({
tokenizerJSON: await fetch(sourceUrls.tokenizerJSON).then((r) => r.json()),
tokenizerConfig: await fetch(sourceUrls.tokenizerConfig).then((r) =>
r.json()
),
});
`

$3

`ts
import { fromPreTrained } from "@lenml/tokenizer-llama3";
const tokenizer = fromPreTrained();
`

chat template

`ts
const tokens = tokenizer.apply_chat_template([
{
role: "system",
content: "You are helpful assistant.",
},
{
role: "user",
content: "Hello, how are you?",
},
]) as number[];

const chat_content = tokenizer.decode(tokens);

console.log(chat_content);
`

output:

`
<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are helpful assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>

Hello, how are you?<|eot_id|><|start_header_id|>assistant<|end_header_id|>
`

tokenizer api

`ts
console.log(
"encode() => ",
tokenizer.encode("Hello, my dog is cute", null, {
add_special_tokens: true,
})
);
console.log(
"_encode_text() => ",
tokenizer._encode_text("Hello, my dog is cute")
);
`

> fully tokenizer api: transformer.js tokenizers document



get lightweight transformers.tokenizers



In the @lenml/tokenizers package, you can get a lightweight no-dependency implementation of tokenizers:

> Since all dependencies related to huggingface have been removed in this library, although the implementation is the same, it is not possible to load models using the form hf_user/repo.

`ts
import { tokenizers } from "@lenml/tokenizers";

const {
CLIPTokenizer,
AutoTokenizer,
CohereTokenizer,
VitsTokenizer,
WhisperTokenizer,
// ...
} = tokenizers;
``

License

Apache-2.0