picoLLM Inference Engine Web Binding

Made in Vancouver, Canada by Picovoice

picoLLM Inference Engine

picoLLM Inference Engine is a highly accurate and cross-platform SDK optimized for running compressed large language
models. picoLLM Inference Engine is:

- Accurate; picoLLM Compression improves GPTQ by significant margins
- Private; LLM inference runs 100% locally.
- Cross-Platform
- Runs on CPU and GPU
- Free for open-weight models

Compatibility

- Chrome / Edge
- Firefox
- Safari

Requirements

PicoLLM Web Binding uses SharedArrayBuffer.

Include the following headers in the response to enable the use of SharedArrayBuffers:

``
Cross-Origin-Opener-Policy: same-origin
Cross-Origin-Embedder-Policy: require-corp
`

Refer to our Web demo for an example on creating a server with the corresponding response headers.

NOTE: IndexedDB, SIMD and SharedArrayBuffers are required to use picoLLM. PicoLLM only supports running in a worker thread.

Installation

Using Yarn:

`console
yarn add @picovoice/picollm-web
`

or using npm:

`console
npm install --save @picovoice/picollm-web
`

Models

picoLLM Inference Engine on Web supports the following open-weight models. The models are on
Picovoice Console.

- Gemma
- gemma-2b
- gemma-2b-it
- Llama-2
- llama-2-7b
- llama-2-7b-chat
- Llama-3
- llama-3-8b
- llama-3-8b-instruct
- Llama-3.2
- llama3.2-1b-instruct
- llama3.2-3b-instruct
- Mistral
- mistral-7b-v0.1
- mistral-7b-instruct-v0.1
- mistral-7b-instruct-v0.2
- Phi-2
- phi2
- Phi-3
- phi3
- Phi-3.5
- phi3.5

NOTE: Only Gemma, Phi-2, and Phi-3 models have been tested on multiple browsers across different platforms.
The rest of the models depend on the user's system in order to run properly.

AccessKey

AccessKey is your authentication and authorization token for deploying Picovoice SDKs, including picoLLM. Anyone who is
using Picovoice needs to have a valid AccessKey. You must keep your AccessKey secret. You would need internet
connectivity to validate your AccessKey with Picovoice license servers even though the LLM inference is running 100%
offline and completely free for open-weight models. Everyone who signs up for
Picovoice Console receives a unique AccessKey.

Usage

$3

picoLLM accepts model files in three different types:

#### File URL(s):

`typescript
const modelFile = ${SERVER_URL}/${PATH_TO_MODEL_FILE};
`

or if the model file is too big (2GB or larger) consider using chunks:

`typescript
const modelFile = [
${SERVER_URL}/${PATH_TO_MODEL_FILE_PART1},
${SERVER_URL}/${PATH_TO_MODEL_FILE_PART2},
... // add more parts if needed
];
`

#### File Object(s):

`typescript
const modelFile = new File([/ file contents /]);
`

or if the model file is too big (2GB or larger) consider using chunks:

`typescript
const modelFile = [
new File([/ file contents part 1 /]),
new File([/ file contents part 2 /]),
... // add more parts if needed
];
`

File objects are usually used with HTML's input tag:

`html



`

#### Blob Object(s):

`typescript
const modelFile = new Blob([new Uint8Array(/ model bytes /)]);
`

or if the model file is too big (2GB or larger) consider using chunks:

`typescript
const modelFile = [
new Blob([new Uint8Array(/ model bytes part 1 /)]),
new Blob([new Uint8Array(/ model bytes part 2 /)]),
... // add more parts if needed
];
`

#### picoLLM Model

picoLLM saves and caches your parameter model file (.pllm) in IndexedDB to be
used by Web Assembly. Use a different cacheFilePath variable to hold and cache
multiple model values and set the cacheFileOverwrite value to true to force
re-save the model file. If the model file changes, cacheFileVersion should be
incremented to force the cached models to be updated. Use numFetchRetries to
change the number of fetch retry attempts for the model file.

`typescript
const picoLLMModel = {
modelFile: modelFile, // Based on the sections before,

// Optional
cacheFilePath: 'custom_model',
cacheFileOverwrite: true,
cacheFileVersion: 1,
numFetchRetries: 0,
}
`

$3

Initialize an instance of picoLLM in a worker thread:

`typescript
const picoLLM = await PicoLLMWorker.create(
${ACCESS_KEY},
picoLLMModel,
);
`

Replace ${ACCESS_KEY} with yours obtained from Picovoice Console.

$3

`typescript
const res = await picoLLM.generate(${PROMPT});
console.log(res.completion);
`

Replace ${PROMPT} with a prompt string.

$3

Instruction-tuned models (e.g., llama-2-7b-chat, and gemma-2b-it) have a specific chat
template. You can either directly format the prompt or use a dialog helper:

`typescript
const dialog = picoLLM.getDialog()
dialog.addHumanRequest(prompt)

const res = await picoLLM.generate(dialog.prompt())
dialog.addLLMResponse(res.completion)
print(res.completion)
`

$3

`typescript
picoLLM.interrupt();
`

This will stop text generation and if it was properly interrupted, it will set res.completion.endpoint
as an interrupted state.

$3

Clean up used resources by picoLLM or picoLLMWorker:

`typescript
await picoLLM.release()
`

Demos

Refer to our Web demos for examples using LLM completion
and chat using picoLLM`.