picoLLM Inference Engine is a highly accurate and cross-platform SDK optimized for running compressed large language models.
npm install @picovoice/picollm-webMade in Vancouver, Canada by Picovoice
picoLLM Inference Engine is a highly accurate and cross-platform SDK optimized for running compressed large language
models. picoLLM Inference Engine is:
- Accurate; picoLLM Compression improves GPTQ by significant margins
- Private; LLM inference runs 100% locally.
- Cross-Platform
- Runs on CPU and GPU
- Free for open-weight models
- Chrome / Edge
- Firefox
- Safari
PicoLLM Web Binding uses SharedArrayBuffer.
Include the following headers in the response to enable the use of SharedArrayBuffers:
```
Cross-Origin-Opener-Policy: same-origin
Cross-Origin-Embedder-Policy: require-corp
Refer to our Web demo for an example on creating a server with the corresponding response headers.
NOTE: IndexedDB, SIMD and SharedArrayBuffers are required to use picoLLM. PicoLLM only supports running in a worker thread.
Using Yarn:
`console`
yarn add @picovoice/picollm-web
or using npm:
`console`
npm install --save @picovoice/picollm-web
picoLLM Inference Engine on Web supports the following open-weight models. The models are on
Picovoice Console.
- Gemma
- gemma-2bgemma-2b-it
- llama-2-7b
- Llama-2
- llama-2-7b-chat
- llama-3-8b
- Llama-3
- llama-3-8b-instruct
- llama3.2-1b-instruct
- Llama-3.2
- llama3.2-3b-instruct
- mistral-7b-v0.1
- Mistral
- mistral-7b-instruct-v0.1
- mistral-7b-instruct-v0.2
- phi2
- Phi-2
- phi3
- Phi-3
- phi3.5
- Phi-3.5
-
NOTE: Only Gemma, Phi-2, and Phi-3 models have been tested on multiple browsers across different platforms.
The rest of the models depend on the user's system in order to run properly.
AccessKey is your authentication and authorization token for deploying Picovoice SDKs, including picoLLM. Anyone who is
using Picovoice needs to have a valid AccessKey. You must keep your AccessKey secret. You would need internet
connectivity to validate your AccessKey with Picovoice license servers even though the LLM inference is running 100%
offline and completely free for open-weight models. Everyone who signs up for
Picovoice Console receives a unique AccessKey.
picoLLM accepts model files in three different types:
#### File URL(s):
`typescript${SERVER_URL}/${PATH_TO_MODEL_FILE}
const modelFile = ;`
or if the model file is too big (2GB or larger) consider using chunks:
`typescript${SERVER_URL}/${PATH_TO_MODEL_FILE_PART1}
const modelFile = [
,${SERVER_URL}/${PATH_TO_MODEL_FILE_PART2}
,...
// add more parts if needed`
];
#### File Object(s):
`typescript`
const modelFile = new File([/ file contents /]);
or if the model file is too big (2GB or larger) consider using chunks:
`typescript`
const modelFile = [
new File([/ file contents part 1 /]),
new File([/ file contents part 2 /]),
... // add more parts if needed
];
File objects are usually used with HTML's input tag:
`html
`
#### Blob Object(s):
`typescript`
const modelFile = new Blob([new Uint8Array(/ model bytes /)]);
or if the model file is too big (2GB or larger) consider using chunks:
`typescript`
const modelFile = [
new Blob([new Uint8Array(/ model bytes part 1 /)]),
new Blob([new Uint8Array(/ model bytes part 2 /)]),
... // add more parts if needed
];
#### picoLLM Model
picoLLM saves and caches your parameter model file (.pllm) in IndexedDB to becacheFilePath
used by Web Assembly. Use a different variable to hold and cache cacheFileOverwrite
multiple model values and set the value to true to force cacheFileVersion
re-save the model file. If the model file changes, should benumFetchRetries
incremented to force the cached models to be updated. Use to
change the number of fetch retry attempts for the model file.
`typescript`
const picoLLMModel = {
modelFile: modelFile, // Based on the sections before,
// Optional
cacheFilePath: 'custom_model',
cacheFileOverwrite: true,
cacheFileVersion: 1,
numFetchRetries: 0,
}
Initialize an instance of picoLLM in a worker thread:
`typescript`
const picoLLM = await PicoLLMWorker.create(
${ACCESS_KEY},
picoLLMModel,
);
Replace ${ACCESS_KEY} with yours obtained from Picovoice Console.
`typescript${PROMPT}
const res = await picoLLM.generate();`
console.log(res.completion);
Replace ${PROMPT} with a prompt string.
Instruction-tuned models (e.g., llama-2-7b-chat, and gemma-2b-it) have a specific chat
template. You can either directly format the prompt or use a dialog helper:
`typescript
const dialog = picoLLM.getDialog()
dialog.addHumanRequest(prompt)
const res = await picoLLM.generate(dialog.prompt())
dialog.addLLMResponse(res.completion)
print(res.completion)
`
`typescript`
picoLLM.interrupt();
This will stop text generation and if it was properly interrupted, it will set res.completion.endpoint
as an interrupted state.
Clean up used resources by picoLLM or picoLLMWorker:
`typescript`
await picoLLM.release()
Refer to our Web demos for examples using LLM completion
and chat using picoLLM`.