llama.cpp-ts 🦙


LlamaCPP-ts is a Node.js binding for the LlamaCPP library, which wraps the llama.cpp framework. It provides an easy-to-use interface for running language models in Node.js applications, supporting asynchronous streaming responses.

Supported Systems:

- MacOS
- Windows (not tested yet)
- Linux (not tested yet)

Models

You can find some models here

Example is using this one Meta-Llama-3.1-8B-Instruct-Q3_K_S.gguf.

Installation

Ensure that you have CMake installed on your system:

- On MacOS: brew install cmake
- On Windows: choco install cmake
- On Linux: sudo apt-get install cmake

Then, install the package:

``bash
npm install llama.cpp-ts


or


yarn add llama.cpp-ts
`

Usage

$3

`javascript
const { Llama } = require('llama.cpp-ts');

async function main() {
const llama = new Llama();
const modelPath = "./path/to/your/model.gguf";
const modelParams = { nGpuLayers: 32 };
const contextParams = { nContext: 2048 };

if (!llama.initialize(modelPath, modelParams, contextParams)) {
console.error("Failed to initialize the model");
return;
}

llama.setSystemPrompt("You are a helpful assistant. Always provide clear, concise, and accurate answers.");

const question = "What is the capital of France?";
const tokenStream = llama.prompt(question);

console.log("Question:", question);
console.log("Answer: ");

while (true) {
const token = await tokenStream.read();
if (token === null) break;
process.stdout.write(token);
}
}

main().catch(console.error);
`

API Reference

$3

The Llama class provides methods to interact with language models loaded through llama.cpp.

#### Public Methods

- constructor(): Creates a new Llama instance.
- initialize(modelPath: string, modelParams?: object, contextParams?: object): boolean: Initializes the model with the specified path and parameters.
- setSystemPrompt(systemPrompt: string): void: Sets the system prompt for the conversation.
- prompt(userMessage: string): TokenStream: Streams the response to the given prompt, returning a TokenStream object.
- resetConversation(): void: Resets the conversation history.

$3

The TokenStream class represents a stream of tokens generated by the language model.

#### Public Methods

- read(): Promise: Reads the next token from the stream. Returns null when the stream is finished.

Example

Here's a more comprehensive example demonstrating the usage of the library:

`javascript
const { Llama } = require('llama.cpp-ts');

async function main() {
const llama = new Llama();
const modelPath = __dirname + "/models/Meta-Llama-3.1-8B-Instruct-Q3_K_S.gguf";
const modelParams = { nGpuLayers: 32 };
const contextParams = { nContext: 2048 };

if (!llama.initialize(modelPath, modelParams, contextParams)) {
console.error("Failed to initialize the model");
return;
}

llama.setSystemPrompt("You are a helpful assistant. Always provide clear, concise, and accurate answers.");

const questions = [
"What is the capital of France?",
"What's the population of that city?",
"What country is the city in?"
];

for (const question of questions) {
const tokenStream = llama.prompt(question);

console.log("Question:", question);
console.log("Answer: ");

while (true) {
const token = await tokenStream.read();
if (token === null) break;
process.stdout.write(token);
}

console.log("\n");
}
}

main().catch(console.error);
``