lloyal.node

Covalent inference for Node.js

Forkable inference state for llama.cpp — Branch a generation into a tree — prefix sharing is the bond across branches while each owns its own machinery (sampler chain, seed, grammar, logits snapshot, perplexity tracker) enabling controlled divergence at decode time.

Branch API

Fork from root for best-of-N, fork from children for MCTS/beam search, fork from a draft for speculative decoding. The produce/commit protocol separates sampling from state advancement — sample without writing to KV, inspect the result, then decide whether to commit.

``javascript
import { createContext, Branch } from "@lloyal-labs/lloyal.node";

const ctx = await createContext({ modelPath: "./model.gguf", nSeqMax: 8 });
const tokens = await ctx.tokenize("Once upon a time");
await ctx.decode(tokens, 0, 0);

// Create root branch, freeze logits from prefill
const root = Branch.create(ctx, 0, tokens.length, { temperature: 0.8 });
root.captureLogits();

// Fork N candidates — KV prefix shared, sampler/grammar/logits/perplexity cloned
const candidates = [1, 2, 3, 4, 5].map((seqId, i) => {
const branch = root.fork(seqId);
branch.reseedSampler(1000 + i);
return branch;
});

// Generate (interleaved round-robin)
for (let t = 0; t < 50; t++) {
for (const branch of candidates) {
const { token, isStop } = branch.produce(); // Sample, no KV write
if (isStop) continue;
branch.commit(token); // Accept + forward pass + capture
}
}

// Select by perplexity, prune losers
const best = candidates.reduce((a, b) => (a.perplexity < b.perplexity ? a : b));
for (const c of candidates) {
if (c !== best) c.prune();
}
`

What fork() shares: KV cache prefix (metadata-only under unified KV — no tensor buffers copied).

What fork() clones: Logits snapshot, sampler chain (penalties + PRNG), grammar state, logit bias, perplexity tracker.

Key methods:

- produce() / commit() — two-phase: sample without KV write, then advance
- prune() — discard loser and its divergent KV entries
- destroy() — release handle, keep KV (for winners continuing with raw ops)
- reseedSampler() — unique PRNG per fork for stochastic diversity
- perplexity — rolling PPL per branch for quality-based selection

---

Install

`bash
npm install @lloyal-labs/lloyal.node
`

Prebuilt binaries for 13 platform/GPU combinations. GPU selection at runtime, not install time.

| Platform | Arch | Acceleration |
| -------- | ----- | ------------------- |
| macOS | arm64 | Metal |
| macOS | x64 | CPU |
| Linux | x64 | CPU / CUDA / Vulkan |
| Linux | arm64 | CPU / CUDA / Vulkan |
| Windows | x64 | CPU / CUDA / Vulkan |
| Windows | arm64 | CPU / Vulkan |

CI integration testing (real inference) matrix:

| Architecture | Test Model | Template |
| ------------ | -------------- | -------- |
| Llama | Llama 3.2 1B | llama3 |
| Phi | Phi 3.5 Mini | phi3 |
| Qwen | Qwen 3 1.7B | chatml |
| Gemma | Gemma 3 1B | gemma |
| SmolLM | SmolLM2 1.7B | chatml |
| TinyLlama | TinyLlama 1.1B | zephyr |

See distribution.md for details.

---

Examples

| Example | Pattern |
| ----------------------------------------- | -------------------------------------------------------------------------- |
| best-of-n/ | Branch API: fork, produce/commit, perplexity selection |
| speculative/ | Branch API: draft/verify, fork/prune, bonus token sampling |
| streaming/ | Infinite context via BlinkKV reseeding with sidecar summarization |
| entropy/ | modelEntropy() mid-generation as control signal |
| grammar/ | Pull loop with generators, JSON schema constraints, KV + grammar branching |
| chat/ | Interactive streaming chat |
| embed/ | Text embeddings extraction |

`bash
node examples/best-of-n/best-of-n.mjs
node examples/speculative/speculative.mjs
`

Each example has a README explaining the pattern.

---

Other Patterns

$3

Model uncertainty mid-generation enables dynamic behavior:

`javascript
const entropy = ctx.modelEntropy("bits");

if (entropy > 4.0) {
// High uncertainty — model is guessing
// Trigger retrieval, reduce temperature, or branch
}
`

See examples/entropy/ for entropy-triggered sampling strategies.

$3

For fine-grained control without Branch:

| Approach | Method | Use Case |
| -------------------- | --------------------------------- | -------------------------------------------- |
| Sequence copy | kvSeqCopy(src, dst) | Share prefix across sequences |
| Snapshot/restore | kvCacheSave() / kvCacheLoad() | Sequential exploration, return to checkpoint |

$3

`javascript
const grammar = ctx.jsonSchemaToGrammar(schema);
const handle = ctx.createSampler(grammar);
// Pull loop — consumer controls pace, can branch at any point
`

See examples/grammar/ for the full pull loop pattern.

---

API Reference

Full API documentation: lloyal-ai.github.io/lloyal.node

Generated from lib/index.d.ts` with TypeDoc.

---

Ecosystem

| Package | Runtime | Description |
| ------------------------------------------------------- | ------------ | --------------------------------- |
| liblloyal | C++ | Header-only inference kernel |
| lloyal.node | Node.js | This package |
| nitro-llama | React Native | Mobile bindings via Nitro Modules |
| tsampler | TypeScript | Reference sampler implementation |

Contributing

See CONTRIBUTING.md for development setup and release process.

License

Apache 2.0 — See LICENSE for details.