llm

Package to connect and trace LLM calls.

Usage

``ts
import { LLM } from "@empiricalrun/llm";

const llm = new LLM({
provider: "openai",
defaultModel: "gpt-4o",
});
const llmResponse = await llm.createChatCompletion({ ... });
`

Vision utilities

This package also contains utilities for vision.

$3

Ask a question against the image (e.g. to extract some info, make a decision) and get the answer.

`ts
import { query } from "@empiricalrun/llm/vision";

// With Appium
const data = await driver.saveScreenshot("dummy.png");
const instruction =
"Extract number of ATOM tokens from the image. Return only the number.";

const text = await query(data.toString("base64"), instruction);
// Example response: "0.01"
`

$3

`ts
import { getBoundingBox } from "@empiricalrun/llm/vision/bbox";

// With Appium
const data = await driver.saveScreenshot("dummy.png");
// Give a line describing the screen and then the element that you want to find
const instruction =
"This screenshot shows a screen to send crypto tokens. What is the bounding box for the dropdown to select the token?";

const bbox = await getBoundingBox(data.toString("base64"), instruction);
const centerToTap = bbox.center; // { x: 342, y: 450 }

// Note: These coordinates are relative to the image dimensions, and actions like
// tap require scaling the coordinates to Appium coordinates
`

Bounding box can require some prompt iterations, and you can do that with a debug flag. This flag
returns a base64 image that has the bounding box drawn on top of the original image.

`ts
const bbox = await getBoundingBox(data.toString("base64"), instruction, {
debug: true,
});
console.log(bbox.annotatedImage);
``