evaluhealth

!Evaluhealth: the TypeScript-native, local-first tool for testing LLM-powered apps.

> By Kernelius — Building AI tools for healthcare and life sciences.

What Is Evaluhealth?

Evaluhealth is a fork of Evalite specifically designed for healthcare and life sciences applications. It's a TypeScript-native, local-first evaluation framework for testing LLM-powered healthcare applications, medical AI systems, and life science research tools.

$3

- Fully open source: No API Key required.
- Local-first: runs on your machine, your data never leaves your laptop.
- Based on Vitest, the best TypeScript test runner around.
- Terminal UI for quick prototyping.
- Supports tracing and custom scorers.

$3

Evaluhealth is tailored for developers and researchers working in healthcare and life sciences:

- Medical AI Evaluation: Test clinical decision support systems, diagnostic tools, and medical chatbots with comprehensive evaluation metrics
- Life Sciences Research: Evaluate LLM performance on scientific literature analysis, drug discovery pipelines, and biomedical data processing
- HIPAA-Compliant Testing: Local-first architecture ensures sensitive healthcare data (PHI) never leaves your secure environment
- Regulatory Readiness: Build evaluation frameworks that support FDA submissions, CE marking, and other regulatory compliance requirements
- Clinical Validation: Create robust test suites for validating AI models in real-world healthcare contexts with domain-specific scorers

How Do I Learn More?

- Read the Docs
- Join the Discord

Evaluhealth Is Experimental

Evaluhealth is still an experimental project. I'm actively working on it, and for now am pushing breaking changes.

If you run into any unexpected behavior:

1. Delete the node_modules/.evalite folder.
2. Update evaluhealth to the latest version.
3. Rerun your evals.

If, after that, you run into unexpected behavior, report an issue.

Guides

$3

You can run Evaluhealth in watch mode by running evaluhealth watch:

``bash
evaluhealth watch
`

This will watch for changes to your .eval.ts files and re-run the evals when they change.

> [!IMPORTANT]
>
> I strongly recommend implementing a caching layer in your LLM calls when using watch mode. This will keep your evals running fast and avoid burning through your API credits.

$3

You can run specific files by passing them as arguments:

`bash
evaluhealth my-eval.eval.ts
`

This also works for watch mode:

`bash
evaluhealth watch my-eval.eval.ts
``