llmverify

Stop shipping unsafe AI output. Local-first verification, safety monitoring, and guardrails for any LLM — in one npm install.


100% Local | No Telemetry | Privacy-First | 500 free calls/day

---

The Problem

You're building with GPT-4, Claude, Gemini, or any LLM. Your AI:

- Hallucinates facts and citations that don't exist
- Leaks PII — emails, phone numbers, SSNs in responses
- Gets prompt-injected — users trick it into ignoring instructions
- Returns broken JSON that crashes your parser
- Drifts in quality over time without anyone noticing

You need a safety layer between your LLM and your users. That's llmverify.

What You Get

``
npm install llmverify
`

| Feature | One-liner | What it catches |
|---------|-----------|-----------------|
| Verify output | await verify(aiResponse) | Hallucinations, harmful content, quality issues |
| Block injections | isInputSafe(userMessage) | Prompt injection, jailbreaks, malicious input |
| Redact PII | redactPII(text) | Emails, phones, SSNs, credit cards, API keys |
| Fix broken JSON | repairJSON(broken) | Missing brackets, trailing commas, unquoted keys |
| Monitor health | monitorLLM(client) | Latency spikes, token drift, behavioral changes |
| Sentinel tests | sentinel.quick(client, model) | LLM regressions before users see them |
| Classify output | classify(prompt, response) | Intent detection, content categorization |
| Audit trail | audit.log(event) | SOC2, HIPAA, GDPR compliance evidence |

Everything runs locally. No data leaves your machine. No API keys needed for free tier.

$3

| Category | Feature | Function | What it does |
|----------|---------|----------|-------------|
| Safety | Prompt injection defense | isInputSafe() | Block jailbreaks, instruction override, data exfiltration |
| Safety | PII detection & redaction | redactPII() / containsPII() | Strip emails, phones, SSNs, credit cards, API keys |
| Safety | Harmful content filtering | checkHarmfulContent() | Flag toxic, violent, or inappropriate responses |
| Safety | Injection risk scoring | getInjectionRiskScore() | 0–1 numerical severity score |
| Quality | Hallucination risk | calculateHallucinationRisk() | Detect hedging, self-contradiction, unsupported claims |
| Quality | JSON validation & repair | detectAndRepairJson() | Fix missing brackets, trailing commas, unquoted keys |
| Quality | Output classification | classify(prompt, output) | Categorize by intent, topic, compliance risk |
| Quality | Consistency analysis | ConsistencyEngine | Detect divergent answers to the same question |
| DevEx | One-line verify | verify(text) | Full risk assessment in one call |
| DevEx | Zod-like guard API | guard() / safe() / parse() | Familiar patterns — {ok, data, risk} |
| DevEx | 9 LLM adapters | createAdapter({provider}) | OpenAI, Anthropic, Groq, Google, DeepSeek, Mistral, Cohere, local, custom |
| DevEx | Plugin system | use(createPlugin()) | Add custom verification rules |
| DevEx | CLI presets | --preset dev\|prod\|strict | One command for any environment |
| Ops | Runtime health monitor | monitorLLM(client) | Track latency, token drift, behavioral changes |
| Ops | Sentinel regression tests | runAllSentinelTests() | 4 proactive tests to catch LLM regressions |
| Ops | Audit logging | AuditLogger | Local-only trail for SOC2/HIPAA/GDPR evidence |
| Ops | Baseline drift detection | BaselineStorage | Alert when LLM behavior changes from baseline |
| Privacy | 100% local processing | — | Zero network requests, verifiable with tcpdump |
| Privacy | No telemetry | — | No tracking, no phone-home, no data collection |

> All detection is 100% deterministic — regex + heuristics, zero ML, zero AI inference. Same input = same output, always.

> Part of the HAIEC AI governance platform. Use alongside AI Security Scanner, CI/CD Pipeline, and Runtime Injection Testing.

---

Install

`bash
npm install llmverify
`

---

Quick Start (30 seconds)

`javascript
const { verify, isInputSafe, redactPII } = require('llmverify');

// 1. Check user input before sending to AI
if (!isInputSafe(userMessage)) {
return { error: 'Invalid input detected' };
}

// 2. Get AI response (your existing code)
const aiResponse = await yourLLM.generate(userMessage);

// 3. Verify the output before showing to users
const result = await verify(aiResponse);

if (result.risk.level === 'critical') {
return { error: 'Response failed safety check' };
}

// 4. Redact any PII before logging
const { redacted } = redactPII(aiResponse);
console.log(redacted); // "Contact [REDACTED] at [REDACTED]"
`

That's it. Three lines of safety between your LLM and your users.

---

For IDE Users (Windsurf, VS Code, Cursor)

llmverify includes a REST server and clipboard monitor for real-time verification inside your IDE.

`bash

Terminal 1: Start verification server

npm run serve

Terminal 2: Start clipboard monitor

npm run monitor

Now copy any AI response → see risk scores automatically

`

Port conflict? Run npm run serve:force

You can also use the programmatic IDE extension:

`javascript
import { LLMVerifyIDE, createIDEExtension } from 'llmverify';

const ide = new LLMVerifyIDE({ useLocalFallback: true });
const result = await ide.verify(clipboardText);
`

Full IDE Setup Guide →

---

Core Functions

$3

Checks AI responses for safety, accuracy, and quality before showing to users.

`javascript
const { verify } = require('llmverify');

// Object form
const result = await verify({ content: aiResponse });

// String shorthand (same thing)
const result = await verify(aiResponse);

console.log(result.risk.level); // "low", "moderate", "high", "critical"
console.log(result.risk.overall); // 0.172 (17.2% risk)
console.log(result.limitations); // What wasn't checked (transparency)
`

Risk Levels:
- LOW (0-25%): Safe to use
- MODERATE (26-50%): Review recommended
- HIGH (51-75%): Fix before using
- CRITICAL (76-100%): Do not use

Risk Levels Guide →

---

$3

What it does: Validates user input before sending to AI
When to use: On every user message
Benefit: Prevent prompt injection attacks

`javascript
const { isInputSafe } = require('llmverify');

if (!isInputSafe(userMessage)) {
return { error: 'Invalid input detected' };
}

// Safe to send to AI
const aiResponse = await callAI(userMessage);
`

What it checks:
- Prompt injection patterns
- Malicious commands
- Suspicious formatting
- Excessive length

---

$3

What it does: Redacts personal information from text
When to use: Before logging or storing AI responses
Benefit: GDPR/CCPA compliance, privacy protection

`javascript
const { redactPII } = require('llmverify');

const { redacted, piiCount } = redactPII(aiResponse);

console.log(redacted); // "Contact us at [REDACTED]"
console.log(piiCount); // 2
`

Detects:
- Email addresses
- Phone numbers
- Social Security Numbers
- Credit card numbers
- API keys

---

$3

What it does: Repairs broken JSON from AI responses
When to use: When AI returns invalid JSON
Benefit: Avoid parsing errors, improve reliability

`javascript
const { repairJSON } = require('llmverify');

const brokenJSON = '{"name": "John", "age": 30'; // Missing }
const fixed = repairJSON(brokenJSON);

console.log(fixed); // {"name": "John", "age": 30}
`

Fixes:
- Missing brackets/braces
- Trailing commas
- Unquoted keys
- Escape issues

---

$3

What it does: Categorizes AI responses by intent/type
When to use: Routing, filtering, or organizing responses
Benefit: Automated content organization

`javascript
const { classify } = require('llmverify');

const result = classify('Summarize this article', aiResponse);

console.log(result.intent); // "question", "answer", "code", "explanation"
console.log(result.confidence); // 0.85
`

Categories:
- Question
- Answer
- Code
- Explanation
- Instruction
- Creative

---

$3

What it does: Tracks AI system health and performance
When to use: Production monitoring
Benefit: Detect issues before users do

`javascript
const { monitor } = require('llmverify');

monitor.track({
model: 'gpt-4',
latency: 1200,
tokens: 500,
cost: 0.02
});

const health = monitor.getHealth();
console.log(health.status); // "healthy", "degraded", "critical"
`

Monitors:
- Response latency
- Token usage
- Error rates
- Cost tracking
- Behavioral drift

---

$3

What it does: Creates compliance-ready audit trails
When to use: Regulated industries, compliance requirements
Benefit: Meet SOC2, HIPAA, GDPR requirements

`javascript
const { audit } = require('llmverify');

audit.log({
action: 'ai_response_verified',
user: 'user_123',
riskLevel: 'low',
timestamp: new Date()
});

const logs = audit.query({ user: 'user_123', limit: 10 });
`

Captures:
- All verifications
- Risk scores
- User actions
- Timestamps
- Findings

---

$3

What it does: Creates verification badges for your content
When to use: Show users content is verified
Benefit: Build trust, demonstrate safety

`javascript
const { generateBadge } = require('llmverify');

const badge = generateBadge({
riskLevel: 'low',
score: 17.2,
verified: true
});

// Use in HTML
//
`

Badge Types:
- Verification status
- Risk level
- Score display
- Custom branding

---

Server Mode (REST API)

What it does: Run llmverify as a standalone service
When to use: IDE integration, microservices, external tools
Benefit: Language-agnostic, easy integration

`bash

Start server

npm run serve

Server runs on http://localhost:9009

`

Endpoints:

`bash

Health check

GET /health

Verify content

POST /verify
Body: {"content": "AI response text"}

Check input

POST /check-input
Body: {"input": "User message"}

Detect PII

POST /check-pii
Body: {"content": "Text with PII"}

Classify

POST /classify
Body: {"content": "AI response"}
`

Example:
`javascript
const response = await fetch('http://localhost:9009/verify', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ content: aiResponse })
});

const result = await response.json();
`

Server API Reference →

---

CLI Tools

What it does: Verify content from command line
When to use: Scripts, automation, testing
Benefit: No code required

`bash

Verify a file

llmverify verify response.txt

Check input

llmverify check-input "User message"

Redact PII

llmverify redact-pii document.txt

System health

llmverify doctor
`

CLI Reference →

---

Integration Examples

$3

`javascript
const OpenAI = require('openai');
const { verify, isInputSafe } = require('llmverify');

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

async function safeChat(userMessage) {
// 1. Validate input
if (!isInputSafe(userMessage)) {
throw new Error('Invalid input');
}

// 2. Get AI response
const completion = await openai.chat.completions.create({
model: "gpt-4",
messages: [{ role: "user", content: userMessage }]
});

const aiResponse = completion.choices[0].message.content;

// 3. Verify output
const verification = await verify(aiResponse);

if (verification.risk.level === 'critical') {
throw new Error('Response failed safety check');
}

return {
response: aiResponse,
risk: verification.risk
};
}
`

$3

`javascript
const express = require('express');
const { verify, isInputSafe, redactPII } = require('llmverify');

const app = express();

app.post('/chat', async (req, res) => {
const { message } = req.body;

// Validate input
if (!isInputSafe(message)) {
return res.status(400).json({ error: 'Invalid input' });
}

// Get AI response
const aiResponse = await callAI(message);

// Verify output
const verification = await verify(aiResponse);

// Redact PII
const { redacted } = redactPII(aiResponse);

// Block critical risk
if (verification.risk.level === 'critical') {
return res.status(400).json({ error: 'Response failed safety check' });
}

res.json({
response: redacted,
riskScore: verification.risk.overall,
riskLevel: verification.risk.level
});
});
`

$3

`javascript
const { verify } = require('llmverify');

async function streamWithVerification(userMessage) {
let fullResponse = '';

// Stream AI response
const stream = await openai.chat.completions.create({
model: 'gpt-4',
messages: [{ role: 'user', content: userMessage }],
stream: true
});

for await (const chunk of stream) {
const content = chunk.choices[0]?.delta?.content || '';
fullResponse += content;
process.stdout.write(content); // Stream to user
}

// Verify after streaming completes
const verification = await verify(fullResponse);

if (verification.risk.level === 'high' ||
verification.risk.level === 'critical') {
console.log('\n[WARNING] Response has elevated risk');
console.log(Risk: ${verification.risk.overall * 100}%);
}

return { fullResponse, verification };
}
`

---

Configuration

Create .llmverify.json in your project:

`json
{
"ignoreFindings": ["injection-marker"],
"context": "ai-agent-development",
"whitelist": {
"pii": ["example.com", "test.com"]
},
"strictMode": false,
"logLevel": "info"
}
`

Configuration Guide →

---

Pricing & Tiers

All features available on every tier. Usage tracked locally — no data leaves your machine.

| Tier | Price | Daily Calls | Content Limit | Support |
|------|-------|-------------|---------------|---------|
| Free | $0 | 500/day | 50KB | Community |
| Starter | $79/mo | 5,000/day | 200KB | Email (72hr) |
| Pro | $299/mo | 50,000/day | 1MB | Email (24hr) |
| Business | $999/mo | Unlimited | Unlimited | Slack (4hr SLA) |

`bash

Check your usage anytime

npx llmverify usage

See current tier and limits

npx llmverify tier
`

All calls share a single daily pool: verify, guard, safe, parse, sentinel, monitor.
Usage resets at midnight local time. A 10% grace period lets you finish work without hard cutoffs.

---

Sentinel Tests

Proactive behavioral tests that verify your LLM is responding correctly — run in CI/CD, health checks, or before deployment.

`javascript
const { sentinel } = require('llmverify');

// One-liner — runs all 4 sentinel tests
const suite = await sentinel.quick(myClient, 'gpt-4');
console.log(suite.passed); // true/false
console.log(suite.passRate); // 0.75 = 3/4 passed

// Run a single test
const echo = await sentinel.test('staticEchoTest', myClient, 'gpt-4');
`

Built-in tests:
- Static Echo — Does the LLM echo back exact content?
- Duplicate Query — Are responses consistent across identical prompts?
- Structured List — Can it follow output format instructions?
- Short Reasoning — Does it show step-by-step reasoning?

---

Health Monitoring

Wraps any LLM client with real-time health tracking. Detects latency spikes, token drift, and behavioral changes automatically.

`javascript
const { monitorLLM } = require('llmverify');

const monitored = monitorLLM(openaiClient, {
hooks: {
onUnstable: (report) => alert('LLM unstable!'),
onDegraded: (report) => console.warn('Degraded'),
onRecovery: (report) => console.log('Recovered')
}
});

const response = await monitored.generate({ prompt: 'Hello' });
console.log(response.llmverify.health); // 'stable' | 'degraded' | 'unstable'
`

---

LLM Provider Adapters

Unified interface for any LLM provider. Swap providers without changing your verification code.

`javascript
const { createAdapter, monitorLLM } = require('llmverify');

// Works with OpenAI, Anthropic, Groq, Google, DeepSeek, Mistral, Cohere, local models
const client = createAdapter({ provider: 'openai', apiKey: process.env.OPENAI_API_KEY });
const monitored = monitorLLM(client);

const response = await monitored.generate({ prompt: 'Hello', model: 'gpt-4' });
`

---

Plugin System

Extend llmverify with custom verification rules.

`javascript
const { use, createPlugin, createRegexPlugin } = require('llmverify');

// Built-in plugin helpers
use(createRegexPlugin('no-urls', /https?:\/\/\S+/g, 'URL detected in output'));

// Custom plugin
use(createPlugin('custom-check', async (ctx) => {
const hasBadWord = ctx.content.includes('forbidden');
return { passed: !hasBadWord, message: hasBadWord ? 'Forbidden word found' : 'Clean' };
}));
`

---

Error Handling

Every error includes a code, severity, recoverability flag, and a human-readable suggestion.

`javascript
const { verify, LLMVerifyError } = require('llmverify');

try {
const result = await verify(aiResponse);
} catch (err) {
if (err instanceof LLMVerifyError) {
console.log(err.code); // "LLMVERIFY_1003"
console.log(err.metadata.severity); // "medium"
console.log(err.metadata.recoverable); // true
console.log(err.metadata.suggestion); // "Reduce content size or split into smaller chunks"
}
}
`

Error code ranges:
- 1xxx — Input validation (empty, too large, bad format)
- 2xxx — Configuration (malformed config, invalid tier)
- 3xxx — Runtime (timeout, engine failure)
- 4xxx — Server (port in use, rate limit)
- 5xxx — Privacy/security (privacy violation, unauthorized)
- 6xxx — Plugins (not found, load failed)
- 7xxx — Usage/tier (daily limit, content length)

All errors are typed — use instanceof checks or the code property for programmatic handling.

---

DX Shortcuts

`javascript
// String shorthand — no object wrapper needed
const result = await verify("The Earth is flat.");

// Default export
import llmverify from 'llmverify';
const result = await llmverify.verify(text);

// Shorthand APIs
import { ai, llm, guardrails } from 'llmverify';
const safe = await ai.safe(text); // returns text or null
const guard = await ai.guard(text); // returns { ok, risk, findings }
const score = ai.riskScore(text); // injection risk 0-1
`

---

Performance

- Verification: <100ms for typical responses
- PII Detection: <50ms
- Input Validation: <10ms
- Memory: <50MB baseline
- Bundle: No heavy ML models — pattern-based, runs anywhere Node runs

---

Requirements

- Node.js: ≥18.0.0
- OS: Windows, Linux, macOS
- Memory: 512MB minimum
- Disk: 50MB

---

Quick Reference

| Task | Function | Example |
|------|----------|---------|
| Verify AI output | verify() | await verify(response) |
| Verify (string) | verify() | await verify("text") |
| Check input | isInputSafe() | isInputSafe(message) |
| Remove PII | redactPII() | redactPII(text) |
| Fix JSON | repairJSON() | repairJSON(broken) |
| Classify | classify() | classify(prompt, response) |
| Monitor health | monitorLLM() | monitorLLM(client) |
| Sentinel tests | sentinel.quick() | await sentinel.quick(client, model) |
| Guard (Zod-like) | guard() | await guard(text) |
| Safe check | safe() | await safe(text) |
| Plugin | use() | use(createRegexPlugin(...)) |
| Check usage | CLI | npx llmverify usage |
| Audit | audit.log() | audit.log(event) |
| Badge | generateBadge() | generateBadge(result) |
| Adapter | createAdapter() | createAdapter({ provider: 'openai', apiKey })` |

---

Documentation

- Quick Start Guide — IDE setup in 5 minutes
- Integration Guide — Add to your app
- API Reference — Complete API docs
- Risk Levels — What scores mean
- Error Guide — Fix common errors
- CLI Reference — Command line tools
- AI Guide — Help AI assistants use llmverify

---

Support

- Issues: GitHub Issues
- Docs: Full Documentation
- Examples: /examples

---

License

MIT License — See LICENSE for details.

---

Made with care for AI safety and developer experience.