Doc-driven testing validation - ensure tests reference documentation and those references are valid
npm install @fieldtest/doc-ref> Doc-driven testing validation - ensure tests reference documentation and those references are valid.
Documentation and tests often drift apart. Tests claim to validate behavior described in docs, but:
- The doc file was renamed or deleted
- The referenced line numbers changed after edits
- The section heading was reworded
@fieldtest/doc-ref scans your test files for documentation references and validates they point to real, existing content.
``bash`
npm install @fieldtest/doc-refor
pnpm add @fieldtest/doc-ref
This tool supports two complementary workflows:
`typescript
import { getSections, parseMarkdownFile } from '@fieldtest/doc-ref';
// Check that install guide has platform sections
const sections = getSections('docs/install.md');
const titles = sections.map(s => s.title.toLowerCase());
expect(titles.some(t => t.includes('linux'))).toBe(true);
`
`typescript
import { generateReport } from '@fieldtest/doc-ref';
// Validate all DOC: references in test files point to real docs
const report = await generateReport('./my-project');
if (report.invalidRefs > 0) {
console.error(${report.invalidRefs} broken doc references);`
}
`typescript
import { generateReport, formatReport } from '@fieldtest/doc-ref';
const report = await generateReport('./my-project');
console.log(formatReport(report));
`
Add references in your test files using these patterns:
`javascript
// DOC: comment (recommended - most explicit)
/**
* DOC: docs/public/reference/api.md
* Section: Rate Limits
*/
describe("Rate Limits", () => {
// ...
});
// Parenthesized path in test descriptions
describe("Rate Limits (docs/public/reference/api.md)", () => {
// ...
});
// Line reference
it("returns 1 for identical vectors (reference.md:207)", () => {
// ...
});
// Line range
describe("Feature A (docs/reference.md:206-209)", () => {
// ...
});
// Section anchor
describe("TF-IDF behavior (explainer.md#how-tfidf-works)", () => {
// ...
});
`
| Pattern | Example | Use Case |
|---------|---------|----------|
| DOC: comment | DOC: docs/api.md | Explicit doc reference in JSDoc |(docs/api.md)
| Parenthesized path | | In describe/it strings |api.md:207
| Line reference | | Link to specific line |api.md:206-209
| Line range | | Link to line range |api.md#auth
| Anchor reference | | Link to heading/section |
All patterns support nested directory paths:
- docs/public/reference/api.mddocs/guides/setup.md#installation
- docs/public/api.md:50-75
-
`typescript
// Markdown parsing - these take file paths, not ParsedDoc objects
function parseMarkdownFile(filePath: string): ParsedDoc
function parseMarkdown(content: string): ParsedDoc // For raw strings
function getSections(filePath: string): DocSection[] // Convenience wrapper
function getSection(filePath: string, anchor: string): DocSection | undefined
function hasAnchor(filePath: string, anchor: string): boolean
// Report generation
function generateReport(projectDir: string, options?: DocRefOptions): Promise
function formatReport(report: ValidationReport): string
// Low-level scanning
function scanDocReferences(projectDir: string, options?: DocRefOptions): Promise
function validateReferences(refs: DocReference[], projectDir: string): ValidationResult[]
`
`typescript
interface ParsedDoc {
sections: Map
anchors: string[]; // All section slugs
lineCount: number;
}
interface DocSection {
title: string; // Original heading text
slug: string; // URL-friendly anchor
level: number; // 1-6 for markdown, 0 for HTML id
line: number; // Line number in file
content: string; // Section text (excluding heading)
examples: CodeExample[];
assertions: DocAssertion[];
}
`
`typescript
import { getSections } from '@fieldtest/doc-ref';
const sections = getSections('docs/api.md');
expect(sections.some(s => s.title === 'Authentication')).toBe(true);
`
`typescript
import { getSections } from '@fieldtest/doc-ref';
const sections = getSections('docs/install.md');
const titles = sections.map(s => s.title.toLowerCase());
expect(titles.some(t => t.includes('linux'))).toBe(true);
expect(titles.some(t => t.includes('macos'))).toBe(true);
expect(titles.some(t => t.includes('windows'))).toBe(true);
`
`typescript
import { parseMarkdownFile } from '@fieldtest/doc-ref';
import * as fs from 'fs';
import * as path from 'path';
const docsDir = 'docs';
const files = fs.readdirSync(docsDir).filter(f => f.endsWith('.md'));
for (const file of files) {
const content = fs.readFileSync(path.join(docsDir, file), 'utf-8');
const linkRegex = /\[([^\]]+)\]\(([^)]+)\)/g;
let match;
while ((match = linkRegex.exec(content)) !== null) {
const [, , linkPath] = match;
if (linkPath.startsWith('/') || linkPath.startsWith('./')) {
const resolved = path.resolve(docsDir, linkPath.replace(/^\//, ''));
expect(fs.existsSync(resolved)).toBe(true);
}
}
}
`
`typescript
import { getSections } from '@fieldtest/doc-ref';
const sections = getSections('docs/api.md');
const authSection = sections.find(s => s.slug === 'authentication');
// Verify examples exist
expect(authSection?.examples.length).toBeGreaterThan(0);
// Get the first code example
const example = authSection?.examples[0];
expect(example?.lang).toBe('typescript');
`
`typescript`
// DOC: docs/api.md#rate-limits
describe('Rate Limits', () => {
it('enforces 100 requests per minute', () => {
// Test implementation tied to documented behavior
});
});
Generate a full validation report for a project.
`typescript`
const report = await generateReport('./my-project', {
testPatterns: ['*/.test.ts'],
docsDir: 'docs',
validateLines: true,
validateAnchors: true,
});
Returns a ValidationReport:
`typescript`
interface ValidationReport {
projectDir: string;
totalRefs: number;
validRefs: number;
invalidRefs: number;
results: ValidationResult[];
coverage: DocCoverage[];
orphanedDocs: string[];
testsWithoutRefs: string[];
}
Scan test files for doc references without validating.
`typescript`
const refs = await scanDocReferences('./my-project');
// Returns: DocReference[]
Validate an array of doc references.
`typescript`
const results = validateReferences(refs, './my-project');
// Returns: ValidationResult[]
Format a report as a human-readable string.
`typescript`
console.log(formatReport(report));
`typescript
interface DocRefOptions {
// Glob patterns for test files
testPatterns?: string[]; // default: ["/.test.{js,ts}", "/.spec.{js,ts}"]
// Directory containing docs
docsDir?: string; // default: "docs"
// Doc file extensions
docExtensions?: string[]; // default: [".md"]
// Validate line numbers exist
validateLines?: boolean; // default: true
// Validate anchor references
validateAnchors?: boolean; // default: true
}
`
`
=== Doc-Test Validation Report ===
Project: /path/to/my-project
Total References: 22
Valid: 22
Invalid: 0
Doc Coverage:
reference.md: 20 refs (covered)
explainer.md: 2 refs (covered)
tutorial.md: 0 refs (orphan)
Orphaned Docs (no test references): 1
Tests Without Doc References: 3
`
Add to your CI pipeline to catch doc-test drift:
`typescript
import { generateReport } from '@fieldtest/doc-ref';
const report = await generateReport('.');
if (report.invalidRefs > 0) {
console.error(Found ${report.invalidRefs} invalid doc references);
process.exit(1);
}
console.log(All ${report.validRefs} doc references are valid);``
1. Living Documentation - Docs stay accurate because tests enforce them
2. Traceability - Every test traces back to a documented behavior
3. Coverage Visibility - See which docs have test coverage
4. Drift Detection - Catch when docs and tests diverge
MIT