ai-index

🤖 ai-index v3.0.0 - Local AI-powered code indexing and search

AI-powered local code indexing and search system with efficient reindexing. Enables semantic code search across any codebase using local embeddings and vector storage, with smart change detection to only reprocess modified files.

Features

- 🔍 Semantic Search: Find code using natural language queries with local embeddings
- ⚡ Efficient Reindexing: Smart change detection - only reprocesses modified files using SHA-256 hashing
- 🗑️ Automatic Cleanup: Removes outdated chunks and handles deleted files automatically
- 🎯 Hybrid Search: Combines lexical and vector search for optimal results
- 📦 Local Storage: No cloud dependencies - uses local vector database (Vectra)
- 🏷️ Area Filtering: Filter by codebase areas (backend, frontend, docs, infra)
- 🔧 Force Reindex: Option to reprocess all files when needed
- 📚 Programmatic API: Use as a library in your applications

Installation

$3

``bash
npm install -g ai-index
`

$3

`bash
npm install ai-index
`

Prerequisites

- Node.js 18+: Required for running the local embedding models
- Local Storage: ~100MB for embedding models and vector data (per project)

Configuration

The system works out of the box with local defaults. Optional configuration via ~/.ai-index/config.json:

`json
{
"MODE": "local",
"DATA_PATH": "~/.ai-index/data",
"EMBED_MODEL": "Xenova/all-MiniLM-L6-v2"
}
`

$3


- Embedding Model: Xenova/all-MiniLM-L6-v2 (384 dimensions, ~90MB download)
- Storage: ~/.ai-index/data/ (vector databases stored per project)
- Hash Tracking: ai_index/file_hashes.json (tracks file changes for efficient reindexing)

Usage

$3

#### 1. Index Your Codebase

`bash


Index current folder

ai-index index

Index specific directory

ai-index index /path/to/your/project

Force full reindex (reprocess all files)

ai-index index --force
`

Efficient Reindexing Performance:
- First run: Processes all files (~30 seconds for medium codebase)
- Subsequent runs: Only changed files (~2-5 seconds, 80-95% faster)
- Automatic cleanup of deleted files and outdated chunks

#### 2. Search Your Code

`bash


Basic search

ai-index query --q "user authentication logic"

Compact format (optimized for AI consumption)

ai-index query --q "database models" --compact

Filter by area and relevance

ai-index query --q "error handling" --area backend --min-score 2.0 --k 5

Available options:

--q "query" Natural language search query (required)

--k 20 Number of results to return

--area backend Filter by area (backend|frontend|infra|docs)

--min-score 1.5 Minimum relevance score threshold

--compact Return compact format for AI consumption

`

$3

`javascript
import { CodeSearcher } from 'ai-index'

// Local mode - works out of the box
const searcher = new CodeSearcher({
mode: 'local',
indexName: 'my_project' // optional, defaults to folder name
})

// Basic search
const results = await searcher.search("user authentication", {
k: 10,
compact: true
})

// Advanced search with filtering
const filteredResults = await searcher.search("database models", {
area: 'backend',
minScore: 2.0,
k: 5
})
`

$3

`javascript
import { buildIndex } from 'ai-index'

// Smart reindexing (only changed files)
await buildIndex('/path/to/project')

// Force full reindex
await buildIndex('/path/to/project', { force: true })
`

Output Formats

$3

Optimized for AI system consumption:

`json
{
"query": "authentication middleware",
"results": [
{
"path": "app/middleware/auth.js",
"snippets": ["15-45", "67-89"]
}
]
}
`

$3

Includes relevance scores and detailed metadata:

`json
{
"query": "authentication middleware",
"area": "backend",
"total_results": 3,
"files": [
{
"path": "app/middleware/auth.js",
"area": "backend",
"score": 15.7,
"snippets": [
{"start": 15, "end": 45, "score": 15.7},
{"start": 67, "end": 89, "score": 12.3}
]
}
]
}
`

File Area Classification

Files are automatically categorized:
- backend: API routes, models, helpers, workers
- frontend: Components, pages, client-side code
- infra: Terraform, Kubernetes, Docker configs
- docs: Markdown documentation

How Efficient Reindexing Works

1. Hash Tracking: Each file's SHA-256 content hash is stored in ai_index/file_hashes.json
2. Change Detection: On reindex, compares current file hashes with stored hashes
3. Selective Processing: Only files with changed hashes are reprocessed
4. Chunk Cleanup: Old vector embeddings are removed before adding new ones
5. Deleted Files: Automatically removes chunks for files that no longer exist
6. Performance: Typical 80-95% reduction in processing time after initial index

Security

This package:
- ✅ Runs completely locally - no data sent to external services
- ✅ Uses local embedding models (Transformers.js)
- ✅ Excludes all .env files and secrets from indexing
- ✅ Stores vector data locally in ~/.ai-index/data/
- ✅ No hardcoded credentials or external dependencies

Development

`bash


Clone and install dependencies

git clone https://github.com/homevote/ai-index.git
cd ai-index
npm install

Test locally (works out of the box)

ai-index index # Index current directory
ai-index query --q "test query"

Test efficient reindexing

ai-index index # First run - processes all files
ai-index index # Second run - only changed files
ai-index index --force # Force reindex all files
`

Performance Benchmarks

Tested on a TypeScript codebase with 1,200 files:

| Operation | First Run | Subsequent Runs | Savings |
|-----------|-----------|-----------------|---------|
| File Processing | 1,200 files | 15-50 files | 92-96% |
| Time | 45 seconds | 3-8 seconds | 82-93% |
| Embeddings | 3,200 chunks | 45-180 chunks | 91-98% |

Performance varies based on code change frequency

License

MIT

Contributing

1. Fork the repository
2. Create your feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add some amazing feature')
4. Push to the branch (git push origin feature/amazing-feature`)
5. Open a Pull Request

Support

- 🐛 Issues: GitHub Issues
- 📧 Email: dev@homevote.at
- 📖 Docs: GitHub Repository