Zero-dependency, memory-efficient recursive directory walker using Async Iterators.
npm install fstream-walkfs.opendir and Async Iterators (Generators) to handle huge directory trees without bloating RAM.
AbortSignal to cancel long-running scans.
include/exclude using Strings, Regex, or Functions.
bash
npm install fstream-walk
`
Quick Start
`javascript
import walker from 'fstream-walk';
// Basic usage - walk all files recursively
for await (const file of walker('./src')) {
console.log(file.path);
}
// With filters
for await (const file of walker('./src', {
include: /\.js$/, // Only .js files
exclude: /node_modules/, // Ignore node_modules
maxDepth: 3 // Max 3 levels deep
})) {
console.log(file.path);
}
`
API
$3
Returns an AsyncGenerator that yields file/directory entries.
#### WalkerEntry Object
Each yielded entry has the following structure:
`typescript
{
path: string; // Full path to the file/directory
dirent: fs.Dirent; // Directory entry with type info
depth: number; // Current depth level (0 = root)
stats?: fs.Stats; // File stats (if withStats: true)
}
`
#### Options
| Option | Type | Default | Description |
| :--- | :--- | :--- | :--- |
| maxDepth | number | Infinity | Maximum depth to recurse into subdirectories. |
| include | String\|Regex\|Fn | null | Filter to include files. Can be string, regex, or function. |
| exclude | String\|Regex\|Fn | null | Filter to exclude files. Can be string, regex, or function. |
| yieldDirectories | boolean | false | Whether to yield directory paths in addition to files. |
| followSymlinks | boolean | false | Follow symbolic links (may cause infinite loops). |
| suppressErrors | boolean | true | Suppress permission errors (EACCES/EPERM). |
| signal | AbortSignal | null | AbortSignal to cancel the operation. |
| sort | 'asc'\|'desc'\|Fn | null | Sort entries alphabetically or with custom function. |
| onProgress | Function | null | Callback function called for each entry (for progress tracking). |
| withStats | boolean | false | Include fs.Stats object in entries (adds size, timestamps, etc.). |
Usage Examples
$3
`javascript
import walker from 'fstream-walk';
// Find all JavaScript files
for await (const file of walker('./src', { include: /\.js$/ })) {
console.log(file.path);
}
// Exclude test files
for await (const file of walker('./src', {
exclude: /\.(test|spec)\.js$/
})) {
console.log(file.path);
}
// Custom filter function
for await (const file of walker('./src', {
include: (name) => name.startsWith('component-')
})) {
console.log(file.path);
}
`
$3
`javascript
// Sort files alphabetically (A-Z)
for await (const file of walker('./src', { sort: 'asc' })) {
console.log(file.dirent.name);
}
// Sort in reverse (Z-A)
for await (const file of walker('./src', { sort: 'desc' })) {
console.log(file.dirent.name);
}
// Custom sorting - directories first, then files
for await (const entry of walker('.', {
yieldDirectories: true,
sort: (a, b) => {
if (a.isDirectory() && !b.isDirectory()) return -1;
if (!a.isDirectory() && b.isDirectory()) return 1;
return a.name.localeCompare(b.name);
}
})) {
console.log(entry.path);
}
`
$3
`javascript
let fileCount = 0;
let totalSize = 0;
for await (const file of walker('./src', {
withStats: true,
onProgress: (entry) => {
fileCount++;
if (entry.stats) {
totalSize += entry.stats.size;
}
if (fileCount % 100 === 0) {
console.log(Processed ${fileCount} files...);
}
}
})) {
// Process files
}
console.log(Total: ${fileCount} files, ${totalSize} bytes);
`
$3
`javascript
const controller = new AbortController();
// Cancel after 5 seconds
setTimeout(() => controller.abort(), 5000);
try {
for await (const file of walker('./large-dir', {
signal: controller.signal
})) {
console.log(file.path);
}
} catch (err) {
if (err.name === 'AbortError') {
console.log('Operation cancelled');
}
}
`
$3
`javascript
// Get file sizes and timestamps
for await (const file of walker('./src', { withStats: true })) {
if (file.stats) {
const sizeKB = (file.stats.size / 1024).toFixed(2);
const modified = file.stats.mtime.toISOString();
console.log(${file.path}: ${sizeKB} KB (modified: ${modified}));
}
}
`
Advanced Examples
Check out the examples directory for more advanced usage patterns:
- demo.js - Basic usage examples
- filtering.js - Advanced filtering techniques
- abort-signal.js - Cancellation with AbortSignal
- advanced-usage.js - File counting, size analysis, tree building
- advanced-features.js - Sorting, progress, and stats
Performance
Run benchmarks to see how fstream-walk performs:
`bash
Performance benchmarks
node benchmarks/performance.js
Memory usage comparison
node --expose-gc benchmarks/memory.js
`
$3
On a directory with 10,000 files:
- Memory efficient: Uses ~50% less memory than loading all files into an array
- Fast: Processes 10,000+ files in under 100ms
- Streaming: Constant memory usage regardless of directory size
Testing
`bash
Run all tests
npm test
Run specific test suites
node --test test/index.test.js
node --test test/edge-cases.test.js
Watch mode
npm run test:watch
`
TypeScript Support
Full TypeScript definitions are included. No need for @types packages!
`typescript
import walker, { WalkerOptions, WalkerEntry } from 'fstream-walk';
const options: WalkerOptions = {
maxDepth: 3,
include: /\.ts$/,
sort: 'asc'
};
for await (const file: WalkerEntry of walker('./src', options)) {
console.log(file.path);
}
`
Comparison with Other Tools
| Feature | fstream-walk | node-walk | walk-sync |
| :--- | :---: | :---: | :---: |
| Zero Dependencies | ✅ | ❌ | ❌ |
| Async Iterators | ✅ | ❌ | ❌ |
| Memory Efficient | ✅ | ❌ | ❌ |
| AbortSignal Support | ✅ | ❌ | ❌ |
| TypeScript Support | ✅ | ⚠️ | ⚠️ |
| Sorting | ✅ | ❌ | ❌ |
| Progress Callbacks | ✅ | ❌ | ❌ |
Migration Guide
$3
`javascript
// Before (node-walk)
const walk = require('walk');
const walker = walk.walk('./dir');
walker.on('file', (root, fileStats, next) => {
console.log(path.join(root, fileStats.name));
next();
});
// After (fstream-walk)
import walker from 'fstream-walk';
for await (const file of walker('./dir')) {
console.log(file.path);
}
`
$3
`javascript
// Before (walk-sync)
const walkSync = require('walk-sync');
const files = walkSync('./dir');
// After (fstream-walk)
import { findFiles } from 'fstream-walk/helpers';
const files = await findFiles('./dir');
`
$3
`javascript
// Before (Node.js 18.17.0+)
import fs from 'fs/promises';
const files = await fs.readdir('./dir', { recursive: true });
// After (fstream-walk - more control)
import walker from 'fstream-walk';
const files = [];
for await (const file of walker('./dir', {
maxDepth: 5,
include: /\.js$/,
exclude: 'node_modules'
})) {
files.push(file.path);
}
`
Real-World Integration Examples
$3
`javascript
import express from 'express';
import walker from 'fstream-walk';
import { groupByExtension } from 'fstream-walk/helpers';
const app = express();
app.get('/api/files', async (req, res) => {
const files = await groupByExtension('./public');
res.json(files);
});
app.listen(3000);
`
$3
`javascript
import walker from 'fstream-walk';
import { findRecentFiles } from 'fstream-walk/helpers';
async function buildChangedFiles() {
const lastBuild = Date.now() - 60000; // Last minute
const changed = await findRecentFiles('./src', lastBuild, {
include: /\.(js|ts)$/
});
for (const file of changed) {
await compile(file.path);
}
}
`
$3
`javascript
import walker from 'fstream-walk';
import { patterns } from 'fstream-walk/glob';
async function generateDocs() {
const docs = [];
for await (const file of walker('./src', {
include: patterns.javascript
})) {
const content = await fs.readFile(file.path, 'utf-8');
const docComments = extractDocs(content);
docs.push({ file: file.path, docs: docComments });
}
return docs;
}
`
$3
`javascript
import { findEmptyDirectories, calculateSize } from 'fstream-walk/helpers';
async function cleanup() {
// Find and remove empty directories
const emptyDirs = await findEmptyDirectories('./project');
for (const dir of emptyDirs) {
await fs.rmdir(dir);
console.log(Removed empty directory: ${dir});
}
// Report space savings
const size = await calculateSize('./project');
console.log(Project size: ${size.totalSizeMB} MB);
}
`
FAQ
$3
A: fstream-walk is focused on directory walking with zero dependencies, while glob libraries often have many dependencies. If you need simple glob patterns, use our built-in glob module. For complex glob patterns, you can combine fstream-walk with a dedicated glob library.
$3
A: By default, permission errors are suppressed. To handle them:
`javascript
import walker from 'fstream-walk';
import { PermissionError } from 'fstream-walk/errors';
try {
for await (const file of walker('./dir', { suppressErrors: false })) {
console.log(file.path);
}
} catch (err) {
if (err instanceof PermissionError) {
console.log('Access denied to:', err.path);
}
}
`
$3
A: This package uses ES Modules. For CommonJS projects, use dynamic import:
`javascript
// CommonJS
(async () => {
const { default: walker } = await import('fstream-walk');
for await (const file of walker('./dir')) {
console.log(file.path);
}
})();
`
$3
A: Use AbortController:
`javascript
const controller = new AbortController();
setTimeout(() => controller.abort(), 5000);
for await (const file of walker('./huge-dir', {
signal: controller.signal
})) {
console.log(file.path);
}
`
$3
A: By default, no. Enable with followSymlinks: true. The library includes cycle detection to prevent infinite loops.
$3
A: Excellent! See benchmarks:
- 10,000 files: ~100ms
- 50,000 files: ~500ms
- Memory: Constant, ~5-10MB regardless of directory size
$3
A: No, this is a Node.js-only package as it uses fs module.
$3
A: See CONTRIBUTING.md for guidelines.
Troubleshooting
$3
Solution: Ensure you're using Node.js 18+ and have "type": "module" in your package.json.
$3
Solution: Avoid using sort option on large directories, or use findFiles helper for batch processing.
$3
Solution:
- Use maxDepth to limit recursion
- Add include/exclude filters early
- Consider using AbortSignal for early termination
Roadmap
- [ ] Watch mode for file system changes
- [ ] Parallel directory scanning
- [ ] Streaming API for very large directories
- [ ] Plugin system for custom filters
- [ ] Built-in cache layer
Changelog
See CHANGELOG.md for version history.
Contributing
Contributions are welcome! Please read CONTRIBUTING.md for details on our code of conduct and the process for submitting pull requests.
$3
`bash
git clone https://github.com/ersinkoc/fstream-walk.git
cd fstream-walk
npm test
npm run example
``