Generator based line reader
npm install @prantlf/gen-readlines
A generator based line reader. This node package will return the lines of a file
as a generator when given file descriptor and the size of the file.
This is a fork of the original project with additional features and fixes:
* Support CR alone as a line break.
* Allow limiting the maximum line length.
* No warnings about usage of a deprecated API.
I created this project primarily for better flow control of reading lines in a file.
Instead of using callbacks for reading lines within a file, this will use a generator which has some unique benefits.
* Because it is a generator, the lines of the file are not stored in a large array
* Code is synchronous, so you don't have to worry about the asynchronous effects of waiting for a callback
* Performant, our micro-benchmark demonstrates it can compete with callback-based line readers and in fact appears to be faster.
Also, there are no external depencies and the library was built using TypeScript.
``
npm install @prantlf/gen-readlines
`js
import { fromFile } from 'gen-readlines';
async function readFile() {
for (let line of fromFile('./file.txt')) {
console.log(line.toString());
}
}
`
If you already have the file open and you know the filesize you can use gen-readlines and it will
create a generator which will iterate through all the lines in that file.
`js
import fs from 'fs';
import util from 'util';
import readlines from '@prantlf/gen-readlines';
const open = util.promisify(fs.open);
const fstat = util.promisify(fs.fstat);
async function readFile() {
const fd = await open('./file.txt');
const stat = await fstat(fd);
const fileSize = stat.size;
for (let line of readlines(fd, fileSize)) {
console.log(line.toString());
}
console.log('continue code execution, no callbacks!');
fs.closeSync(fd);
}
`
readlines returns a generator object and calling next will get the next
line as a buffer object:
`js`
var file = readlines(fd, stats.size);
var line = file.next();
console.log(line);
// { value:
Convert the buffer to a string:
`js`
line.toString();
// This is the first line of the file
You can limit the maximum line length. When the specified length is reached
while reading a line, the buffer will be returned as a new line just like when a
line break was encountered:
`js`
// If original lines are longer than 255 characters, an artificial line break
// will be enforced after each 255 characters reached on a single line. More
// then original lines will be returned by the generator then.
var file = readlines(fd, stats.size, { maxLineLength: 255 });
for (let line of readlines(fd, fileSize)) {
console.log(line.toString());
}
You can change the maximum line length for each generated line. If you do not
specify the maximum length, when you read the next line, the original maximum
line length passed to readlines will be used:
`js`
// Lines will not be longer than 255 characters by default.
var file = readlines(fd, stats.size, { maxLineLength: 255 });
var line = file.next(); // 255 characters maximum
line = file.next(127); // 127 characters maximum
line = file.next(); // 255 characters maximum again
Note: The very first generation (call to the next method) cannot accept an readlines
alternative maximum line length. It will always use the default value passed to . First the following calls to next allow to specify alternative yield
values. This is caused by the nature of JavaScript generators, which obtain the
value from first when when resuming the generation.
Also you can use the simplified version of readlines:`js
import readlines from '@prantlf/gen-readlines';
for (let line of readlines.fromFile('./file.txt')) {
console.log(line.toString());
}
`
* node 6 or newer, to get access to generators and new methods of the Buffer object.
* fd {Number} The file descriptor
* filesize {Number} The size of the file in bytes
bufferSize {Number} The size of the buffer in bytes, default: 64\1024
* position {Number} The position where to start reading the file in bytes, default: 0
* maxLineLength {Number} The length to stop reading at if no line break has been reached, default: Infinity
* filename {string} Name of input file
bufferSize {Number} The size of the buffer in bytes, default: 64\1024
* maxLineLength {Number} The length to stop reading at if no line break has been reached, default: Infinity
We are using mocha for unit testing
`
npm test
./perf` contains benchmarks of several libraries, including this one:
Reading lines using...
byline x 159 ops/sec ±1.71% (76 runs sampled)
line-reader x 158 ops/sec ±1.35% (75 runs sampled)
linebyline x 111 ops/sec ±1.14% (85 runs sampled)
readline x 244 ops/sec ±1.13% (81 runs sampled)
gen-readlines returning each line x 96.68 ops/sec ±1.08% (76 runs sampled)
gen-readlines skipping all lines x 143 ops/sec ±1.55% (83 runs sampled)