A fixed-width file format toolset with streaming support and flexible options.
npm install @evologi/fixed-width
!Libraries.io dependency status for latest release
A fixed-width file format toolset with streaming support and flexible options.
- Features
- File format specs
- Install
- Usage
- In memory parsing
- Parse with Node.js streams
- Custom parsing
- In memory serializing
- Serialize with Node.js streams
- Custom serializing
- Options
- Errors
- Flexible: lots of options
- Zero dependencies: small footprint
- Native streaming support
- Well tested: code coverage above 90%
- Support big datasets: production ready
- Native ESM support: future proof
- TypeScript support
The fixed-width file format is a simple method to store data on a file. A separator divides rows (typically a newline char), and all fields have a fixed width on the row.
This format is useful to seek data directly from the file without any parsing procedure, but It has no compression mechanism. This limit will result in an over-usage of memory (both RAM and disk) to store the same data with other file formats (like JSON or XML).
But, in the year [insert current year here] A.D., you can still find someone using such file format. Fear not, my friend! This Node.js library will provide a toolset to parse and generate fixed-width files.
The next section will contain an example of fixed-width data with two fields: username and age. The first field has a width of 12 chars, and the second one has a width of 3 chars right-aligned.
``
username age
alice 024
bob 030
`
npm i @evologi/fixed-width
`javascript`
import { parse, stringify, Parser, Stringifier } from '@evologi/fixed-width'
It parses a text or a buffer into an array of elements. If It receives string of buffer It returns an array of all parsed items. It It receives an iterable (could be async), returns the same type of iterable inputted.
- input The raw data to parse.options
-
`javascript
import { parse } from '@evologi/fixed-width'
const text = 'alice 024\nbob 030\n'
const users = parse(text, {
eol: '\n',
fields: [
{
property: 'username',
width: 12
},
{
cast: value => parseInt(value, 10),
property: 'age',
width: 3
}
]
})
// [{ username: 'alice', age: 24 }, { username: 'bob', age: 30 }]
console.log(users)
`
It serializes an array of elements into a string. If the argument is an array, the output will be a string. The whole conversion is performed at the moment and in-memory. If the argument is some kind of iterable (sync or async), the output will be the same kind of inputted iterable.
- input
-
`javascript
import { stringify } from '@evologi/fixed-width'
const users = [
{ username: 'alice', age: 24 },
{ username: 'bob', age: 30 }
]
const text = stringify(users, {
eof: true,
eol: '\n',
fields: [
{
align: 'left',
property: 'username',
width: 12
},
{
align: 'right',
pad: '0',
property: 'age',
width: 3
}
]
})
// 'alice 024\nbob 030\n'
console.log(text)
`
It returns a Transform stream that accepts strings (or buffers) as input and emits the parsed elements.
- options
`javascript
import { Parser } from '@evologi/fixed-width'
const stream = Parser.stream({
eol: '\n',
fields: [
{
property: 'username',
width: 12
},
{
cast: value => parseInt(value, 10),
property: 'age',
width: 3
}
]
})
stream
.on('error', err => console.error(err))
.on('data', data => console.log(data))
.on('end', () => console.log('end'))
stream.write('alice 024\nbob 030')
stream.end()
`
It returns a Transform stream that accepts objects (o arrays) as input and emits the serialized strings.
- options
`javascript
import { Stringifier } from '@evologi/fixed-width'
const stream = Stringifier.stream({
eof: true,
eol: '\n',
fields: [
{
align: 'left',
property: 'username',
width: 12
},
{
align: 'right',
pad: '0',
property: 'age',
width: 3
}
]
})
let text = ''
stream
.on('error', err => console.error(err))
.on('data', data => { text += data.toString() })
.on('end', () => {
// 'alice 024\nbob 030\n'
console.log(text)
})
stream.write({ username: 'alice', age: 24 })
stream.write({ username: 'bob', age: 30 })
stream.end()
`
It creates a Parser instance. This object is useful when a custom optimized procedure is necessary. This object is used internally by the Node.js stream and the parse() function.
- options
It consists of only two methods, and all those methods are strictly synchronous.
#### Parser#write(stringOrBuffer)
It accepts a string or a buffer as input and returns an iterable that outputs all parsed objects up to the last completed row (line).
- stringOrBuffer
- Returns:
#### Parser#end()
It resets the Parser status and returns an iterable that could output other objects contained on the last partial row (line).
- Returns:
`javascript
import { Parser } from '@evologi/fixed-width'
const parser = new Parser({
eol: '\n',
fields: [
{
property: 'username',
width: 12
},
{
cast: value => parseInt(value, 10),
property: 'age',
width: 3
}
]
})
const users = Array.from(
parser.write('alice 024\nbob 030')
).concat(
Array.from(
parser.end()
)
)
// [{ username: 'alice', age: 24 }, { username: 'bob', age: 30 }]
console.log(users)
`
It creates a Stringifier instance. This object is useful when a custom optimized procedure is necessary. This object is used internally by the Node.js stream and the stringify() function.
- options
It consists of only two methods, and all those methods are strictly synchronous.
#### Stringifier#write(obj)
Push an object to serialize. Returns the serialized text of the passed object, including new line terminators.
- obj <*>
- Returns:
#### Stringifier#end()
Close the parsing and returns a final string.
- Returns:
`javascript
import { Stringifier } from '@evologi/fixed-width'
const stringifier = new Stringifier({
eof: true,
eol: '\n',
fields: [
{
align: 'left',
property: 'username',
width: 12
},
{
align: 'right',
pad: '0',
property: 'age',
width: 3
}
]
})
let text = ''
text += stringifier.write({ username: 'alice', age: 24 })
text += stringifier.write({ username: 'bob', age: 30 })
text += stringifier.end()
// 'alice 024\nbob 030\n'
console.log(text)
`
Type:
Default: "uft8"
The encoding used to handle strings and buffers. Only Node.js encodings are supported.
Type:
The End Of Line character that divides record rows.
It will defautl to os.EOL for serialization.
For parsing, the Parser will try to guess the correct line separator.
Type:
Default: true
Appends the End Of File char. If true, an End Of Line character is added at the end of the file.
Type: string
Default: " "
Values shorter than their field's width will be padded with this value while serializing. It's also the trimming value removed while parsing.
See trim, field.pad, and field.align options.
Type:
Default: true
It enables or disables values' trimming while parsing. Possibles values are:
- true: Trim the value from both sides (left and right)false
- : Disable trim "left"
- : Trim value's left side"right"
- : Trim value's right side"auto"
- : Guess the trim side by looking at field's align option:field.align: "right"
- if then trim leftfield.align: "left"
- if then trim right
> The trimmed value corresponds to the field's padding value.
`javascript`
trim('004200', { pad: '0', trim: 'right' })
// the trimmed value will be '0042'
Type:
Default: 1
The first line to consider while parsing (inclusive).
It is a 1-based integer (1 is the first line).
Type:
Default: Infinity
The last line to consider while parsing (inclusive).
It is a 1-based integer (1 is the first line).
Type:
Allow lines to be longer than the declared fields while parsing.
Default: true
Type:
Allow lines to be shorter than the declared fields while parsing.
Default: false
Type:
Completely ignore all empty lines. This options does not change the behaviour of the allowShorterLines option.
Default: true
Type:
This option is the only required one. It contains the specs for all the fields.
#### field.align
Type:
Default: "left"
Field's value alignment. Can be "left" or "right".
#### field.cast
Type:
A casting function that accepts the raw string value and returns the parsed one. It also provides a context object as second argument. Only used while parsing.
`javascript`
const options = {
fields: [
{
width: 5,
cast: (value, ctx) => {
// value is always a string
// ctx = { column: 1, line: 1, width: 5 }
return parseInt(value)
}
}
]
}
#### field.column
Type:
Field's columns. This is 1-based value (one is the first column). It defaults to the sum of all the previously defined fields' widths.
#### field.pad
Type:
Field level padding value. It defaults to the global one.
#### field.property
Type:
This option controls the expected format of both input and output objects.
##### Parsing
By defining this option, the Parser will emit objects. If the option is omitted, the emitted values will be arrays.
##### Serializing
By defining this option, the Stringifier will expect objects. If the option is omitted, the expected values will be arrays.
#### field.width
Type:
Field's width. Required.
#### field.stringify
Type:
A function that converts the field into a custom string. Useful for custom formatting, for example, formatting a Date into DD/MM/YYYY format. Only used while stringifying.
#### field.trim
Type:
Default: fallbacks to global trim option value.
Field-level trim option.trim
See option docs.
All errors that can occur during the parsing or serializing phase contain an error code. Error objects also contain enough info (properties) to debug the problem.
It's possible to detect custom fixed-width errors with their constructor:
`javascript
import { FixedWidthError } from '@evologi/fixed-width'
try {
// parse or stringify...
} catch (err) {
if (err instanceof FixedWidthError) {
console.log(err.code)
} else {
console.log('UNKNOWN_ERROR')
}
}
`
This error is raised when a partial line is found.
You can suppress this error with allowLongerLines or allowShorterLines` options.
This error is raised when a value cannot be serialized into a string.
This error is raised when a string value has a width that exceeds its field's width.