rotating-file-stream

[![Build Status][travis-badge]][travis-url]
[![Code Climate][code-badge]][code-url]
[![Test Coverage][cover-badge]][code-url]

[![NPM version][npm-badge]][npm-url]
[![NPM downloads][npm-downloads-badge]][npm-url]
[![Stars][stars-badge]][stars-url]

[![Types][types-badge]][npm-url]
[![Dependents][deps-badge]][npm-url]
[![Donate][donate-badge]][donate-url]

[code-badge]: https://codeclimate.com/github/iccicci/rotating-file-stream/badges/gpa.svg
[code-url]: https://codeclimate.com/github/iccicci/rotating-file-stream
[cover-badge]: https://codeclimate.com/github/iccicci/rotating-file-stream/badges/coverage.svg
[deps-badge]: https://img.shields.io/librariesio/dependents/npm/rotating-file-stream?logo=npm
[deps-url]: https://www.npmjs.com/package/rotating-file-stream?activeTab=dependents
[donate-badge]: https://img.shields.io/static/v1?label=donate&message=bitcoin&color=blue&logo=bitcoin
[donate-url]: https://blockchain.info/address/12p1p5q7sK75tPyuesZmssiMYr4TKzpSCN
[github-url]: https://github.com/iccicci/rotating-file-stream
[npm-downloads-badge]: https://img.shields.io/npm/dw/rotating-file-stream?logo=npm
[npm-badge]: https://img.shields.io/npm/v/rotating-file-stream?color=green&logo=npm
[npm-url]: https://www.npmjs.com/package/rotating-file-stream
[stars-badge]: https://img.shields.io/github/stars/iccicci/rotating-file-stream?logo=github&style=flat&color=green
[stars-url]: https://github.com/iccicci/rotating-file-stream/stargazers
[travis-badge]: https://img.shields.io/travis/com/iccicci/rotating-file-stream?logo=travis
[travis-url]: https://app.travis-ci.com/github/iccicci/rotating-file-stream
[types-badge]: https://img.shields.io/static/v1?label=types&message=included&color=green&logo=typescript

$3

Creates a stream.Writable to a file which is
rotated. Rotation behavior can be deeply customized; optionally, classical UNIX logrotate behavior can be used.

$3

``javascript
const rfs = require("rotating-file-stream");
const stream = rfs.createStream("file.log", {
size: "10M", // rotate every 10 MegaBytes written
interval: "1d", // rotate daily
compress: "gzip" // compress rotated files
});
`

$3

With npm:

`sh
$ npm install --save rotating-file-stream
`

$3

- Upgrading from v2 to v3
- Upgrading from v1 to v2
- API
- [rfs.createStream(filename[, options])](#rfscreatestreamfilename-options)
- filename
- [filename(time[, index])](#filenametime-index)
- filename(index)
- Class: RotatingFileStream
- Event: 'external'
- Event: 'history'
- Event: 'open'
- Event: 'removed'
- Event: 'rotation'
- Event: 'rotated'
- Event: 'warning'
- options
- compress
- encoding
- history
- immutable
- initialRotation
- interval
- intervalBoundary
- intervalUTC
- maxFiles
- maxSize
- mode
- omitExtension
- path
- rotate
- size
- teeToStdout
- Rotation logic
- Under the hood
- Compatibility
- TypeScript
- License
- Bugs
- ChangeLog
- Donating

Upgrading from v2 to v3

In v3 the package was completely refactored using async / await.

TypeScript types for events and the external event were added.

Breaking change: by default the .gz extension is added to the rotated compressed files.

Breaking change: the way the _external compression command_ is executed was slightly changed; possible breaking
change.

To maintain back compatibility upgrading from v2 to v3, just follow this rules:

- using a _file name generator_ or not using options.compress: nothing to do
- using a _file name_ and using options.compress: use options.omitExtension or check
how rotated files are treated.

Upgrading from v1 to v2

There are two main changes in package interface.

In v1 the _default export_ of the package was directly the RotatingFileStream _constructor_ and the caller
have to use it; while in v2 there is no _default export_ and the caller should use the
createStream exported function and should not directly use
RotatingFileStream class.
This is quite easy to discover: if this change is not applied, nothing than a runtime error can happen.

The other important change is the removal of option rotationTime and the introduction of intervalBoundary.
In v1 the time argument passed to the _filename generator_ function, by default, is the time when _rotation job_
started, while if options.interval option is used, it is the lower boundary of the time interval within
_rotation job_ started. Later I was asked to add the possibility to restore the default value for this argument so I
introduced options.rotationTime option with this purpose. At the end the result was something a bit confusing,
something I never liked.
In v2 the time argument passed to the _filename generator_ function is always the time when _rotation job_
started, unless options.intervalBoundary option is used. In a few words, to maintain back
compatibility upgrading from v1 to v2, just follow this rules:

- using options.rotation: nothing to do
- not using options.rotation:
- not using options.interval: nothing to do
- using options.interval:
- using options.rotationTime: to remove it
- not using options.rotationTime: then use options.intervalBoundary.

API

`javascript
const rfs = require("rotating-file-stream");
`

rfs.createStream(filename[, options])

- filename <string> |
<Function> The name
of the file or the function to generate it, called _file name generator_. See below for
details.
- options <Object>
Rotation options, See below for details.
- Returns: <RotatingFileStream> The rotating file stream!

This interface is inspired to
fs.createWriteStream one. The file is rotated
following _options_ rules.

$3

The most complex problem about file name is: _how to call the rotated file name?_

The answer to this question may vary in many forms depending on application requirements and/or specifications.
If there are no requirements, a string can be used and _default rotated file name generator_ will be used;
otherwise a Function which returns the _rotated file name_ can be used.

Note:
if part of returned destination path does not exists, the rotation job will try to create it.

#### filename(time[, index])

- time <Date>

- By default: the time when rotation job started;
- if both options.interval and intervalBoundary options are enabled: the start
time of rotation period.

If null, the _not-rotated file name_ must be returned.

- index <number> The
progressive index of rotation by size in the same rotation period.

An example of a complex _rotated file name generator_ function could be:

`javascript
const pad = num => (num > 9 ? "" : "0") + num;
const generator = (time, index) => {
if (!time) return "file.log";

var month = time.getFullYear() + "" + pad(time.getMonth() + 1);
var day = pad(time.getDate());
var hour = pad(time.getHours());
var minute = pad(time.getMinutes());

return ${month}/${month}${day}-${hour}${minute}-${index}-file.log;
};

const rfs = require("rotating-file-stream");
const stream = rfs.createStream(generator, {
size: "10M",
interval: "30m"
});
`

Note:
if both of options.interval options.size are used, returned _rotated file name_ must be
function of both arguments time and index.

#### filename(index)

- index <number> The
progressive index of rotation. If null, the _not-rotated file name_ must be returned.

If classical logrotate behavior is enabled (by options.rotate), _rotated file name_ is only a
function of index.

Class: RotatingFileStream

Extends stream.Writable. It should not be directly
used. Exported only to be used with instanceof operator and similar.

$3

- stdout <string> The
standard output of the external compression command.
- stderr <string> The
standard error of the external compression command.

The external event is emitted once an _external compression command_ completes its execution to give access to the
command output streams.

$3

The history event is emitted once the _history check job_ is completed.

$3

- filename <string> Is
constant unless options.immutable is true.

The open event is emitted once the _not-rotated file_ is opened.

$3

- filename <string> The
name of the removed file.
- number <boolean>
- true if the file was removed due to options.maxFiles
- false if the file was removed due to options.maxSize

The removed event is emitted once a _rotated file_ is removed due to options.maxFiles or
options.maxSize.

$3

The rotation event is emitted once the _rotation job_ is started.

$3

- filename <string> The
_rotated file name_ produced.

The rotated event is emitted once the _rotation job_ is completed.

$3

- error <Error> The
non blocking error.

The warning event is emitted once a non blocking error happens.

options

- compress:
<boolean> |
<string> |
<Function>
Specifies compression method of rotated files. Default: null.
- encoding:
<string>
Specifies the default encoding. Default: 'utf8'.
- history:
<string>
Specifies the _history filename_. Default: null.
- immutable:
<boolean>
Never mutate file names. Default: null.
- initialRotation:
<boolean>
Initial rotation based on _not-rotated file_ timestamp. Default: null.
- interval:
<string>
Specifies the time interval to rotate the file. Default: null.
- intervalBoundary:
<boolean>
Makes rotated file name with lower boundary of rotation period. Default: null.
- intervalUTC:
<boolean>
Boundaries for rotation are computed in UTC. Default: null.
- maxFiles:
<number>
Specifies the maximum number of rotated files to keep. Default: null.
- maxSize:
<string>
Specifies the maximum size of rotated files to keep. Default: null.
- mode:
<number>
Forwarded to fs.createWriteStream.
Default: 0o666.
- omitExtension:
<boolean>
Omits the .gz extension from compressed rotated files. Default: null.
- path:
<string>
Specifies the base path for files. Default: null.
- rotate:
<number>
Enables the classical UNIX logrotate behavior. Default: null.
- size:
<string>
Specifies the file size to rotate the file. Default: null.
- teeToStdout:
<boolean>
Writes file content to stdout as well. Default: null.

$3

Specifies the default encoding that is used when no encoding is specified as an argument to
stream.write().

$3

Forwarded to fs.createWriteStream.

$3

If present, it is prepended to generated file names as well as for history file.

$3

If true, it makes the file content to be written to stdout as well. Useful for debugging purposes.

$3

Accepts a positive integer followed by one of these possible letters:

- B: Bites
- K: KiloBites
- M: MegaBytes
- G: GigaBytes

`javascript
size: '300B', // rotates the file when size exceeds 300 Bytes
// useful for tests
`

`javascript
size: '300K', // rotates the file when size exceeds 300 KiloBytes
`

`javascript
size: '100M', // rotates the file when size exceeds 100 MegaBytes
`

`javascript
size: '1G', // rotates the file when size exceeds a GigaByte
`

$3

Accepts a positive integer followed by one of these possible letters:

- s: seconds. Accepts integer divider of 60.
- m: minutes. Accepts integer divider of 60.
- h: hours. Accepts integer divider of 24.
- d: days. Accepts integer.
- M: months. Accepts integer. EXPERIMENTAL

`javascript
interval: '5s', // rotates at seconds 0, 5, 10, 15 and so on
// useful for tests
`

`javascript
interval: '5m', // rotates at minutes 0, 5, 10, 15 and so on
`

`javascript
interval: '2h', // rotates at midnight, 02:00, 04:00 and so on
`

`javascript
interval: '1d', // rotates at every midnight
`

`javascript
interval: '1M', // rotates at every midnight between two distinct months
`

$3

If set to true, the argument time of _filename generator_ is no longer the time when _rotation job_ started, but
the _lower boundary_ of rotation interval.

Note:
this option has effect only if options.interval is used.

$3

If set to true, the boundaries of the rotation interval are computed against UTC time rather than against system time
zone.

Note:
this option has effect only if options.intervalBoundary is used.

$3

The best choice here is to use the value "gzip" to use Node.js internal compression library.

For historical reasons external compression can be used.

To enable external compression, a _function_ can be used or simply the _boolean_ true value to use default
external compression.
The function should accept source and dest file names and must return the shell command to be executed to
compress the file.
The two following code snippets have exactly the same effect:

`javascript
var rfs = require("rotating-file-stream");
var stream = rfs.createStream("file.log", {
size: "10M",
compress: true
});
`

`javascript
var rfs = require("rotating-file-stream");
var stream = rfs.createStream("file.log", {
size: "10M",
compress: (source, dest) => cat ${source} | gzip -c9 > ${dest}
});
`

Note:
this option is ignored if options.immutable is used.

Note:
the shell command to compress the rotated file should not remove the source file, it will be removed by the package
if rotation job complete with success.

$3

From v3 the package adds by default the .gz extension to the rotated compressed files. Simultaneously this option
was added: set this option to true to not add the extension, i.e. to keep backward compatibility.

$3

When program stops in a rotation period then restarts in a new rotation period, logs of different rotation period will
go in the next rotated file; in a few words: a rotation job is lost. If this option is set to true an initial check
is performed against the _not-rotated file_ timestamp and, if it falls in a previous rotation period, an initial
rotation job is done as well.

Note:
this option has effect only if both options.interval and options.intervalBoundary
are used.

Note:
this option is ignored if options.rotate is used.

$3

If specified, classical UNIX logrotate behavior is enabled and the value of this option has same effect in
_logrotate.conf_ file.

Note:
if this option is used following ones take no effect: options.history, options.immutable,
options.initialRotation, options.intervalBoundary,
options.maxFiles, options.maxSize.

$3

If set to true, names of generated files never changes. New files are immediately generated with their rotated
name. In other words the _rotated file name generator_ is never called with a null _time_ argument unless to
determinate the _history file_ name; this can happen if options.history is not used while
options.maxFiles or options.maxSize are used.
The filename argument passed to 'open' _event_ evaluates now as the newly created file name.

Useful to send logs to logstash through filebeat.

Note:
if this option is used, options.compress is ignored.

Note:
this option is ignored if options.interval is not used.

$3

Due to the complexity that _rotated file names_ can have because of the _filename generator function_, if number or
size of rotated files should not exceed a given limit, the package needs a file where to store this information. This
option specifies the name _history file_. This option takes effect only if at least one of
options.maxFiles or options.maxSize is used. If null, the _not rotated filename_ with
the '.txt' suffix is used.

$3

If specified, it's value is the maximum number of _rotated files_ to be kept.

$3

If specified, it's value must respect same syntax of option.size and is the maximum size of _rotated files_ to
be kept.

Rotation logic

Regardless of when and why rotation happens, the content of a single
stream.write
will never be split among two files.

by size

Once the _not-rotated_ file is opened first time, its size is checked and if it is greater or equal to
size limit, a first rotation happens. After each
stream.write,
the same check is performed.

by interval

The package sets a Timeout
to start a rotation job at the right moment.

Under the hood

Logs should be handled so carefully, so this package tries to never overwrite files.

At stream creation, if the _not-rotated_ log file already exists and its size exceeds the rotation size,
an initial rotation attempt is done.

At each rotation attempt a check is done to verify that destination rotated file does not exists yet;
if this is not the case a new destination _rotated file name_ is generated and the same check is
performed before going on. This is repeated until a not existing destination file name is found or the
package is exhausted. For this reason the _rotated file name generator_ function could be called several
times for each rotation job.

If requested through options.maxFiles or options.maxSize, at the end of a rotation job, a
check is performed to ensure that given limits are respected. This means that
while rotation job is running both the limits could be not respected. The same can happen till the end of first
_rotation job_ if options.maxFiles or options.maxSize are changed between two runs.
The first check performed is the one against options.maxFiles, in case some files are removed, then the
check against options.maxSize is performed, finally other files can be removed. When
options.maxFiles or options.maxSize are enabled for first time, an _history file_ can be
created with one _rotated filename_ (as returned by _filename generator function_) at each line.

Once an error _event_ is emitted, nothing more can be done: the stream is closed as well.

Compatibility

Requires Node.js v14.

The package is tested under all Node.js versions
currently supported accordingly to Node.js Release.

To work with the package under Windows, be sure to configure bash.exe as your _script-shell_.

`
> npm config set script-shell bash.exe
``

TypeScript

TypeScript types are distributed with the package itself.

License

MIT License

Bugs

Do not hesitate to report any bug or inconsistency @github.

ChangeLog

ChangeLog

Donating

If you find useful this package, please consider the opportunity to donate some satoshis to this bitcoin address:
12p1p5q7sK75tPyuesZmssiMYr4TKzpSCN