winston-opensearch

transport for the winston logging toolkit.

Most of code is from the winston-elasticsearch version 0.17.3

Features

- logstash compatible message structure.
- Thus consumable with kibana.
- Date pattern based index names.
- Custom transformer function to transform logged data into a different message structure.
- Buffering of messages in case of unavailability of ES. The limit is the memory as all unwritten messages are kept in memory.

$3

I tested Winston 3.8.2, Opensearch 2.7 for AWS usage.

Installation

``sh
npm install --save winston winston-opensearch
`

Usage

`js
const winston = require('winston');
const { OpensearchTransport } = require('winston-opensearch');

const esTransportOpts = {
level: 'info',
};
const esTransport = new OpensearchTransport(esTransportOpts);
const logger = winston.createLogger({
transports: [esTransport],
});
// Compulsory error handling
logger.on('error', (error) => {
console.error('Error in logger caught', error);
});
esTransport.on('error', (error) => {
console.error('Error in logger caught', error);
});
`

The winston API for logging
can be used with one restriction: Only one JS object can only be logged and indexed as such.
If multiple objects are provided as arguments, the contents are stringified.

Options

- level [info] Messages logged with a severity greater or equal to the given one are logged to ES; others are discarded.
- index [none | when dataStream is true, logs-app-default] The index to be used. This option is mutually exclusive with indexPrefix.
- indexPrefix [logs] The prefix to use to generate the index name according to the pattern -. Can be string or function, returning the string to use.
- indexSuffixPattern [YYYY.MM.DD] a Day.js compatible date/ time pattern.
- transformer [see below] A transformer function to transform logged data into a different message structure.
- useTransformer [true] If set to true, the given transformer will be used (or the default). Set to false if you want to apply custom transformers during Winston's createLogger.
- ensureIndexTemplate [true] If set to true, the given indexTemplate is checked/ uploaded to ES when the module is sending the first log message to make sure the log messages are mapped in a sensible manner.
- indexTemplate [see file index-template-mapping.json] the mapping template to be ensured as parsed JSON.
ensureIndexTemplate is true and indexTemplate is undefined
- flushInterval [2000] Time span between bulk writes in ms.
- retryLimit [400] Number of retries to connect to ES before giving up.
- dataStream [false] Use Opensearch datastreams.
- healthCheckTimeout [30s] Timeout for one health check (health checks will be retried forever).
- healthCheckWaitForStatus [yellow] Status to wait for when check upon health. See its API docs for supported options.
- healthCheckWaitForNodes [>=1] Nodes to wait for when check upon health. See its API docs for supported options.
- client An opensearch client instance. If given, the clientOpts are ignored.
- clientOpts An object passed to the ES client. See its docs for supported options.
- waitForActiveShards [1] Sets the number of shard copies that must be active before proceeding with the bulk operation.
- pipeline [none] Sets the pipeline id to pre-process incoming documents with. See the bulk API docs.
- buffering [true] Boolean flag to enable or disable messages buffering. The bufferLimit option is ignored if set to false.
- bufferLimit [null] Limit for the number of log messages in the buffer.
- source [none] the source of the log message. This can be useful for microservices to understand from which service a log message origins.
- internalLogger [console.error] A logger of last resort to log internal errors.

$3

The default client and options will log through console.

$3

When changing the indexPrefix and/or the transformer,
make sure to provide a matching indexTemplate.

Transformer

The transformer function allows mutation of log data as provided
by winston into a shape more appropriate for indexing in Opensearch.

The default transformer generates a @timestamp and rolls any meta
objects into an object called fields.

Params:

- logdata An object with the data to log. Properties are:
- timestamp [new Date().toISOString()] The timestamp of the log entry
- level The log level of the entry
- message The message for the log entry
- meta The meta data for the log entry

Returns: Object with the following properties

- @timestamp The timestamp of the log entry
- severity The log level of the entry
- message The message for the log entry
- fields The meta data for the log entry

The default transformer function's transformation is shown below.

Input A:

`js
{
"message": "Some message",
"level": "info",
"meta": {
"method": "GET",
"url": "/sitemap.xml",
...
}
}
`

Output A:

`js
{
"@timestamp": "2019-09-30T05:09:08.282Z",
"message": "Some message",
"severity": "info",
"fields": {
"method": "GET",
"url": "/sitemap.xml",
...
}
}
`

The default transformer can be imported and extended

$3

`js
const { OpensearchTransformer } = require('winston-opensearch');
const esTransportOpts = {
transformer: (logData) => {
const transformed = OpensearchTransformer(logData);
transformed.fields.customField = 'customValue';
return transformed;
},
};
const esTransport = new OpensearchTransformer(esTransportOpts);
`

Note that in current logstash versions, the only "standard fields" are
@timestamp and @version, anything else is just free.

A custom transformer function can be provided in the options initiation.

Events

- error: in case of any error.

Example

An example assuming default settings.

$3

`js
logger.info('Some message', {});
`

Only JSON objects are logged from the meta field. Any non-object is ignored.

$3

The log message generated by this module has the following structure:

`js
{
"@timestamp": "2019-09-30T05:09:08.282Z",
"message": "Some log message",
"severity": "info",
"fields": {
"method": "GET",
"url": "/sitemap.xml",
"headers": {
"host": "www.example.com",
"user-agent": "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)",
"accept": "/",
"accept-encoding": "gzip,deflate",
"from": "googlebot(at)googlebot.com",
"if-modified-since": "Tue, 30 Sep 2019 11:34:56 GMT",
"x-forwarded-for": "66.249.78.19"
}
}
}
`

$3

This message would be POSTed to the following endpoint:

http://localhost:9200/logs-2019.09.30/log/

So the default mapping uses an index pattern logs-*`.