logagent-filter-output-sql

Filter and aggregate parsed logs with SQL and @sematext/logagent

This applies SQL queries on parsed log events. The result of the query is emitted as new event, while the original events are omitted.

Using SQL it is very easy to aggregate values, e.g. group HTTP requests by status codes. The SQL WHERE statement is useful to filter events, before they get shipped to Elasticsearch or Logsene.

Installation

Assuming @sematext/logagent is installed gloabally:

``
npm i -g @sematext/logagent
npm i -g logagent-filter-output-sql
`

Configuration

Add following section 'outputFilter' to @sematext/logagent configuration file. Please note you could use the plugin with multiple configurations for different event sources.

`
input:
files:
- './access.log'

outputFilter:
- module: logagent-filter-output-sql
config:
source: !!js/regexp /access.log|httpd/
interval: 1 # every second
queries:
- # calculate average page size for different HTTP methods
SELECT 'apache_stats' AS _type,
AVG(size) AS size_avg,
COUNT(method) AS method_count,
method as http_method
FROM ?
GROUP BY method
- # log each request to the login page
SELECT *
FROM ?
WHERE path like "/wp-login%"
output:
elasticsearch:
url: http://localhost:9200
index: mylogs
`

Run logagent with your config:
`
logagent --config logagent-example-config.yml
``