ginx

Ginx
===================

Ginx is a fast Nginx log parser written in Javascript; It can persist cursors of each file, to continue where it left off in case of a
shutdown, unexpected exception or Ctrl+D, all that with the option of parsing a directory of files instead of one file.

EXAMPLE USAGE
-------------

var Ginx = require("ginx");
var ginx = new Ginx();
//example read from file
ginx.parseFile("nginx_prod.log",
function(err, row){
if (err) throw err;
console.log("this will print each parsed line:" + JSON.stringify(row));
},
function(err, file){
if (err) throw err;
console.log(file + " parsing complete");
}
);

API
---

Ginx
---------

##Ginx(format, options={})##

Construct a 'new Ginx();
* Arguments
* format - Optional string representing the Nginx access_log format, usually in your nginx conf, check ./lib/ginx.js source to see the default format if you don't pass any.
* options - Optional hash that may contain two key:value pairs - default {'persistent': true, 'fieldsToObjects': false}
* 'persistent' defaults to true, whether your program will persist file positions to a local file in ./tmp/stored.cursors
* 'fieldsToObjects' defaults to false, whether the program will attempt to parse every column to its corresponding object, i.e Date, Number, or Null - if turned to True, it may impact the performance depending on Number and Size of files getting parsed.
* 'storageFile' defaults to ./tmp/stored.cursors, which file you want to store the cursors in
* 'originalText' - defaults to false, a boolean, if true, it will augment each row JSON object with its original text on an __originalText property

##ginx.parseLine(line, rowCallback, options={})##

Parse one line string
* Arguments
* line - a string representing the line to be parsed
* callback(err, row) - a callback function with the error if any, and the result row object which may contain custom attributes parsed from the format
Also, the result object has more attributes __file which is the file parsed from, __originalText which the original text before parsing and __lastrow if it is
* options, optional hash, currently only supports 'file': path

##ginx.parseFile(path, rowCallback, fileCallback, options={})##

Parse a file
* Arguments
* path - a string representing the path to the file to be parsed
* rowCallback(err, row) - a callback function for each row's end of parse, with the error if any, and the result row object
* fileCallback(err, file) - a callback function for file's end of parse
* dirCallback(err) - an Optional callback function for Dir's end of parse, if any
* options, optional hash, currently only supports 'isPool': boolean, which is used internally to determine if this file was in the waiting pool or not, you don't need it at all if you're parsing a single file, even if you're not, parseDir will take care of that part if you're parsing multiple files.

##ginx.parseDir(directory, rowCallback, fileCallback, dirCallback)##

Parse all files in the first level of a directory (they have to all have the same format)
* Arguments
* directory - a string representing the directory that has the files to be parsed
* rowCallback(err, row) - a callback function for each row's end of parse, with the error if any, and the result row object
* fileCallback(err, file) - a callback function for file's end of parse
* dirCallback(err, filesParsedCount) - a callback function for Dir's end of parse

##ginx.pause([readFile], [callback])##

Pauses all ReadStreams in Ginx Memory, or a single file's stream if passed.
* Arguments
* readFile - optional, the file path of the file being read by the ReadStream, if not passed, it will try to pause them all
* callback - optional, the callback after pausing completes

##ginx.resume([readFile], [callback])##

Resumes all ReadStreams in Ginx Memory, or a single file's stream if passed.
* Arguments
* readFile - optional, the file path of the file being read by the ReadStream, if not passed, it will try to resume them all.
* callback - optional, the callback after resuming completes

##ginx.stop([callback])##

Stops all ReadStreams in Ginx Memory, then persists all cursors in memory
* Arguments
* callback - optional, the callback after stoping completes

INSTALL
-------
npm install ginx

or manually
git clone https://github.com/akhoury/ginx.git
# cp the ginx dir into your program's node_modules
npm install # to install dependencies

you can 'node demo.js' to try it out quick, but please open demo.js and read the comments quickly.

##TODO##

* add tail -f feature to the command line tool using fs.watch or node-tail module
* add DeleteStorage to the API
* add auto detection feature for the format based on an example log line.
* add support for Error logs
* log with different levels (info, error, warning, debug, trace) to an optional log file
* refactor some of the big functions
* support other encodings
* optional DEBUG param flag, and input/output paths flags
* increase test coverage, seriously
... along with few others entered as issues.