@rymdtiden/eventlog


eventlog
========

Append-only local data store, made for event sourcing.

All your events will be added to logfiles (for persistent storage),
as JSON blobs (one event/JSON blob per row).

Events are written to logfiles with the current date in the filename, so each
individual logfile will not grow larger than what is needed for one day's
events.

You don't have to keep more than one day of history, if you do not have to
process those events again.

how to use
----------

``
const eventlog = require("eventlog");
const { add, consume } = eventlog("data/events-%y-%m-%d.log");

// To add events to the event log:
add({ type: "someevent", id: 1337, customstuff: "anything you' like!" });

// To consume events:
consume((event, meta) => {
// Do whatever.
// Return a promise if you want to block more events.
// No events will be emitted until the promise resolves/rejects.
});
`

functions
---------

$3

Where filenamePattern is the path to where logfiles will be written. It must
contain the substrings %y, %m and %d, which will be replaced by the
year, month and day. Example: data/events-%y-%m-%d.log

Will return { add, consume }.

Both add and consume are functions. See documentation for those below.

$3

Where eventObj is a json-stringify:able object.

Returns { id, logfile, promise }

* id is a unique random string identifying the event you just added.
* logfile is the file path to the logfile that the event was written to.
* promise is a Promise that will resolve when the write to disk is verified
and the event has been given a position.

The promise resolves with this object { id, pos, prevPos }.

* id is the same id as returned from add().
* pos is the position number for this event.
* prevPos is the position number for the previous event. If the event is the
first ever written, prevPos will be undefined.

$3

Where callback is the a function to be called for every event. fromPosition
is the starting position from where events will be consumed.

The callback function will be called with the arguments:
callback(eventObj, meta); where eventObj is the event that was added with
the add() function. meta is an object with metadata for the event:
{ id, pos, prevPos }.

event positions
---------------

Every event will be enumbered with an unique number, which determines the
order of the events.

Event numbers are 16 digits long. The first four digits are the year of the
event, the next two are the month and the comes to digits which are the day.
The following eight digits are just an incremental number for that day,
starting with 00000000 for every day.

So the first event on 31 of January 1984 is 1984013100000000 and the second
event that day is 1984013100000001`.

limits
------

* Maximum 100 million events per day.

roadmap
-------

* Upload historic logfiles to AWS S3, and streaming events from there when you
need to replay your history.
* Better support for multiple processes. Today, multipe processes can write to
the same log without risk for data corruption, as long as the events are
small enough (4096 bytes on Linux) and your filsystem is POSIX compatible.
This is just a rudimentary support for multiple processes working with the
samt files. We would need a proper locking mechanism to better support
multiple processes adding events. Multiple processes just reading/consuming
events is 100% safe already.