pull-lend-stream

A refinement of the paramap module for dynamic, unbounded, and fault-tolerant parallel processing.

Continuously lends values to concurrent sub-streams. Transparent re-lending in case of failure of sub-streams.

Each sub-stream continuously borrows (different) values from a lender and returns results back after processing.

* Supports multiple concurrent sub-streams.
* Produces results in the order in which the lender reads the values.
* If a sub-stream closes before producing all its results, the original source
values for missing results are lent transparently to other sub-streams.
* Each sub-stream obtains values as fast as they read them.

Useful for delegating processing to a dynamic number of concurrent,
cooperative, connected, but unreliable clients.

Quick Example
=============

var pull = require('pull-stream')
var lendStream = require('pull-lend-stream')

var lender = lendStream()

function minus (x, cb) {
setTimeout(function () {
cb(null, -x)
}, 201)
}

// Twice faster
function addTen (x, cb) {
setTimeout(function () {
cb(null, 10 + x)
}, 100)
}

function borrower (mapper) {
return function (err, stream) {
if (err) return console.log(err.message)

pull(
stream,
pull.asyncMap(mapper),
stream
)
}
}

// Prints -0,11,12,-3,14,15,-6,17,18,-9,20
pull(
pull.count(10),
lender,
pull.collect(function (err, results) {
if (err) throw err
console.log(results)
})
)

lender.lendStream(borrower(minus))
lender.lendStream(borrower(addTen))

Signature
=========

The following signature follows the js module signature
syntax and conventions.
All callbacks ('cb') have the '(err, value)' signature.

lendStream: () =>
lender: {
sink: (read: (abort, cb)),
lendStream: (borrower: (
err,
subStream: {
source: (abort, cb),
sink: (read: (abort, resultCb)),
close: (?err)
})),
source: (abort, resultCb)
}

Properties
==========
Italic names refer to the function signature above.

1. If no subStream is read then read is never called.
2. Multiple subStreams can be created by calling lendStream multiple times.
3. Once lendStream has been called,
3.1 the borrower will eventually be called either with a subStream or
an err;
3.2 if there is no err and values are read by calling subStream.source,
subStream.source will eventually abort.
4. lender.source closes after lender.sink has received an abort and all
subStreams have closed.
5. lender.source produces results in the order in which the values were
read by the lender.sink.
6. If a subStream ends before successfully sinking results computed for all
values read, then the values for missing results are transparently
migrated to other subStreams.
7. Unfair: if a subStream reads values faster than other subStreams, it
will obtain more values.
8. When a borrower is called, err is truthy if and only if:
8.1 lender.sink has not been called yet (lender is not connected to an
upstream source);
8.2 lender.source was aborted;
8.3 all available values have been borrowed and all results have been sourced.
9. subStream.close(?err) ends the corresponding subStream with the error err if present, or with true otherwise.

Expectations on the sub-streams
===============================

1. Sub-streams should correctly close when subStream.source aborts and the
event should propagate to their sink. Otherwise it will indefinitely prevent
the lender from closing (Prop. 4).

Debugging
=========

You can obtain a trace of the internal events of the module by activating the logging using the DEBUG=pull-lend-stream environment variable (see debug).

You can also obtain the internal state of the module at a specific point in time by calling the _state() method. It returns an object with the following properties:
```
return {
connected: Boolean, // The lender is connecter to an upstream source
ended: Boolean, // Upstream is closed
closed: Boolean, // lender is closed
openedNb: Number, // Number of sub-streams opened
lendState: Object // State of the internal pull-lend module
}

``

The output of the _state()` method should not be relied on for regular operations because it depends on the implementation of the module and may change in the future.

Random Testing
==============

A tool for random testing is available under ./test/random. The methodology used is explained in the README.