A module to execute function in parallel
npm install parallel-executionAuthor: Neal Hu(xhu@us.ibm.com)
Description: A fault-tolerant NodeJS module to execute script in parallel
executionFunc on each partition of partitionSpace, partitioned by partitionFunc.logDir with a name 'task-xxxx' where xxxx is a timestamp partitionFunc is a function that takes partitionSpace and returns an array of partitions, which will
be passed to executionFunc. executionFunc MUST be in the form executionFunc(partition, callback), and the callback function MUST be in the form callback(err, partition).
Read a execution log file specified in logFilePath, parse it and find all failed partitions.
A completed partition is defined as a partition that is logged "started" and logged "completed" in the log.
A failed partition is defined as a partition that is not completed.
Error and an array of string presentation of failed partitions is passed to callback
This dummy example shows how to print an array in parallel.
Define a partition space, a partition function and a execution function
``
var partitionSpace = [0, 1, 2, 3, 4, 5];
var partitionFunc = function(space) {
var getModPartition = function(space, mod, total) {
return space.filter(function(value, index) {
return index % total == mod;
})
}
return [0, 1, 2].map(function(v) {
return getModPartition(space, v, 3);
});
}
var executionFunc = function(partition, callback) {
console.log("Executing on partition: " + partition);
callback(null, partition);
}
`[0, 1, 2, 3, 4, 5]
This partition function will split the partition space into 3 partitions: [0, 3], [1, 4] and [2, 5]. The execution function will print each partition.executionFunc
Then you can call execution to execute the on each partition in parallel.``
var execution = new ParallelExecution();
execution.execute(partitionFunc, partitionSpace, executionFunc, '.', function(err, log) {
execution.readFailedParitions(log, function(err, failed) {
if (err) {
console.log("Error: " + err);
} else {
console.log("Failed partitions: " + failed);
}
});
})readFailedParitions`
The status of the execution will be saved into '.' and then read back by