Process thousands of asynchronous or synchronous jobs, concurrently or sequentially, safely and efficiently, without creating thousands of closures.
npm install @ronomon/queueProcess thousands of asynchronous or synchronous jobs, concurrently or
sequentially, safely and efficiently, without creating thousands of closures.
@ronomon/queue has no dependencies.
``
npm install @ronomon/queue
var queue = new Queue([concurrency])
* concurrency An integer >= 1, the maximum number of jobs to run concurrently. Default: 1
Example:
`javascript
var Queue = require('@ronomon/queue');
var queue = new Queue(4);
queue.onData = function(integer, end) {
console.log('Processing ' + integer + '...');
// Notify the queue that we are done processing job:
end();
};
queue.onEnd = function(error) {
if (error) throw error;
console.log('Done');
};
queue.push(1);
queue.push(2);
queue.push(3);
queue.push(4);
queue.push(5);
queue.push(6);
queue.push(7);
queue.push(8);
queue.push(9);
queue.push(10);
// Notify the queue that we are done pushing jobs:
queue.end();
`
queue.onData = function(job, end) { end() }
* job A job which was pushed individually or pushed as part of an Array ofend
jobs.
* A callback for onData() to call when onData() has finishedjob
processing .
A function which must be provided and which will be called once for each job. If
an error is encountered while processing the job, this can be passed to the
end callback to stop the queue and return an error to the onEnd() function.
queue.onEnd = function([error]) {}
* error An error encountered (if any) while pushing jobs onto the queue, or
while processing jobs in the queue.
A function which must be provided and which will be called once queue.end()
has been called and once all running and pending jobs in the queue complete. If
an error is encountered, then will be called as soon as allstop()
running jobs complete. If the queue is stopped using , then onEnd()
will be called as soon as all running jobs complete.
queue.push(job)
* job Any object, will be passed to the onData() function.
queue.concat(jobs)
* jobs An Array of jobs, each of which will be passed to the onData()
function.
queue.end()
Notify the queue that no further jobs will be pushed. The queue will wait for
all running and all pending jobs to complete, and will then call the onEnd()
function. is idempotent, successive calls will be ignored.
queue.end(error)
* error An error encountered while pushing jobs onto the queue.
Notify the queue that no further jobs will be pushed because an error was
encountered while pushing jobs onto the queue. The queue will ignore any pending
jobs, will wait for all running jobs to complete, and will then call the
onEnd() function. The onEnd() function will be called with the same error,error
if no jobs return a different before queue.end(error) is called. queue.end(error) is idempotent, successive calls will be ignored.
queue.stop([error])
* error An error encountered while pushing jobs onto the queue, or while
processing jobs in the queue.
An optional method to notify the queue that any pending jobs should be ignored.
The queue will ignore any pending jobs, will wait for all running jobs to
complete, and will then call the onEnd() function. If no error argument isonEnd()
provided, then the function will be called without an error,error
provided no jobs return an before stop() is called (if any runningerror
jobs return an after stop() is called, their errors will bestop()
ignored). is idempotent, successive calls will be ignored.
There are several differences between stop() and queue.end(). stop() willqueue.end()
ignore any pending jobs, whereas will wait for all pending jobs toonEnd()
complete before calling the function. A successive call toqueue.end(error) will be ignored if queue.end() has already been called,stop(error)
whereas will stop the queue even if queue.end() has already been
called.
@ronomon/queue ships with a long-running fuzz test:
`
node test.js
To benchmark the cost of the queue implementation:
`
node benchmark.js