collectortoqueue

This library is in alpha. We will make an effort to support the library, but we reserve the right to make incompatible changes when necessary.

It is a distributed task Manager based on Kue. (node.js)

The goal of library collectortoqueue is a library to manage task with kue (node.js)

It is composed with 3 parts:

1) Collector : collect data from source (database, Mongo, Mysql, Redis, text file or array)
2) TaskManager : Do Something with data
3) Saver : Save Data Somewhere (Mongo, MySql, Redis, ...)

Simple Example

We create 2 files :
- produce.js : for produce task.
- consume.js : for compute tasks. Comsume.js can be deploy in a large number of machine or VM. The task has process

###Create file producer.js
``

//define the source of data (here mysql)
var collector = new ctq.GetSqlData('database', 'table', config.mysql);

//prepare data to be send in kue
var jobMaster = new ctq.JobKue(config.redis, collector);

//run : the database in send in the redis kue by groupe of 5000.
jobMaster
.init()
.then(jobMaster.exec())
.then(function () {
console.log('====================== all task has been send to Kue ======================');
})
.catch(function (e) {
console.log(e)
});

`




More complexe Example


In file producer.js
`

//define the source of data
var collector = new ctq.GetSqlData('myDatabase', 'emails', config.mysql);

//we don't want to collect every thing we add a filter
collector.filter = "WHERE email<>''";
//it means we start by row 0
collector.start = 0;
//it mean we collect 5000 rows each time
collector.size = 5000;

//prepare data to be send in kue
var jobMaster = new ctq.JobKue(config.redis, collector);
//nb of job send in the same time
jobMaster.type = 'url';

//run : the database in send in the redis kue by groupe of 5000.
jobMaster
.init()
.then(jobMaster.exec())
.then(function () {
console.log('====================== all task has been send to Kue ======================');
})
.catch(function (e) {
console.log(e)
});

``

$3

The collector collect data from source (database, text file or array)

$3

$3

example of collector in this module

Txt file
MySql
Mongo