System for distributed computing
npm install codis-masterbash
$ yarn global add codis-master
`
or
`$xslt
$ npm install -g codis-master
`
Getting Started
After installing the package, you must call the command:
`bash
$ codis-init-master
`
The command created the system configuration file codis.config.js and a directory /plan will be created and the system's initial script of the system tasks: /plan.js.
To start the system, you must call the command:
`bash
$ codis-run-master
`
Configuration
The default configuration file contains:
`javascript
exports.config = {
port: 5299,
acceptedOrigins: ['http://localhost:3000'],
threadNumberPerNode: 4,
planLocalization: '\\plan\\plan.js',
administrationPanel: {
publishAPI: true,
port: 9251,
refreshRateNodesPage: 5000,
refreshRateTasksPage: 5000,
refreshRateSystemPage: 1000,
},
};
`
###Options:
- port - the port on which the system is to be exposed
- acceptedOrigins - all acceptable origins from which communication to the system will come. It means that only on these given origins the system will allow to run threads of executive nodes. There can be many possible values. In order for the system to accept each origin, you set true in the array: `acceptedOrigins: [true],`
- threadNumberPerNode - determine how many threads will be run at a single execution node. That is, if we have the value set to (for example 4), then when opening a page with an injected [CODIS-SLAVE] script, browser will run 4 working threads under the given tab in the browser / or the entire browser. The larger the number ~ the greater efficiency of a single execution node (of course, only to some extent).
- planLocalization - location of the initial system script
- administrationPanel
- publishAPI - providing api for the administration panel CODIS-ADMINISTRATION-PANEL
- port - the port on which the API is to be exposed
- refreshRateNodesPage - frequency of refreshing nodes information [ms]
- refreshRateTasksPage - frequency of refreshing tasks information [ms]
- refreshRateSystemPage - frequency of refreshing general system information [ms]
Plan of system tasks
The initial content of the file plan.js
`javascript
exports.run = (CODIS) => {};
`
In the body of the function, we write the whole logic of what the system has to do. The CODIS object that we receive as a parameter of this function is used for communication with the system's memory. The CODIS object provides two methods to use:
- createTask()
- controlNotification()
$3
Function definition:
`typescript
createTask:
(module: string, task: any, options?: TaskOptions, parentTaskId?: number)
=> {promise: Promise, promiseResolve: (PromiseLike | T)) => void, promiseReject: (reason?: any) => void) => void, idTask: number, createSubTask: createTask}
`
##### Parameters:
Only two fields are required to provide: task, module. The other arguments are optional.
- module - the name of the module to be downloaded for the given task.
The definition of a given module created by the user allows to perform appropriate calculations for a given set of input data of tasks (given as the second function argument). Otherwise it is a file with the given module name with the extension .mjs. For example: exampleModule.mjs
- task - the content of the task to be handled and executed by the module specified in the first createTask() argument.
Task can have any format.
- parentTaskId - we give the identification number of the task, which will be the main (parent) task for the task being created.
- options - definition of TaskOptions:
`typescript
TaskOptions {
endless?: boolean;
lazyLoadingData?(task: Task): any;
handleRegisterSolution?(task: Task, solution: any): any;
}
`
endless - when we want to create a task that will be performed by execution nodes endlessly.
Each free system thread will be sent the specified task.
lazyLoadingData - function that will be called when a free thread comes to the task.
The function should return the content of the task for the thread (that is what is normally given as the second argument of the createTask () function).
This can be done so that the data that is to be used as the content of the task are not kept in the system's memory.
But it also creates a real way for the data to be extracted, for example, from external sources (databases or files).
The argument of the function that we can use is the content of the task originally given as the second argument of the createTask() function. And it should be remembered that the content of this original task will not be sent to the execution thread, if we have the given function lies the LoadingData in the options of the task being created (unless in this function we actually return this value).
handleRegisterSolution - a method of use similar to that offered by lazyLoadingData. Only that this function is called when we get a response from the execution thread where our tasks were processed. And this function gets just a solution to a given task and decides what to do next with a given solution (eg save to an external database or to a file). As the first argument receives the details of the task, in the second argument we get received solution.
#### The returned object from createTask()
You should know that the entire communication process of the script written by the user and the system relies on Promise objects from JavaScript. When creating a task, Promise is created and returned, which is connected to the memory. This promise will be solved when the task is resolved. Thanks to this, we can create an extensive and task-dependent system logic.
The returned object has 5 fields:
- promise - it is an object of type Promise. It will be resolved only when the task is resolved by one of the nodes and returns the value of the solution.
- promiseResolve, rejectPromise - thanks to these functions, we can solve or reject a given Promise object, which is assigned to the created object, from the level of the script related to plan.js. Give the value of the solution to the function.
- idTask - identification number of the created task. We can use it at a later stage to, for example, create dependencies with other tasks or to report solutions or errors related to this task.
- createSubTask - function through which we can create tasks depending on the task. The syntax and return values of this function are the same as createTask, but the difference here is the fact that the last argument parentTaskId is already filled with the identification number of the task just created.
`typescript
const task = CODIS.createTask('exampleModule', {task});
const subTask = task.createSubTask('exampleModule', {task2});
`
is the same as:
`typescript
const task = CODIS.createTask('exampleModule', {task});
const subTask = CODIS.createTask('exampleModule', {task2}, null, task.idTask);
`
$3
Function definition:
`typescript
controlNotification: (message: string, type: NotificationType = 'info') => void;
`
`typescript
type NotificationType = 'error' | 'primary' | 'info' | 'custom' | 'warning' | 'success';
`
This function is mainly used to display notifications / information on the console of the administration panel.
We can choose the category of notification from the available list of options NotificationType. Each of these types will be displayed in a special way and color.
Modules
An important part of the system are modules. So the scripts that are downloaded as threads of executive nodes. The single module is a single script with the extension .mjs. In these files we define the logic of solving incoming tasks to the thread.
Modules should be provided by the service that serves the website and be stored in a location designated by CODIS-SLAVE.
For more information, go to the CODIS-SLAVE package page:
Simple Examples
The examples will be illustrated by the example of the system for finding prime numbers:
In the first example, we want to count the number of prime numbers in the range of 1 - 100,000:
`javascript
exports.run = (CODIS) => {
let promises = [];
let counter = 0;
CODIS.controlNotification('Create task', 'primary');
for (let i = 1; i < 100000; i++) {
promises.push(CODIS.createTask("primeNumber", {n: i}).promise.then(
() => {
CODIS.controlNotification(Prime number found: ${i}, 'info');
counter++;
}));
}
Promise.all(promises).then(
() => {CODIS.controlNotification(Found ${counter} prime numbers, 'success');}
);
};
`
In the second example we will extract numbers from the database and found prime numbers will be stored in the same database. And we will do it forever.
`javascript
const getNumberFromDatabase = () => {
const number = con.query(CALL get_number());
return {n: number};
};
const savePrimeNumber = (task, solution) => {
if (solution.isPrimeNumber) {
const number = solution.number;
con.query('INSERT INTO prime_num SET ?', {number})
}
};
exports.run = (CODIS) => {
CODIS.createTask("primeNumber", null, {endless: true, lazyLoadingData: getNumberFromDatabase, handleRegisterSolution: savePrimeNumber});
};
`
In the third example, we'll create a parent thread over threads to check prime numbers. The parent's thread will accept incoming primes from the sub-tasks threads.
`javascript
exports.run = (CODIS) => {
const parentTask = CODIS.createTask("collectNumbers");
for (let i = 1; i <= 100000; i++) {
parentTask.createSubTask("exampleModule", {n: i});
}
parentTask.promise.then((solution) => {
CODIS.controlNotification(Prime Numbers: ${solution.primeNumbers.toString()}., 'success');
CODIS.controlNotification(Composite Numbers: ${solution.compositeNumbers.toString()}, 'success');
});
};
``