DCP-Worker

This is the official DCP Worker program for the Distributive Compute Platform.

This package implements a DCP Worker which can be executed interactively, or a as system service, using
Node.js to communicate with the scheduler and control the DCP Evaluator.

A companion program, the DCP Evaluator, is required to use this worker. When you install the DCP Worker
with your system's package manager, the installer will automatically install the DCP Evaluator as
dcp-evaluator-v8 or dcp-evaluator. The DCP Evaluator is a secure sandboxing tool which uses
Google's V8 JavaScript engine and Google's Dawn WebGPU implementation for secure execution of JavaScript, WebAssembly, and WebGPU code.

You can find a complete package for Linux, including Evaluator binaries at https://gitlab.com/Distributed-Compute-Protocol/dcp-native/-/releases.
The build is CMake with GN and largely based around V8.

This package is also used for DCP LocalExec, a local debugging tool for developers using DCP Client.

Release Notes

This document was last updated Sep 12, 2024.

$3

- Node.js (all maintenance, active, current releases)
- Microsoft Windows 10, 11
- Ubuntu Linux 20.04
- Ubuntu Linux 22.04
- Ubuntu Linux 24.04
- More to come

Manifest

| File | Purpose
|:---------------------------|:------------------------------------------------
| bin/dcp-worker | Program which requests work from the scheduler and coordinates its evaluation in a secure environment called dcp-evaluator-v8
| bin/dcp-evaluator-start | Program which launches dcp-evaluator-v8
| bin/dcp-evaluator-manager | Program which which manages the launching of dcp-evaluator-v8; e.g. based on system load, terminal activity, screensaver activity, etc.
| examples/dcp-config.js | Default configuration for dcp-worker, copy to /etc/dcp/overrides/dcp-worker to override sections

$3

- dcp-evaluator-v8, our isolated JS/WASM/WebGPU evaluation environment, ships separately. Installing dcp-worker with your system's package manager will automatically install dcp-evaluator-v8 as a dependency.

Troubleshooting

The following conditions must be met for a Worker to do work:
* the Evaluator must be startable - normally managed with dcp-evaluator-manager
* the Worker must be running
* there must be work available on the scheduler that is suitable for this worker
* the worker must have the correct capabilities (GPU?)
* the job's payment must exceed the worker's minimum wage
* the worker and the job must be in the same Compute Group

$3

This varies from platform to platform. This document assumes the Worker was installed from an
OS-specific package supplied by Distributive.

#### All Operating Systems
The dcp-worker program in this package normally runs as a system service, but it can also be run as an
interactive program. This program displays configuration information and shows what it is doing with the
Evaluator in real time. dcp-worker -h shows a help screen; to run in interactive mode, simply start
bin/dcp-worker from a terminal window without any -o options.

Administrators with a large number of workers may wish to configure syslog
logging, and use an off-the-shelf logging aggregation service. The full gamut of options relating to
syslog output are documented in the dcp-worker help screen.

The Worker can log its activities to a variety of destinations (dependent on operating system), and
emits logs when new jobs are fetched, job slices are completed, and so on.

#### Microsoft Windows
The Worker is installed as a Windows Service, and can be managed via the Windows Service Manager.
To open the Windows Services Manager on Windows 10 or 11:
* Press Windows-X or right-click the start button to open the WinX menu
* Choose Run
* Type sevices.msc in the Run box and press enter

By default, the DCP Worker service writes logs to the Windows Event Viewer, but can be sent to a remote
syslog service if so desired.

To open the event viewer on Windows 10 or 11, click
* Start
* Control Panel
* System and Security
* Administrative Tools

Next, double-click Event Viewer, and select DCP Worker logs.

#### Ubuntu Linux
On Ubuntu Linux, the Worker is started under the control of systemd, running as
the user dcp. The Worker communicates with the Scheduler and uses an Evaluator to execute workload.
By default, the Worker logs are written to the system journal, but can be sent to a remote syslog
service if so desired. The systemd unit is named dcp-worker.

| Action | Command
|:---------------------------|:------------------------------------------------
| View all DCP systemd units | systemctl --no-pager list-units 'dcp*' --all
| Stop the Worker | sudo systemctl stop dcp-worker
| Start the Worker | sudo systemctl start dcp-worker
| Restart the Worker | sudo systemctl restart dcp-worker
| View the journal (logs) | sudo systemctl journalctl -u dcp-worker -f

The default configuration runs bin/dcp-worker -o console, which sends log output to stdout/stderr; this
output is captured by systemd and recorded in the system journals.
sudo systemctl journalctl -u dcp-worker --since='15 minutes ago' will show recent activity.

$3

This varies from platform to platform. This document assumes the Worker was installed from an
OS-specific package supplied by Distributive.

#### Microsoft Windows
The Evaluator is automatically started by the Distributive Screensaver.

#### Ubuntu Linux
On Ubuntu Linux, the Evaluator is started by dcp-evaluator-manager, under the control of
systemd, running as the user dcp. The Evaluator Manager is responsible for
disabling the Evaluator when the system load changes, the screen saver deactivatees, or a terminal
becomes active.

| Action | Command
|:---------------------------|:------------------------------------------------
| View all DCP systemd units | systemctl --no-pager list-units 'dcp*' --all
| Stop the Evaluator | sudo systemctl stop dcp-evaluator-manager
| Start the Evaluator | sudo systemctl start dcp-evaluator-manager
| Restart the Evaluator | sudo systemctl restart dcp-evaluator-manager
| View the journal (logs) | sudo systemctl journalctl -u dcp-evaluator-manager -f

Systems Integration

This DCP Worker package can be integrated into a wide variety of systems. Distributive currently
(Aug 2024) ships a Windows screensaver, an Ubuntu (Debian) package, and a Dockerized version of the
Debian package.

Systems integrators should be aware that the Evaluator (processes named dcp-evaluator-v8) can be
killed at any time to immediately decrease system load. A program called dcp-evaluator-manager
can be used to automate this; it can kill running Evaluators based on screensaver or terminal activity,
and refuse to spawn new Evaluators based on current system load. Systems Integrators familiar with
inetd will understand that dcp-evaluator-manager is a variation on this theme, and can be replaced
a similar program should a more suitable one exist to manage load for the environment in question.

Systems integrators should leave the dcp-worker program running as much as possible. This process
consumes very few resources, but manages the transmission of results to the Scheduler. During system
shutdown, delivering a single SIGINT to dcp-worker will cause it submit all pending results to the
Scheduler, return all pending slices, etc. This graceful shutdown will take less than 30 seconds
(usually much less).

No DCP job or process will be irrepairably damanged by killing any process on a Worker node. The worst
that can happen is that work with pending results will be routed to another node.

DCP Glossary

$3

#### Scheduler
A Node.js daemon which
* receives work functions and data sets from Compute API
* slices data into smaller sets
* transmits work and data points to Worker
* determines cost of work and instructs the Bank to distribute funds between entities accordingly
* ensures that all tasks eventually complete, provided appropriate financial and computation resources can be deployed in furtherance of this goal

#### Bank
A Node.js daemon which
* manages a ledger for DCC. This is not a blockchain currency.
* enables the movement of DCC between entities requesting work and entities performing work
* enables the movement of DCC between the ledger and the blockchain
* enables the placement of DCC in escrow on behalf of the Scheduler for work which is anticipated to be done

#### Compute Group
A collection of Workers and Jobs

#### Portal
A user-facing web application which allows or enables
* creation and management of user accounts
* management of bank accounts (ledgers)
* transfer of DCC between bank accounts
* transfer of DCC to and from the blockchain
* execution of the browser-based Worker

#### Worker
A JavaScript program which includes a Supervisor and one or more Sandboxes
* performs computations
* retrieves work and data points from Scheduler
* retrieves work dependencies from Package Server
* returns results and cost metrics to Scheduler
* Specific instances of Worker include
- a browser-based Worker
- a standalone Worker operating on Google's v8 engine

#### Sandbox
A component of a Worker, used to execute arbitrary JavaScript code in a secure environment.

#### Supervisor
The component of a Worker which communicates with the Scheduler and Sandboxes.

$3

#### Job
The collection consisting of an input set, Work Function and result setup. Referred to in early versions of the Compute API (incorrectly) as a Generator.

#### Slice
A unit of work, represented as source code plus data and meta data, which has a single entry point and return type. Each Slice in a Job corresponds to exactly one element in the Job's input set.

#### Task
A unit of work which is composed of one or more slices, which can be executed by a single worker. Each Slice of each Task will be from the same Job.

#### Work or Work Function
A function which is executed once per Slice for a given Job, accepting the input datum and returning a result which is added to the result set.

#### Module
A unit of source code which can be used by, but addressed independently of, a Work Function. Compute API modules are similar to CommonJS modules.

#### Package
A group of related modules

#### Distributed Computer
A parallel supercomputer consisting of one or more schedulers and workers. When used as a proper noun, the distributed computer being discussed is the one hosted at https://portal.distributed.computer/

#### Bank Account
A ledger which acts a repository for DCC which is not on the block chain. The Bank can move DCC between Bank Accounts much more quickly than it can move DCC between Addresses on the Ethereum block chain network. Meta data attached to bank accounts can restrict certain operations, such as ear-marking funds for use only by job deployment.

#### Address
A unique identifier in DCP which can be used to identity a bank account, user, compute group, etc.

#### Key
A secret number that corresponds to an address. The address can be derived mathematically from the key, but the key cannot be derived from the address. In DCP, access is granted to a resource identitified by an address by signing a message with the key. This a very common public-key-private-key encryption technique, and is why it is important to safeguard your keys.

#### Keystore
A data structure which stores a key. The key can be safeguarded from prying eyes by encrypting it with a passphrase within the keystore.

$3

A file which stores a Keystore.

#### Wallet
A collection of Keystores.