orkid

!Code Coverage
!Dependencies
!Dev Dependencies
!Required Node
!License

Reliable and modern Redis-Streams based task queue for Node.js.

---

Screenshot

!screenshot

---

Table of Contents

- Features
- Requirements
- Install
- Examples
- API Documentation
- Monitoring and Management UI/Admin Panel
- FAQ

---

Features

- [x] Orkid lets Redis do the heavy lifting with Redis-Streams.
- [x] Adjustable concurrency per consumer instance for scaling task processing. See example code. See example code.
- [x] Job timeouts and retries. All configurable per consumer. See example code.
- [x] Task Deduplication. If a task is already waiting in the queue, it can be configured to avoid queueing the same task again. _(Useful for operations like posting database record updates to elasticsearch for re-indexing. Deduplication is a common pattern here to avoid unnecessary updates)_. See example code.
- [x] Add tasks in bulk in one call. The producer will handle chunking and optimize round-trips to redis using pipelining.
- [x] Monitoring and management UI for better visibility.

- [ ] Rate-limiting workers. (_work in progress_)

---

Requirements

- Node.js >= 10
- Redis >= 5

👏 Important: Redis-Streams feature is not available before Redis version 5.

---

Install

``
npm install orkid --save
`

---

Examples

Basic example of producing and consuming tasks:

Producing tasks:

`js
const { Producer } = require('orkid');

// basic is the queue name here
// We'll use the same name in the consumer to process this task
const producer = new Producer('basic');

async function addTasks() {
for (let i = 0; i < 10; i++) {
console.log('Adding task ', i);
await producer.addTask(i);
}
}

addTasks()
.then(() => {
console.log('Done');
process.exit(); // To disconnect from redis
})
.catch(e => console.error(e));
`

Consuming tasks:

`js
const { Consumer } = require('orkid');

// Worker Function
async function workerFn(data, metadata) {
let result;
/*
Do operation on data here
and store the result in result variable

Anything you return from this function will
be saved in redis and can be viewed in the Orkid UI.

Returning nothing is fine too.

Throwing error will mark the job as failed,
which can be retried too.
*/

console.log('Task done!');
return result;
}

// Consume task from the basic queue
const consumer = new Consumer('basic', workerFn);

// Start processing tasks!
// Important: Until you call this method, orkid consumer will do nothing.
consumer.start();
`

> 👏 More examples are available in the ./examples directory, including how to do task de-duplication, retry on failure, timeout etc. 👏

---

API Documentation

API Documentation is available here.

---

Monitoring and Management UI/Admin Panel

!screenshot
!screenshot

> You need to run orkid-ui separately for the dashboard. Detail instructions on how to run orkid-ui` locally or in production using docker/docker-compose can be found here:
> https://github.com/mugli/orkid-ui#running-locally

---

Task/job life-cycle

[TODO: Add a flowchart here]

---

FAQ

---

Maintainer(s)

- Mehdi Hasan Khan (@MehdiHK)

---

License

MIT

---

$3

- orkid-ui: Dashboard to monitor and manage Orkid task queue.
- orkid-api: GraphQL API to monitor and manage Orkid task queue (used internally by orkid-ui).