Concurrency using chainable async iterables, such as task pool, event queue, pub/sub, merge, chunk, throttle, etc.
npm install @toolbuilder/await-for-itAwait-For-It implements common concurrency patterns using async iterables. The iterables are chainable for ease of use. Or you can use the async generators on their own with the functional, data-last API.
* Event queues - push events or other data into an async iterable: Queue
* Task pool - process up to n tasks at a time: Pool
* Pub/Sub - fork an async iterable to multiple consumers, register subscribers at any time: Publish
* Join - two iterables should advance together: Zip
* Start/Stop iterables from outside the iterable, like a pseudo-thread: Run
* Catch/Finally - just like Promises, except for streams of Promises: Catch
* Polling - execute a task periodically with backpressure from an async iterable: Poll
* Throttle - limit the processing rate of an async iterable: Throttle
* Merge - Merge multiple async/sync streams with each going as fast as possible: Merge
* Chunk - group results into chunks, with timeouts for partial chunks, for efficient batch processing: Chunk
* Node Streams - works out of the box because Node streams are async iterables now. Also see streams.pipeline method.
* Chainable - chain operations such as map, reduce, filter, throttle, and zip: ChainableClass
* Functional - all operations have a functional 'data last' equivalent via separate imports
* Customize - add/remove methods, and support your bundler's tree-shaking: Customize
If you just want synchronous iterables try IterableFu.
``bash`
npm install --save @toolbuilder/await-for-it
If you want the chainable API, use this import.
`javascript`
import { chainable } from '@toolbuilder/await-for-it'
If you want the functional API, use this import.
`javascript`
import { generators, transforms, reducers } from '@toolbuilder/await-for-it'
Users of both API styles will probably want other methods and classes from the main package such as:
`javascript`
import { Poll, Queue, callWithTimeout } from '@toolbuilder/await-for-it'
* module entry point is here
* The bulk of the chainable documentation is here.
The chainable API is dynamically created from the functional API when Await-For-It is loaded. Underneath, the methods are the same.
* module entry point is the same of course: here
* generators - create sequences of data
* transforms - transform sequences of data
* reducers - reduce or control sequences of data
The documentation is in progress. Sometimes the functional API examples show chainable API use. Sometimes it is the other way around. I will continue improving - especially in areas where you provide feedback.
Await-For-It works with your generators and iterables. It is also possible to add or remove methods from Await-For-It to suit your needs. See customization.
Here is a quick set of examples
After the 1.0.0 changes, there should be no breaking changes going forward.
* 1.0.0 -Queue
* See the updated docs.
* no longer throws QueueFull when the buffer reaches capacity. That's properly the buffer's job.Queue
* constructor no longer accepts a Number to specify buffer capacity. The constructor only accepts a buffer.Queue
* the default buffer is an empty Array although that is probably not what you want.Queue
* no longer provides a capacity propertyRingBuffer
* is no longer exported by Await-For-It. It is now here.Semaphore
* and Mutex are no longer exported by Await-For-It. If you really need them they are here.
Await-For-It is focused on solving common asynchronous patterns with asynchronous iterables. There are many other packages that solve common asynchronous patterns without async iterables. There are also a number of packages that provide async iterable support, but don't seem to fully support async concurrency patterns.
There are lots of packages that support synchronous iterables, but that doesn't help with concurrency. For example, Iterablefu is the synchronous version of Await-For-It.
There are popular Observable libraries. I worked with RxJs and Kefir before writing Await-For-It. After working with both, I strongly prefer async generators and iterators to Observables. Here's why:
* As a JavaScript developer, you already need to learn async iterables. Learning Observables is extra work.yield
* Async iterables are literally iterables that return Promises, so the mental model is simpler.
* The async iterator protocol automatically applies back pressure. Compare to this for RxJs.
Async iterables have direct support in the language: async generators, , async functions, Promises, etc.
* Async iterables work directly with synchronous iterables as input since JavaScript handles that for you.
Observables use a push model, and async iterables use a pull model. Await-For-It provides Queue to bridge from push to pull. You might also look at emittery.
Node streams are now async iterables, so this isn't an either/or decision. The pipeline method might be all you need.
Promise chains work just fine if you don't need to control the number of active tasks, or need to run the tasks sequentially.
for await loops are perfect when you don't need to relax that 'one-at-a-time' behavior. But when you refactor a bunch of nested loops they'll look a lot like the functional or chainable API of Await-For-It.
Contributions are welcome. Please create a pull request.
* I use pnpm instead of npm.
* Run the unit tests with pnpm test
* Package verification requires pnpm to be installed globally.
*
*
* to test against Node ES and CommonJS projects, as well as Electron.pnpm run check` to validate the package is ready for commit
*
This project uses Github issues.
MIT