Fully asynchronous iterable-based async pool for JS
npm install eager-async-poolPromise and iteratorsnpm install eager-async-pool or yarn add eager-async-pool``javascript
const { asyncPool } = require('eager-async-pool')
// exceptionally useful executor
const executor = (n) => 'Number is: ' + n
// absolutely realistic dataset
const items = [1, 2, 3, 4]
for await (const { value, error } of asyncPool(executor, items, { limit: 2 })) {
if (error) {
console.error(error)
} else {
console.log(value)
}
}
`
javascript
const executor = (url) => fetch(url).then(i => i.text())
`Then, create an array (or any iterable) that contains the data that will be passed to
the executor one-by-one:
`javascript
const urls = ['https://google.com', 'https://yandex.ru', 'https://bing.com']
`Finally, call the
asyncPool and handle the results!
`javascript
for await (const { idx, item, value, error } of asyncPool(executor, urls)) {
console.log(${item} title: ${cheerio.load(value).find('title').text()})
}
`Alternatively, you can use
asyncPoolCallback that wraps over asyncPool and allows
you to specify a callback rather than using for-await loop:
`javascript
await asyncPoolCallback(executor, urls, ({ idx, item, value, error }) => {
console.log(${item} title: ${cheerio.load(value).find('title').text()})
})
`$3
Of course, while processing items you may very well encounter some kind of error.
Luckily, it is very easy to handle them with eager-async-pool!When
executor throws an error, iterator will yield an object containing .error,
which will contain that very error. To retry the failed item, just push it to the array!
`javascript
for await (const { idx, item, value, error } of asyncPool(executor, urls)) {
if (error) {
urls.push(item)
continue
}
// ...rest of the logic...
}
`$3
eager-async-pool also supports cancel tokens. They are extremely easy to use as well.
Just use CancelToken.source(), pass the token to asyncPool and .cancel() when you feel like:
`javascript
const cancel = CancelToken.source()
for await (const { idx, item, value, error } of asyncPool(executor, urls, { cancel: cancel.token })) {
if (error) {
cancel.cancel('An error has occurred')
break
}
console.log(${item} title: ${cheerio.load(value).find('title').text()})
}
`> Note:
break in an iterable asyncPool will also effectively cancel the pool,
> but break is (for obvious reasons) not available in asyncPoolCallback.
>
> Also, cancel token is compatible with other libraries that support it,
> so you can use it inside executors as well.> Note: cancelling an async pool only guarantees that executor will not be called
> anymore. It DOES NOT cancel pending operations, nor does it ignore them.
> If you want to cancel pending operations as well, you must handle
> cancel token inside executor manually (or provide it to some library).
Motivation
Consider the common task of batch file upload/download.
You could try to do that using built-in Promise.all()`, but that's probably not Also, when downloading you might encounter an error (e.g. rate limit), and you
won't want that error to interrupt the entire download process, and instead you'd want
to retry downloading that particular file.
You may also want to do something immediately after the file is downloaded, and not wait
until all other files are downloaded as well.