iter-duct-utils

iter-duct-utils
===============
Useful pipeline segments for iter-duct.

How to read the documentation
-----------------------------
For each segment there is a description, the type of segment (reader, transformer or writer) and the configuration.
When specified "value or function", the config parameter can get a function that takes as argument the current item returned by the iterable.

Types of segment
----------------
* Reader: it reads the data somewhere, it ignores the iterable in input
* Transformer: it takes the iterable in input and returns another as output
* Writer: it writes the data somewhere
* Special: it implements some special feature

All types returns an iterable, even the "writer" let the iterable in input pass through unchanged.

iter-tools transformers
-----------------------
type: transformer

iter-tools includes many useful pipeline segments. Here are some examples:

``js
const it = require('iter-tools/es2018')

// map every item to a new object
it.asyncMap((obj) => ({ fullname: ${obj.firstname} ${obj.lastname} }))

// filter out all items that returns false
it.asyncFilter((obj) => obj.include)

// it filters all items before the function is true for the first time
it.asyncTakeWhile((obj) => obj.include)

// it drops all items after the function returns true for the first time
it.asyncDropWhile((obj) => obj.exclude)

// slice the iterable (you can define start, end and step)
it.asyncSlice({ start: 1 })

// it generate multiple items from a single one
it.asyncFlatMap((item) => [item, { ... item, clone: true }])

// throttle iterables to 1 sec for each
it.asyncThrottle(1000)
`

You can find how they work in iter-tools documentation.
For example:
`js
const it = require('iter-tools/es2018')
const pipeline = [
...
itools.asyncMap((item) => { ... })
...
]
`

logger
------
type: writer

It logs the current item.
Arguments:
* template: string or function

mongo-reader
------------
type: reader/transformer

It runs a "find" query on mongo and returns the results.
If this receives an iterable, it can run a "find" query for each item.

* host: mongo host, example 'mongodb01-az1.live.xxx.com' string
* user: example 'resource_xxx' string
* password: example: 'xxx' string
* collection: collection name, example 'resources' string
* authDb: db used for authentication string
* db: database name, example 'resource' string
* query: find query, example{ authorId: 123 } object or function

mongo-update
------------
type: writer

It updates a document in a mongo collection (using collection.updateOne from the mongodb library). It returns the same object.

* host: mongo host, example 'mongodb01-az1.live.xxx.com' string
* user: example 'resource_xxx' string
* password: example: 'xxx' string
* collection: collection name, example 'resources' string
* db: database name, example 'resource' string
* authDb: db used for authentication string
* query: find query, example{ authorId: 123 } object or function
* doc: an object containing the fields to update object or function
* options: options for the "updateOne" method object or function

mongo-insert
------------
type: writer

It adds a document in a mongo collection (using collection.insertOne from the mongodb library). It returns the same object.

* host: mongo host, example 'mongodb01-az1.live.xxx.com' string
* user: example 'resource_xxx' string
* password: example: 'xxx' string
* collection: collection name, example 'resources' string
* db: database name, example 'resource' string
* authDb: db used for authentication string
* doc: an object containing the fields to update object or function
* options: options for the "insertOne" method object or function

csv-reader
----------
type: reader

It reads a csv and returns an item for each row (as array of objects).
It can take all options of csv-parse: https://www.npmjs.com/package/csv-parse

* filename: the csv file name, example 'file.csv' string
* encoding: CSV encoding. It defaults to utf8 string
* ...see: https://www.npmjs.com/package/csv-parse

csv-writer
----------
type: writer

It writes a csv, line by line.
It can take all options of csv-write-stream: https://www.npmjs.com/package/csv-write-stream

* filename: the csv file name, example 'file.csv' string
* ...see: https://www.npmjs.com/package/csv-write-stream

download
--------
type: writer

It downloads a file for each iterable.

* url: string or function
* filename: 'string or function
* skipExisting: (default false)
* concurrency: (default 4)

getJSON
-------
type: reader

It reads a json from an url and push it into the pipeline

* url: string or function
* headers: HTTP headers object or function
* concurrency: (default 4)

postJSON
--------
type: writer

It writes a json and push the result into the pipline

* url: string or function
* payload: object or function
* headers: HTTP headers object or function
* method: (default 'POST') string
* concurrency: (default 4)

postForm
--------
type: writer

It writes a file to an endpoint (uses multipart/formdata).

* url: string or function
* formData: object or function
* headers: HTTP headers object or function
* method: (default 'POST') string
* concurrency: (default 4)

formData should return an array of objects with this shape:
`js
[
{
name: 'field1',
value: 'xxx'
},
{
name: 'field2',
value: 'yyy'
},
{
name: 'file',
value: path.join(__dirname, './test.jpg'),
options: { contentType: 'image/jpeg', filename: 'test.jpg' },
attachment: true
}
]
``

json-reader
-----------
type: reader

It reads a json

* files: a path using globbing string

json-writer
-----------
type: writer

It writes a json file for each item.

* filename: json path name value or string

readfile
--------
type: reader

It reads a file and either split it or return matches

* filename: path name value or string
* split: regexp to split the text regexp
* extract: regexp to extract text from a file regexp

exec
----
type: reader

It runs a command and reads standard output, then it either split it or return matches:

* cmd: command to run value or string
* split: regexp to split the output regexp
* extract: regexp to extract text from the output regexp