library for building node-red flows that supports blocks, high volume
npm install node-red-biglib"Big Lib" is the core library for "Big Nodes"
bash
npm install node-red-biglib
`Principles for Big Nodes
See biglib for details on Big Nodes.
and subsequent Big Nodes are a family of nodes built for my own purpose. They are all designed to help me build a complete process for production purposes. For that I needed nodes able to:* Flow big volume of data (memory control, work with buffers)
Work with a flow of blocks* (buffers) (multiple payload within a single job)
Tell what they are doing* with extended use of statuses (color/message)
Use their second output for flow control* (start/stop/running/status)
Reuse messages* in order to propagate _msgid, topic
* Depends on state of the art libraries for parsing (csv, xml, xlsxs, line, ...)
* Acts as filters by default (1 payload = 1 action) or data generators (block flow)
All functionnalities are built under a library named
biglib and all Big Nodes rely on itThe Big Nodes family
This library is used for big nodes developpement.
Here is the family
Dependencies
byline simple line-by-line stream reader
filesize provides an easy way to get a human readable file size String
moment Parse, validate, manipulate, and display dates in JavaScript
API
Work in progress. Please look at the following
Big Nodes:* big file for reading files in multiple ways/format/encoding
* big csv for parsing CSV
* big line for parsing by lines
* big fixed for fixed format parsing
* big status for visual statuses
* big exec for executing commands
* big ssh for executing remote command over SSH
* big mongo fork from node-red-contrib-mongodb2 for executing MongoDB requests with cursor support
* [big db2]() in progress
* big xlsx for parsing XLSX files
* big splitter fork from node-red-contrib-splitter with big control support
* big tail used for converting multiple payloads in one message with line and size constraints
Sample use of Big Nodes
$3
It uses several nodes which are needed for this to work:
`bash
npm install node-red-contrib-bigline
npm install node-red-contrib-bigtail
npm install node-red-contrib-bigexec
npm install node-red-contrib-bigsplitter
npm install node-red-contrib-bigstatus
npm install node-red-contrib-traffic
`json
[{"id":"c5ca4d63.3a35b","type":"bigtail","z":"894b4f7b.76b4b","name":"","size":"2","add_cr":true,"size_kbyte":"","x":620,"y":60,"wires":[["449b97f1.bb6468"],[]]},{"id":"dd148b1f.22eb78","type":"bigline","z":"894b4f7b.76b4b","name":"","filename":"","format":"utf8","keepEmptyLines":false,"x":480,"y":60,"wires":[["c5ca4d63.3a35b"],["c5ca4d63.3a35b"]]},{"id":"99748aff.668b78","type":"debug","z":"894b4f7b.76b4b","name":"tail stdout","active":true,"console":"false","complete":"payload","x":1000,"y":80,"wires":[]},{"id":"449b97f1.bb6468","type":"traffic","z":"894b4f7b.76b4b","name":"","property_allow":"control.ok","filter_allow":"true","ignore_case_allow":false,"negate_allow":false,"send_allow":false,"property_stop":"control.state","filter_stop":"start","ignore_case_stop":false,"negate_stop":false,"send_stop":false,"default_start":false,"differ":false,"x":850,"y":80,"wires":[["99748aff.668b78"]]},{"id":"9323a207.6cdc6","type":"bigline","z":"894b4f7b.76b4b","name":"","filename":"","format":"utf8","keepEmptyLines":false,"x":480,"y":220,"wires":[["d991b04e.266e5"],["d991b04e.266e5"]]},{"id":"d991b04e.266e5","type":"bigtail","z":"894b4f7b.76b4b","name":"","size":"3","add_cr":true,"x":620,"y":220,"wires":[["4ac88b64.b53774"],[]]},{"id":"4ac88b64.b53774","type":"traffic","z":"894b4f7b.76b4b","name":"","property_allow":"control.ok","filter_allow":"true","ignore_case_allow":false,"negate_allow":true,"send_allow":false,"property_stop":"control.state","filter_stop":"start","ignore_case_stop":false,"negate_stop":false,"send_stop":false,"default_start":false,"differ":false,"x":850,"y":200,"wires":[["83cde6d9.7c3218"]]},{"id":"83cde6d9.7c3218","type":"debug","z":"894b4f7b.76b4b","name":"tail stderr","active":true,"console":"false","complete":"payload","x":1000,"y":200,"wires":[]},{"id":"109cae54.ef6352","type":"inject","z":"894b4f7b.76b4b","name":"no file","topic":"","payload":"non/existent","payloadType":"str","repeat":"","crontab":"","once":false,"x":130,"y":160,"wires":[["5e919a71.a16e64"]]},{"id":"dbc04ed6.243fb","type":"bigstatus","z":"894b4f7b.76b4b","name":"","locale":"","x":850,"y":140,"wires":[["cfd54729.302ab8"]]},{"id":"5e919a71.a16e64","type":"bigexec","z":"894b4f7b.76b4b","name":"cat","command":"cat","commandArgs":"","minError":1,"minWarning":1,"cwd":"","shell":"","extraArgumentProperty":"","envProperty":"","format":"utf8","limiter":true,"payloadIs":"argumentNoStdin","x":290,"y":140,"wires":[["dd148b1f.22eb78"],["dd148b1f.22eb78","449b97f1.bb6468","9323a207.6cdc6","4ac88b64.b53774","dbc04ed6.243fb"],["9323a207.6cdc6"]]},{"id":"cfd54729.302ab8","type":"filter","z":"894b4f7b.76b4b","name":"ok?","property":"control.ok","filter":"true","ignorecase":true,"x":990,"y":140,"wires":[["d05541bf.2faac"],["fbb38f7b.044c7"],[]]},{"id":"d05541bf.2faac","type":"debug","z":"894b4f7b.76b4b","name":"next in the flow","active":true,"console":"false","complete":"payload","x":1160,"y":120,"wires":[]},{"id":"fbb38f7b.044c7","type":"debug","z":"894b4f7b.76b4b","name":"KO, other action","active":true,"console":"false","complete":"payload","x":1160,"y":160,"wires":[]},{"id":"6d4a788a.92b588","type":"comment","z":"894b4f7b.76b4b","name":"Change to suit your conf","info":"","x":170,"y":100,"wires":[]},{"id":"321868ab.cde798","type":"comment","z":"894b4f7b.76b4b","name":"Sample use of big nodes","info":"","x":130,"y":40,"wires":[]},{"id":"34616222.cb9e9e","type":"inject","z":"894b4f7b.76b4b","name":"test file","topic":"","payload":"test.csv","payloadType":"str","repeat":"","crontab":"","once":false,"x":130,"y":120,"wires":[["5e919a71.a16e64"]]}]
``- Jacques W
This code is Open Source under an Apache 2 License.
You may not use this code except in compliance with the License. You may obtain an original copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. Please see the
License for the specific language governing permissions and limitations under the License.
Please report any issues or suggestions via the Github Issues list for this repository.
For more information, feedback, or community support see the Node-Red Google groups forum at https://groups.google.com/forum/#!forum/node-red