website-scrap-engine

Configurable website scraper in typescript.

Features

- [x] Resource types
- [x] Configurable process pipeline
- [x] Options
- [x] Logger
- [x] Concurrent downloader
- [x] Multi-thread processing (with native worker_thread)
- [x] Process CSS
- [x] Process HTML
- [x] Process SiteMap (but not replace path in it)
- [x] Configurable logging

Multi-thread processing

Note: use multi-thread processing
only if your process is cpu sensitive.

* Main thread
* resource downloading in queue
* process after download
* save binary resources to disk
* send other resources to worker thread
* enqueue non-duplicated resource from worker thread
* Worker thread
* receive downloaded resource from main thread
* process after download
* parse html, css, etc.
* collect referenced resources
* process and filter referenced resources before download
* send referenced resources to main thread
* save resources to disk

Pipeline life cycle

* skip or redirect link
* detect resource type
* create
* process before download
* download
* process after download
* save to disk