node-crawler

Crawls web urls from a list

Very simple wrapper for puppeteer, with the most basic requirements for a crawler inluded.

Install

npm install @robertsvendsen/node-crawler

If you are getting this:
Error: Failed to launch the browser process! undefined
Fontconfig error: No writable cache directories

https://pptr.dev/troubleshooting#could-not-find-expected-browser-locally

If you had that problem, and you fixed it with ENV during install, you must always keep the environment variable: PUPPETEER_CACHE_DIR=$(pwd)

Example

``
import Crawler, { CrawlerPageOptions } from '@robertsvendsen/node-crawler/src/crawler'

const options = new CrawlerOptions({
name: 'node-crawler-agent',
concurrency: 1,
readRobotsTxt: true,
dataPath: 'data/crawler',
});

const crawler = new Crawler(options);
const links = [{ url: "https://www.google.com" }];

init().then(async () => {
console.info('Crawling complete');
// await delay(10000); // If you have troubles with the script exits before crawling completed make a delay here. The queue is empty but crawling is not.
await crawler.close();
process.exit();
});

async function init() {
const pageOptions = new CrawlerPageOptions({ downloadImages: true });

for (const link of links) {
crawler.add(link.url, pageOptions).then((result) => {
if (result) {
console.info('Crawled', link.url);
}
}

// To avoid saturating the CPU immediately on startup we don't fill the queue up all the way.
await crawler.queue.onSizeLessThan(options.concurrency * 2);
}

await crawler.queue.onEmpty();
}
``

$3

#### CrawlerBrowserOptions
width = 1920; // 3840
height = 1080; // 2160
isLandscape = false;
isMobile = false;
hasTouch = false;

#### CrawlerPageOptions
downloadImages = false;
returnPageInstance = false; // If true, you must close it yourself.
timeout = 10000; // Page load timeout in ms.
waitUntil = 'networkidle2';

#### CrawlerOptions
concurrency = 1;
readRobotsTxt = true;
name = 'node-crawler'; // This should just be the name, no version or anything.
version = '0.1';
email = ''; // contact email for this crawler.
dataPath = 'data';
saveAsPDF = false; // Enable PDF file generation /printing of the site.
saveFiles = true; // Handle this yourself? set to true.
headless = true;

$3

- [x] Concurrency using threads or processes. Actually, it might be possible to just increase the prop because puppeteer
should be able to handle more tabs.
- [ ] Recursive crawling options
- [ ] When crawling recursive, it should handle the robots.txt delay as well.
- [ ] Callback-function in options to determine if a link should be queued (for recursive search)
- Own database (sqllite3)
- [ ] table: sites (site_id, domain, url)
- [ ] table: site_options
- [ ] table: pages (page_id, site_id, path, querystring, last_visited, status_code, redirect_location)
- Logo fetcher (upper left corner, name contains 'logo'?)