simple-node-crawler

A simple web crawler on node.

features

- persistent files on local disk or database (currently support mongodb)
- resume and continue
- multiple paths with content extraction patterns
- convert to markdown
- auto detect html encoding
- save images

install

npm install simple-node-crawler

usage

var Crawler = require('simple-node-crawler');

var c = new Crawler({
host:'developer.51cto.com',
patterns: [{'path': 'art/', 'pattern': '.m_l' }],
usedb: true,
saveImage: true
}).start('http://developer.51cto.com/col/1308/');

configuration

- host - host constraint for the crawling.
- patterns - if you want to crawl a specific path, you can specify the path name or leave it as ''; the pattern is the css patterns for the main body of the webpage, id/class/tag name are supported, if you need all the html body, you can specify 'body'.
- usedb - if you want to use local file system, then set to false. If you have mongodb installed and want to use it, then set to true.
- saveImage - whether to save images to local file system.
- dbConnectionString - mongodb connection string, default to 'mongodb://localhost/test'
- utf8 - whether need to convert to uft8. Default to true.
- crawlerNumber - how many cralwer thread you want to have. Default to 5.

feature to be implemented

- keyword analysis & extraction

license

MIT