claw

claw
===

A very simple web scraper chassis.

Takes:

- a page url
- a selection to scrape
- fields to pull out from within that section
- an output folder
- number of seconds to delay

and it creates CSV and JSON files with the results. Claw creates a separate file for each page it scrapes.

// libararies
var claw = require('claw');

// get settings
var page = 'http://www.bing.com/search?q=hello';

var selector = 'h3 a';

var fields = {
"text" : "text()",
"href" : "attr('href')"
};

claw(page, selector, fields, 'output', 3);

Give it an array of pages, and it will save the results of each page to a separate file.

claw(['http://www.bing.com/search?q=hello', 'http://www.bing.com/search?q=goodbye'], selector, fields, 'output', 3);

Claw can also grab its page list from JSON file that is a list of urls (or an object with .href properties).

claw("pages.json", selector, fields, 'output', 3);

Questions? Ideas? Hit me up on twitter - @dylanized