Multi environment web page parser
npm install goose-parser





This tool moves routine crawling process to the new level.
Now it's possible to parse a web page for a moment.
All you need is to specify parsing rules based on css selectors. It's so simple as Goose can do it.
This library allows to parse such data types as Grid, Collections, and Simple objects.
Parser has support of pagination by extension goose-paginator.
Also it offers you following features: actions to interact with the page and transforms to convert parsed data to friendly format.
``bash`
yarn add goose-parser goose-chrome-environment
`JS
const Parser = require('goose-parser');
const ChromeEnvironment = require('goose-chrome-environment');
const env = new ChromeEnvironment({
url: 'https://www.google.com/search?q=goose-parser',
});
const parser = new Parser({ environment: env });
(async function () {
try {
const results = await parser.parse({
actions: [
{
type: 'wait',
timeout: 10 * 1000,
scope: '.srg>.g',
parentScope: 'body'
}
],
rules: {
scope: '.srg>.g',
collection: [[
{
name: 'url',
scope: 'h3.r>a',
attr: 'href',
},
{
name: 'text',
scope: 'h3.r>a',
}
]]
}
});
console.log(results);
} catch (e) {
console.log('Error occurred:');
console.log(e.stack);
}
})();
`
For now it's available to run goose-parser as a docker service.
Params:
url* - first param is an url to parser
Parsing rules [optional] - Rules to parse. It's optional, if --rules-file* specified.
Options:
-e "DEBUG=" - to enable debug mode and see all what happens inside the goose-parser. Reed more about debug here.
--rules-file* - to specify rules file. Be aware that you need to mount a folder with rules as a volume to the docker container.
There are two options to run it:
`bash`
docker run -it --rm -e "DEBUG=,-puppeteer:" redcode/goose-parser:chrome-1.1.3-parser-0.6.0\
https://www.google.com/search?q=goose-parser\
'{
"actions": [
{
"type": "wait",
"scope": ".g"
}
],
"rules": {
"scope": ".g",
"collection": [
[
{
"scope": ".r>a h3",
"name": "name"
},
{
"scope": ".r>a:eq(0)",
"name": "link",
"attr": "href"
}
]
]
}
}'
Create a file rules/rules.json which contains parser rules and run following command:
`bashpwd
docker run -it --rm --volume="/rules:/app/rules:ro" -e "DEBUG=,-puppeteer:" redcode/goose-parser:chrome-1.1.3-parser-0.6.0 --rules-file="/app/rules/rules.json" 'https://www.google.com/search?q=goose-parser'``
API reference - coming soon