Check user agents against robots.txt files
npm install robots-txtHandles checking of robots.txt. handles concurrent requests and caching.
``js
var robots = require('robots-txt'),
level = require('level');
var bot = robots({
db: level('./robots-txt-cache'),
ttl: 1000 60 60 * 24 // one day
});
bot.isAllowed('woobot', 'http://www.woorank.com/my/path')
.then(function (isAllowed) {
// ...
});
``
This module fetches robots.txt files and optionally caches them in a level-db.
parsing and checking of robots.txt files is done by secondary modules robots-txt-parse and robots-txt-guard and can be used separately.