A tiny node module to detect spiders/crawlers quickly and comes with optional middleware for ExpressJS
npm install @nodebb/spider-detector 
A tiny node module to detect spiders/crawlers quickly and comes with optional middleware for ExpressJS
It might be useful when you have a single page app but want to deliver static pages for spiders.
``
npm install spider-detector // or
`js`
const detector = require('spider-detector')
detector.isSpider('baiduspider') // return true
`js
const detector = require('spider-detector')
const express = require('express')
const app = express()
app.use(detector.middleware())
app.get('/*', function(req, res) {
if (req.isSpider()) {
// do something else, i.E. send a static page
} else {
// send single page app
}
})
`
Well, I wanted one which does not use readFileSync and comes with optional middleware. Furthermore some hackers do not classify Googlebot as a spider anymore which poses a problem sometimes, see next question.
Yep, Googlebot is able to deal with single page apps but this feature is pretty unstable. Especially under AngularJS when hash fragments are disabled with $locationProvider.html5Mode(true). That's why - against all odds - I have classified googlebot` as a spider in this module.