Showing 1-20 of 147 packages
Express middleware that returns the resulting html after executing javascript, allowing crawlers to read on the page
koa detect googlebot, facebookbot, ...
Verify that a request is from Google using Google's recommended DNS verification steps
🤖/👨🦰 Recognise bots/crawlers/spiders using the user agent string.
Validate urls against robots.txt rules.
koa detect googlebot, facebookbot ...
Streaming parser for robots.txt files
A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.
Really simple way to add reveal on scroll animation to your React app.
Tame the robots crawling and indexing your Nuxt site with ease.
Handle http request from crawler (Something like Googlebot)
Simple robots.txt file generation for express
Verifies Googlebot, using Google's recommended method (reverse DNS). This middleware also takes performance into account, and caches the results.
Awesome generator robots.txt
A Slack bot that reminds people to google things before asking their co-workers
<h3>Welcome!</h3> <p>Seo Server is a command line tool that runs a server that allows GoogleBot(and any other crawlers) to crawl your heavily Javascript built websites. The tool works with very little changes to your server or client side code.</p> <p><i>This entire site is driven by Javascript(view the source or see the <a href="https://github.com/apiengine/seoserver-site">code</a>). Click the `What does Google see?` button at the bottom of each page to see Seo Server in action.</i></p>
This repository contains a list of of HTTP user-agents used by robots, crawlers, and spiders as in single JSON file.
JavaScript module detecting bots/crawlers/spiders via user-agent
Detect user-agent is a bot/spider/crawler
TypeScript definitions for woothee