Showing 1-20 of 472 packages
🤖/👨🦰 Recognise bots/crawlers/spiders using the user agent string.
A set of shared utilities that can be used by crawlers
JavaScript module detecting bots/crawlers/spiders via user-agent
Image crawlers
HTTP request module customized for crawlers.
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
This repository contains a list of of HTTP user-agents used by robots, crawlers, and spiders as in single JSON file.
This is an ES6 adaptation of the original PHP library CrawlerDetect, this library will help you detect bots/crawlers/spiders vie the useragent.
A tiny node module to detect spiders/crawlers quickly and comes with optional middleware for ExpressJS
A tiny node module to detect spiders/crawlers quickly and comes with optional middleware for ExpressJS
Analyze HTML content visibility for AI crawlers and citations - compare static HTML vs fully rendered content
Detect search engine crawlers by their User-Agent strings.
Base of the-crawlers
Pre-rendering Single-Page-Application for crawlers.
React component to protect email addresses from crawlers.
A Node.js server that intelligently serves SPAs with server-side rendering for bots and crawlers
AI bot blocker SDK — shared intelligence network to detect and block AI crawlers
CloudFlare Radar verified bots directory - 500+ web crawlers, search engine bots, and user agents as JSON for bot detection and traffic filtering
Deploy and inspect Findkit crawlers
Make your React SPA SEO-friendly without SSR complexity. Automatically generates static HTML snapshots for search engine crawlers.