WebcrawlerAPI MCP Server

WebcrawlerAPI is a powerful web scraping and website crawling service that allows you to extract data from any website with ease.

This MCP (Model Context Protocol) server integrates WebcrawlerAPI's scraping capabilities directly into MCP-compatible applications like Claude Code, enabling integration within your AI workflows. Get started with your free API key at the WebcrawlerAPI Dashboard.

Setup

1. Get your API key from WebcrawlerAPI Dashboard
2. Set the environment variable:
``bash
export WEBCRAWLER_API_KEY="your-api-key-here"
`

Using with Claude Code

Add this server to your Claude Code configuration using npx:

`bash
npx webcrawler-mcp
`

Or add to your MCP settings configuration:

`json
{
"mcpServers": {
"webcrawler": {
"command": "npx",
"args": ["-y", "webcrawler-mcp"],
"env": {
"WEBCRAWLER_API_KEY": "your-api-key-here"
}
}
}
}
`

Available Tools

$3

Scrapes content from any webpage and returns it as markdown.

Parameters:
- url (required): The URL of the webpage to scrape
- prompt (optional): Optional prompt to extract specific information from the page

Example usage:
`
Use the webcrawler-scrape tool to get the content from https://example.com
`

With a prompt for targeted extraction:
`
Use the webcrawler-scrape tool to scrape https://news.ycombinator.com with the prompt "Extract all article titles and their corresponding URLs"
`

The tool returns the scraped content as markdown, along with the page title and HTTP status code. When using a prompt, the API will focus on extracting the specific information you requested.

Running as Standalone App

You can also run the server as a standalone application:

$3


`bash
npm install
npm run build
npm start
`

$3

`bash
npx webcrawler-mcp
`

$3

To run in HTTP mode instead of stdio:
`bash
USE_HTTP=true npx webcrawler-mcp

or

npm run start:http
`

The HTTP server will start on port 8080 by default, with endpoints:
- Health check: http://localhost:8080/health
- MCP endpoint: http://localhost:8080/mcp

Environment Variables

- WEBCRAWLER_API_KEY: Your WebcrawlerAPI.com API key (required)
- USE_HTTP: Set to "true" to use HTTP transport instead of stdio
- PORT: HTTP server port (default: 8080)

Development

`bash
npm install
npm run dev # Watch mode
npm run build
``