!Banner image

n8n WebCrawler API Node

WebcrawlerAPI is a website to markdown API for LLM and RAG. You can get content of the website or a single page with a
simple API call. Check more at WebCrawlerAPI. Our docs are available at WebCrawlerAPI Docs.

This package contains n8n community nodes for integrating with WebCrawler API.

Nodes

$3

The WebCrawlerAPI node allows you to scrape web content using the WebCrawlerAPI service.

#### Features
- Scrape content from any URL using the /v2/scrape endpoint
- Choose output format: Markdown, Cleaned, or HTML
- Remove unwanted elements using CSS selectors
- Use AI-powered prompts to extract specific information from the page

#### Credentials
To use this node, you need a WebCrawlerAPI credential:
- API Key: Obtain from WebCrawlerAPI Dashboard

#### Parameters
- URL to scrape (required): The URL of the page to scrape
- Output format (optional): Markdown (default), Cleaned, or HTML
- CSS selectors to remove (optional): Comma-separated list of selectors (e.g., .ads, .popup, #sidebar)
- Prompt (optional): AI prompt to extract specific information

#### Response Fields
- success: Boolean indicating if the request was successful
- status: Status of the scraping operation
- markdown: Content formatted as Markdown
- cleaned_content: Cleaned text content
- raw_content: Raw HTML content
- page_title: Title of the scraped page

#### Example Usage
1. Add the WebCrawlerAPI node to your workflow
!Add community node
2. Configure your WebCrawlerAPI credentials. Get your API key here.
!Configure credentials
3. Set the URL you want to scrape
4. Optionally, choose output format, add CSS selectors, or a prompt
5. Execute the workflow

#### Installation

``bash
npm install n8n-nodes-webcrawlerapi
`

#### Development

`bash


Install dependencies

npm install

Build the package

npm run build

Run in development mode

npm run dev
``

License

MIT