A performant multi-curl class in Node.js that supports proxies and concurrent requests
npm install node-multi-curlA performant multi-curl class in Node.js that supports working with proxies and concurrent requests.
- Parallel HTTP requests with concurrency control
- Proxy support with automatic rotation
- Docker integration for HTTP/2 support
- Axios-like API for ease of use
- Request/response interceptors
- Automatic retry mechanism
- HTTP/2 support** via Docker curl integration
- Cloudflare capabilities with advanced header management
- Brotli, Gzip and Deflate** compression support
``bash`
npm install node-multi-curl
`javascript
const { MultiCurl } = require('node-multi-curl');
// Simple GET request
const response = await MultiCurl.get('https://example.com/api');
// Multiple concurrent requests
const client = new MultiCurl({ concurrency: 5 });
const responses = await client.execute([
{ url: 'https://example.com/api/users' },
{ url: 'https://example.com/api/products' },
{ url: 'https://example.com/api/orders' }
]);
`
`javascript
const client = new MultiCurl({
curlConfig: {
useDocker: true,
dockerImage: 'badouralix/curl-http2:latest'
}
});
// This request will use HTTP/2 if the server supports it
const response = await client.get('https://http2.pro/api/v1');
// Brotli, Gzip and other compression methods are automatically handled
`
`javascript
// Configure with specialized headers
const client = new MultiCurl();
client.interceptors.request.use(request => {
request.headers = {
...request.headers,
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8',
'Accept-Language': 'en-US,en;q=0.5',
'Accept-Encoding': 'gzip, deflate, br',
'Connection': 'keep-alive',
'Upgrade-Insecure-Requests': '1',
'Sec-Fetch-Dest': 'document',
'Sec-Fetch-Mode': 'navigate',
'Sec-Fetch-Site': 'none',
'Sec-Fetch-User': '?1'
};
return request;
});
// Now you can access Cloudflare-protected sites
const response = await client.get('https://cloudflare-protected-site.com');
`
`javascript
// Using query parameters
const response = await MultiCurl.get('https://httpbin.org/get', {
queryParams: {
param1: 'value1',
param2: 123,
param3: true,
nullParam: null, // This will be skipped
}
});
// Makes a request to: https://httpbin.org/get?param1=value1¶m2=123¶m3=true
// Or with the execute method:
const client = new MultiCurl();
const responses = await client.execute([
{
url: 'https://httpbin.org/get',
queryParams: { search: 'test', page: 1 }
}
]);
``
For detailed documentation and examples, check the examples directory.
MIT