A simple Node.js SDK for the Bright Data API
npm install akbdsdkA simple Node.js SDK for the Bright Data API.
``bash`
npm install akbdsdk
1. Create a new file example.js:`javascript
const { BrightDataClient } = require('akbdsdk');
// STEP 1: Initialize the client with your API key
// Replace 'your-api-key-here' with your actual Bright Data API key
const client = new BrightDataClient({
apiKey: 'your-api-key-here'
});
async function scrapeWebsite() {
try {
// STEP 2: Trigger the collection
// Replace 'your-dataset-id' with your actual Bright Data Dataset ID
const triggerResponse = await client.triggerCollection({
dataset_id: 'your-dataset-id',
urls: [{
// Replace these values with your target website and requirements
url: 'https://www.example.com',
prompt: 'Extract all content',
country: 'US'
}]
});
console.log('Collection started. Snapshot ID:', triggerResponse.snapshot_id);
// STEP 3: Wait for the snapshot to be ready
const status = await client.waitForSnapshot(triggerResponse.snapshot_id, {
maxAttempts: 10,
delayMs: 30000,
onProgress: (status) => {
console.log('Progress:', {
pages_crawled: status.progress?.pages_crawled || 0,
pages_extracted: status.progress?.pages_extracted || 0,
pages_failed: status.progress?.pages_failed || 0
});
}
});
console.log('Snapshot is ready:', status.status);
// STEP 4: Download the results
const results = await client.downloadSnapshot({
job_id: triggerResponse.snapshot_id,
format: 'json'
});
console.log('Scraping completed successfully!');
console.log('Results URL:', results.url);
return results;
} catch (error) {
console.error('Error:', error);
throw error;
}
}
// Run the example
scrapeWebsite().catch(console.error);
`
2. Update the following values in the code:
- Replace 'your-api-key-here' with your Bright Data API key'your-dataset-id'
- Replace with your Bright Data Dataset ID
- Update the URL, prompt, and country code as needed
3. Run the example:
`bash`
node example.js
1. STEP 1: Initialize the Client
- Create a connection to the Bright Data API
- Replace the API key with your actual key
2. STEP 2: Trigger Collection
- Start the scraping process
- Set your dataset ID and target website details
3. STEP 3: Wait for Snapshot
- Monitor the scraping progress
- See real-time updates on pages crawled and extracted
4. STEP 4: Download Results
- Get the URL to access your scraped data
- Data is available in JSON format
If you encounter errors, here are some common issues and solutions:
1. API Key Issues
`javascript`
// Make sure your API key is correct and properly formatted
const client = new BrightDataClient({
apiKey: 'your-api-key-here' // Should be a valid Bright Data API key
});
- Verify the API key in your Bright Data dashboard
- Ensure the key has the necessary permissions
- Check if the key is properly formatted (no extra spaces or quotes)
2. Dataset ID Issues
`javascript`
// Verify your dataset ID exists and is correct
const triggerResponse = await client.triggerCollection({
dataset_id: 'your-dataset-id' // Should be a valid dataset ID
});
- Check the dataset ID in your Bright Data dashboard
- Ensure the dataset is properly configured
- Verify the dataset is active and accessible
3. API Endpoint Issues
- Check if you can reach the Bright Data API:
`bash`
curl -I https://api.brightdata.com
- Verify your network connection
- Check if your firewall allows outbound HTTPS connections
- Ensure you're using the correct API endpoint
4. Common Error Messages
- "API key is required" - You forgot to provide an API key
- "Dataset ID is required" - You forgot to provide a dataset ID
- "No response received" - Network connectivity issue
- "Unknown error occurred" - Check the API response details in the logs
- "Invalid API key" - Your API key is not valid
- "Dataset not found" - Your dataset ID is incorrect
5. Debugging Tips
- Enable verbose logging in your application
- Check the network tab in your browser's developer tools
- Verify the request payload matches the API requirements
- Test the API endpoint with a tool like Postman or curl
The SDK throws errors with descriptive messages. Always wrap your calls in try-catch blocks:
`javascript``
try {
const results = await scrapeWebsite();
console.log(results);
} catch (error) {
console.error('Scraping failed:', error);
// Additional error details are logged automatically
}
MIT