MongoDB queue implementation for simplecrawler
npm install simplecrawler-mongo-queue
This is a queue implementation for simplecrawler powered by MongoDB.
``
npm install --save simplecrawler-mongo-queue
First of all, create a new Simplecrawler instance as described in the documentation. Then create the queue instance and assign it to crawler.queue property.
`javascript
const Crawler = require('simplecrawler');
const MongoQueue = require('simplecrawler-mongo-queue');
(async () => {
const crawler = new Crawler('http://example.com');
crawler.queue = await MongoQueue.create(datastore, name);
crawler.start();
})();
`
The create method returns MongoQueue instance and has two arguments:
- datastore - the application should provide a MongoDB collection where the queue will be stored.name` (optional) - a name of the queue to distinguish the different crawlers. If the argument is omitted the constructor creates a random queue name.
-