Crawl to discover all URL locations defined in a sitemap or sitemap index
npm install crawl-sitemapCrawl to discover all URL locations defined in a sitemap or sitemap index.
``js
import crawl from 'crawl-sitemap';
for await (const url of crawl('https://andrewduthie.com/sitemap.xml')) {
console.log(url);
}
// → https://andrewduthie.com/2019/11/29/back-to-basics-a-possible-future-without-javascript-build-tools/
// → https://andrewduthie.com/2020/03/15/typescript-types-for-productivity/
// → https://andrewduthie.com/2019/12/22/font-loading-performance-for-single-page-applications/
// ...
`
crawl-sitemap is authored as an ESM module, and therefore requires Node 12.0 or newer.
Install using NPM or Yarn:
`
npm install crawl-sitemap
`
yarn add crawl-sitemap
The default export will return an async generator when called, which can be iterated asynchronously with for await to operate on crawled links as soon as they're discovered:
`js`
for await (const url of crawl('https://andrewduthie.com/sitemap.xml')) {
console.log(url);
}
`ts``
function crawl(url: string): AsyncGenerator
Copyright 2020 Andrew Duthie
Released under the MIT License. See LICENSE.md.