paged-request    

> Simplified requests for paged (paginated) content.

Please consider following this project's author, Jon Schlinkert, and consider starring the project to show your :heart: and support.

Install

Install with npm:

``sh
$ npm install --save paged-request
`

Heads up!

See the release notes for information about changes made in v2.0.

Usage

This library recursively calls needle's .get method as long as the user-provided next() function returns a string (the next url to get). See an example.

Example

`js
const request = require('paged-request');

request(url, options, next)
.then(acc => console.log(acc.pages.length))
.catch(console.error);
`

$3

* url {string} - (required) the initial url to get
* options {object} - (optional) options object to pass to needle
* next {function} - (required) function that returns the next url to get, a promise or undefined.

$3

* url {string} - the original (base) user-provided url
* resp {object} - needle response object
* acc {object} - accumulator object with the following properties:
- options {object} - user-provided options object
- pages {array} - array of responses
- urls {array} - array of requested urls

The next function should return a string (the next url to get), promise or undefined.

Example

The following example shows how to loop over pages of CSS posts on smashingmagazine.com (an arbitrary example, but they have great content!).

`js
const request = require('paged-request');

async function next(url, resp, acc) {
// do stuff to check response first if necessary
const regex = /href="\/.*?\/(\d+)\/"/;
const num = (regex.exec(resp.data) || [])[1];

if (num && /^[0-9]+$/.test(num) && +num <= n) {
// use the "original" url to avoid having to reparse
// and recreate the url each time
return ${acc.orig}/page/${num}/;
}
}

request('https://www.smashingmagazine.com/category/css', {}, next)
.then(acc => console.log(acc.pages.length))
.catch(console.error);
`

Release notes

$3

* renamed .hrefs to .urls in response object
* now using axios instead of needle. Please see the axios documentation for API information.

About

$3

You might also be interested in these projects:

* gists: Methods for working with the GitHub Gist API. Node.js/JavaScript | homepage
* github-base: Low-level methods for working with the GitHub API in node.js/JavaScript. | homepage
* repos: Tiny wrapper around github-base for getting publicly available information for a repository, or all of… more | homepage

$3

| Commits | Contributor |
| --- | --- |
| 12 | jonschlinkert |
| 9 | doowb |
| 2 | whitneyit |

$3

Jon Schlinkert

* GitHub Profile
* Twitter Profile
* LinkedIn Profile

$3

Copyright © 2021, Jon Schlinkert.
Released under the MIT License.

*

_This file was generated by verb-generate-readme, v0.8.0, on January 20, 2021._