A lazy-loading API paginator with async generators, exponential retry, and lifecycle hooks
npm install lazy-api-paginator




A TypeScript module for lazily fetching paginated API data using async generators. Features include exponential backoff retry logic and lifecycle hooks.
- Lazy loading of paginated API data using async generators
- Iterate over items one-by-one without loading all pages into memory
- Built-in strategies for cursor, offset, page number, link header, and keyset pagination
- Exponential backoff retry with configurable jitter
- Rate limiting with request throttling and automatic 429 handling
- Lifecycle hooks: onBeforeFetch, onAfterFetch, onError, onData
- SSRF protection for secure server-to-server calls (via ssrf-agent-guard)
- Full TypeScript support
- Works with both ESM and CommonJS
``bash`
npm install lazy-api-paginator
`typescript
import { createPaginator } from 'lazy-api-paginator';
interface ApiResponse {
data: User[];
nextCursor: string | null;
}
interface User {
id: number;
name: string;
}
const paginator = createPaginator
initialUrl: 'https://api.example.com/users',
extractItems: (response) => response.data,
getNextPageUrl: (response) =>
response.nextCursor
? https://api.example.com/users?cursor=${response.nextCursor}
: null,
});
// Iterate lazily - pages are fetched on-demand
for await (const user of paginator) {
console.log(user.name);
}
`
`typescriptFetching page ${pagination.page}: ${url}
const paginator = createPaginator
initialUrl: 'https://api.example.com/users',
extractItems: (response) => response.data,
getNextPageUrl: (response) => response.nextCursor,
hooks: {
onBeforeFetch: ({ url, pagination }) => {
console.log();Fetched ${response.data.data.length} items in ${duration}ms
},
onAfterFetch: ({ response, duration }) => {
console.log();Error (attempt ${attempt}): ${error.message}
},
onError: ({ error, attempt, willRetry }) => {
console.error();Processing item ${globalIndex}: ${item.name}
if (willRetry) console.log('Retrying...');
},
onData: ({ item, globalIndex }) => {
console.log();`
},
},
});
`typescript`
const paginator = createPaginator
initialUrl: 'https://api.example.com/users',
extractItems: (response) => response.data,
getNextPageUrl: (response) => response.nextCursor,
retry: {
maxRetries: 5,
initialDelay: 1000, // 1 second
maxDelay: 30000, // 30 seconds
backoffMultiplier: 2, // Exponential factor
jitter: 0.1, // 10% randomness
retryableStatusCodes: [408, 429, 500, 502, 503, 504],
isRetryable: (error, statusCode) => {
// Custom retry logic
return statusCode === 418; // Retry teapot errors
},
},
});
`typescript`
const paginator = createPaginator
initialUrl: 'https://api.example.com/users',
extractItems: (response) => response.data,
getNextPageUrl: (response) => response.nextCursor,
requestConfig: {
method: 'POST',
headers: {
'Authorization': 'Bearer token123',
'Content-Type': 'application/json',
},
body: { filter: 'active' },
timeout: 10000,
},
});
`typescript
// Get first N items
const firstTen = await paginator.take(10);
// Get all items (use with caution for large datasets)
const allUsers = await paginator.toArray();
`
Use pre-built strategies to eliminate boilerplate for common API patterns:
`typescript
import { createPaginator, strategies } from 'lazy-api-paginator';
// Cursor-based (Slack, Stripe, Notion)
const cursorPaginator = createPaginator({
initialUrl: 'https://api.slack.com/users.list',
...strategies.cursor({
dataPath: 'members',
cursorPath: 'response_metadata.next_cursor',
}),
});
// Offset-based (traditional REST APIs)
const offsetPaginator = createPaginator({
initialUrl: 'https://api.example.com/items?offset=0&limit=100',
...strategies.offset({
dataPath: 'items',
totalPath: 'total',
pageSize: 100,
}),
});
// Page number-based (Laravel, Django)
const pagePaginator = createPaginator({
initialUrl: 'https://api.example.com/items?page=1',
...strategies.pageNumber({
dataPath: 'results',
totalPagesPath: 'total_pages',
}),
});
// Link header (GitHub API)
const linkStrategy = strategies.linkHeader({ dataPath: '' });
const githubPaginator = createPaginator({
initialUrl: 'https://api.github.com/repos/owner/repo/issues',
...linkStrategy,
hooks: {
onAfterFetch: ({ response }) => {
const link = response.headers['link'];
if (link) linkStrategy.setNextFromHeader(link);
},
},
});
// Keyset/Seek (efficient for large datasets)
const keysetPaginator = createPaginator({
initialUrl: 'https://api.example.com/items',
...strategies.keyset({
dataPath: 'data',
keyPath: 'id',
hasMorePath: 'has_more',
}),
});
`
Configure request throttling and automatic handling of 429 (Too Many Requests) responses:
`typescriptRate limited on ${url}, waiting ${delayMs}ms
const paginator = createPaginator
initialUrl: 'https://api.example.com/users',
extractItems: (response) => response.data,
getNextPageUrl: (response) => response.nextCursor,
rateLimit: {
requestsPerSecond: 10, // Throttle to 10 requests/sec
respectRetryAfter: true, // Honor Retry-After headers
maxRateLimitDelay: 60000, // Max wait time: 60 seconds
onRateLimitHit: ({ url, delayMs }) => {
console.log();`
},
},
});
The paginator automatically:
- Throttles requests to stay within requestsPerSecondX-RateLimit-
- Parses , RateLimit-, and Retry-After headers
- Waits and retries when receiving 429 responses
- Preemptively waits when rate limit is about to be exhausted
For server-to-server calls, enable SSRF (Server-Side Request Forgery) protection to block requests to internal networks, cloud metadata endpoints, and other potentially dangerous destinations.
First, install the optional dependency:
`bash`
npm install ssrf-agent-guard
Then enable SSRF protection in your paginator:
`typescript
import { createPaginator } from 'lazy-api-paginator';
const paginator = createPaginator({
initialUrl: 'https://api.example.com/items',
extractItems: (r) => r.items,
getNextPageUrl: (r) => r.next,
ssrfProtection: {
enabled: true,
options: {
// Optional: customize ssrf-agent-guard options
mode: 'block', // 'block' | 'report' | 'allow'
},
},
});
`
You can also use the standalone createSecureFetch utility:
`typescript
import { createSecureFetch, createPaginator } from 'lazy-api-paginator';
const secureFetch = await createSecureFetch({ enabled: true });
const paginator = createPaginator({
initialUrl: 'https://api.example.com/items',
extractItems: (r) => r.items,
getNextPageUrl: (r) => r.next,
fetchFn: secureFetch,
});
`
For complete API documentation including all types, interfaces, error classes, and usage patterns, see API.md.
Creates a new lazy paginator instance.
#### Config Options
| Option | Type | Required | Description |
|--------|------|----------|-------------|
| initialUrl | string | Yes | The URL of the first page to fetch |extractItems
| | (response: TResponse) => TItem[] | Yes | Function to extract items from API response |getNextPageUrl
| | (response: TResponse, pagination: PaginationState) => string \| null | Yes | Function to get next page URL (return null to stop) |requestConfig
| | RequestConfig | No | HTTP request configuration |retry
| | RetryConfig | No | Retry configuration |rateLimit
| | RateLimitConfig | No | Rate limiting configuration |hooks
| | PaginatorHooks | No | Lifecycle hooks |fetchFn
| | typeof fetch | No | Custom fetch function |ssrfProtection
| | SsrfProtectionConfig | No | SSRF protection settings |
| Hook | Context | Description |
|------|---------|-------------|
| onBeforeFetch | { url, config, pagination } | Called before each request |onAfterFetch
| | { url, response, pagination, duration } | Called after successful request |onError
| | { error, url, attempt, maxRetries, willRetry, pagination } | Called on error |onData
| | { item, indexInPage, globalIndex, pagination } | Called for each item yielded |
- MaxRetriesExceededError - Thrown when max retries are exceededFetchTimeoutError
- - Thrown when a request times outHttpError` - Thrown for non-2xx HTTP responses
-
MIT © Swapnil Srivastava