prerendercloud-nodejs

!Github Actions CI

This package is the Node.js client for Headless-Render-API.com (formerly named prerender.cloud from 2016 - 2022)

Use it for pre-rendering (server-side rendering), or taking screenshots of webpages or converting webpages to PDFs.

``bash
npm install prerendercloud-server --save
`

`javascript
// simplest possible example usage of this lib
const prerendercloud = require("prerendercloud");

// if you are pre-rendering a JavaScript single-page app
// served from express (or middleware compatible http server),
// use as middleware in your existing server - or try our all-in-one
// server https://github.com/sanfrancesco/prerendercloud-server
app.use(prerendercloud);

// or take a screenshot of a URL
const fs = require("fs");
prerendercloud
.screenshot("http://example.com")
.then((pngBuffer) =>
fs.writeFileSync("out.png", pngBuffer, { encoding: null })
);

// or create a PDF from a URL
prerendercloud
.pdf("http://example.com")
.then((pdfBuffer) =>
fs.writeFileSync("out.pdf", pdfBuffer, { encoding: null })
);
`

The pre-render/server-side rendering functionality of this package (as opposed to mere screenshots/pdfs) is meant to be included in an existing web server where 404s are rendered as index.html

- For an all-in-one single-page app web server plus server-side rendering see: https://github.com/sanfrancesco/prerendercloud-server

---

- Install
- npm
- Auth \(API Token\)
- Environment variable \(best practice\)
- Hard coded
- Debugging
- Screenshots
- PDFs
- Scrape
- Prerendering or Server-side rendering with Express/Connect/Node http
- Configure a condition for when traffic should go through Headless-Render-API.com
- Enable for bots ONLY \(google, facebook, twitter, slack etc...\)
- Whitelist your own user-agent list \(overrides botsOnly\) \(case sensitive\)
- beforeRender \(short circuit the remote call to service.headless-render-api.com\)
- blacklistPaths
- whitelistPaths
- shouldPrerender
- shouldPrerenderAdditionalCheck
- Caching
- Disable Headless-Render-API.com server cache
- Using the \(optional\) middleware cache
- Clearing the middleware cache
- Server Options
- disableServerCache
- serverCacheDurationSeconds
- metaOnly
- followRedirects
- disableAjaxBypass
- disableAjaxPreload
- disableHeadDedupe
- originHeaderWhitelist
- removeScriptTags
- removeTrailingSlash
- waitExtraLong
- withMetadata
- withScreenshot
- DeviceWidth
- DeviceHeight
- Middleware Options
- host
- protocol
- whitelistQueryParams
- afterRenderBlocking \(executes before afterRender\)
- afterRender \(a noop\) \(caching, analytics\) \(executes after afterRenderBlocking\)
- bubbleUp5xxErrors
- retries
- throttleOnFail
- How errors from the server \(service.headless-render-api.com\) are handled

Install

$3

`bash
npm install prerendercloud --save
`

$3

Get a token after signing up at https://headless-render-api.com - it's necessary to move off of the rate-limited free tier

#### Environment variable (best practice)

`javascript
PRERENDER_TOKEN=mySecretToken node index.js
`

#### Hard coded

`javascript
const prerendercloud = require("prerendercloud");
prerendercloud.set("prerenderToken", "mySecretToken");
`

$3

`javascript
DEBUG=prerendercloud node index.js
`

Screenshots

Capture a screenshot:

`javascript
const prerendercloud = require("prerendercloud");
prerendercloud
.screenshot("http://example.com")
.then((pngBuffer) =>
fs.writeFileSync("out.png", pngBuffer, { encoding: null })
);
`

Capture specific element with padding:

Note: viewportQuerySelector causes viewportWidth/viewportHeight to be ignored

`javascript
prerendercloud
.screenshot("http://example.com", {
viewportQuerySelector: "#open-graph-div",
viewportQuerySelectorPadding: 10,
})
.then((pngBuffer) =>
fs.writeFileSync("out.png", pngBuffer, { encoding: null })
);
`

Customize dimensions and format:

`javascript
prerendercloud
.screenshot("http://example.com", {
deviceWidth: 800,
deviceHeight: 600,
viewportWidth: 640,
viewportHeight: 480,
viewportX: 0,
viewportY: 0,
format: "jpeg", // png, webp, jpeg
})
.then((jpgBuffer) =>
fs.writeFileSync("out.jpg", jpgBuffer, { encoding: null })
);
`

Set emulated media and viewport scale:

`javascript
prerendercloud
.screenshot("http://example.com", {
// (screen, print, braille, embossed, handheld, projection, speech, tty, tv)
emulatedMedia: "print",
viewportScale: 2,
})
.then((pngBuffer) =>
fs.writeFileSync("out.png", pngBuffer, { encoding: null })
);
`

PDFs

Generate a PDF:

`javascript
const prerendercloud = require("prerendercloud");
prerendercloud
.pdf("http://example.com")
.then((pdfBuffer) =>
fs.writeFileSync("out.pdf", pdfBuffer, { encoding: null })
);
`

Disable page breaks and set emulated media:

`javascript
prerendercloud
.pdf("http://example.com", {
// Note: using noPageBreaks forces the following
// - pageRanges: "1",
// - preferCssPageSize: true
noPageBreaks: true,
printBackground: true,

// emulatedMedia options: (screen, print, braille, embossed, handheld, projection, speech, tty, tv)
emulatedMedia: "screen",
})
.then((pdfBuffer) =>
fs.writeFileSync("out.pdf", pdfBuffer, { encoding: null })
);
`

Configure PDF options:

`javascript
const prerendercloud = require("prerendercloud");
prerendercloud
.pdf("http://example.com", {
pageRanges: "1-3",
scale: 1.5,
preferCssPageSize: true,
printBackground: true,
landscape: true,
marginTop: 1.75,
marginRight: 0.5,
marginBottom: 1.75,
marginLeft: 0.5,
paperWidth: 8.5,
paperHeight: 11,
})
.then((pdfBuffer) =>
fs.writeFileSync("out.pdf", pdfBuffer, { encoding: null })
);
`

Scrape

Scrape a webpage/URL. Optimized for scraping, this endpoint is based off our pre-rendering engine so it can correctly handle JavaScript apps.

Scrape just the HTML:

`javascript
const { body } = await prerendercloud.scrape("https://example.com");
console.log(body);
fs.writeFileSync("body.html", body);
`

Or scrape HTML and take a screenshot, and parse important meta tags (title, h1, open graph, etc.):

`javascript
const {
body, // Buffer
meta: {
title,
h1,
description,
ogImage,
ogTitle,
ogType,
ogDescription,
twitterCard,
},
links, // Array
screenshot, // Buffer
statusCode, // number
headers, // object of headers
} = await prerendercloud.scrape("https://example.com", {
withMetadata: true,
withScreenshot: true,
followRedirects: false,
});

if (statusCode === 301 || statusCode === 302) {
// get the redirect location from headers.location
// if instead you'd rather follow redirects, set followRedirects: true
} else {
console.log(body.toString());
console.log({
meta: {
title,
h1,
description,
ogImage,
ogTitle,
ogDescription,
twitterCard,
},
links,
});
fs.writeFileSync("body.html", body);
fs.writeFileSync("screenshot.png", screenshot);

// links is an array of all the links on the page
// meta is an object that looks like: { title, h1, description, ogImage, ogTitle, ogType, ogDescription, twitterCard }
// screenshot and body are Buffers (so they can be saved to file)
// call body.toString() for stringified HTML
}
`

Prerendering or Server-side rendering with Express/Connect/Node http

The prerendercloud middleware should be loaded first, before your other middleware, so it can forward the request to service.headless-render-api.com.

`javascript
// the free, rate limited tier
// and using https://expressjs.com/
const prerendercloud = require("prerendercloud");
expressApp.use(prerendercloud);
`

$3

The default behavior forwards all traffic through Headless-Render-API.com

#### Enable for bots ONLY (google, facebook, twitter, slack etc...)

We don't recommend this setting, instead use the default setting of pre-rendering all user-agents (because of performance boost and potential google cloaking penalties) but there may be a situation where you shouldn't or can't, for example: your site/app has JavaScript errors when trying to repaint the DOM after it's already been pre-rendered but you still want bots (twitter, slack, facebook etc...) to read the meta and open graph tags.

Note: this will add or append 'User-Agent' to the Vary header, which is another reason not to recommend this feature (because it significantly reduces HTTP cacheability)

`javascript
const prerendercloud = require("prerendercloud");
prerendercloud.set("botsOnly", true);
`

You can also append your own agents to our botsOnly list by using an array:

`javascript
const prerendercloud = require("prerendercloud");
prerendercloud.set("botsOnly", ["altavista", "dogpile", "excite", "askjeeves"]);
`

#### Whitelist your own user-agent list (overrides botsOnly) (case sensitive)

Note: this will NOT add or append 'User-Agent' to the Vary header. You should probably set the Vary header yourself, if using this feature.

`javascript
const prerendercloud = require("prerendercloud");
prerendercloud.set("whitelistUserAgents", [
"twitterbot",
"slackbot",
"facebookexternalhit",
]);
`

#### beforeRender (short circuit the remote call to service.headless-render-api.com)

Useful for your own caching layer (in conjunction with afterRender), or analytics, or dependency injection for testing. Is only called when a remote call to service.headless-render-api.com is about to be made.

`javascript
const prerendercloud = require("prerendercloud");
prerendercloud.set("beforeRender", (req, done) => {
// call it with a string to short-circuit the remote prerender codepath
// (useful when implementing your own cache)
done(null, "hello world"); // returns status 200, content-type text/html

// or call it with an object to short-circuit the remote prerender codepath
// (useful when implementing your own cache)
done(null, { status: 202, body: "hello" }); // returns status 202, content-type text/html

done(null, { status: 301, headers: { location: "/new-path" } }); // redirect to /new-path

// or call it with nothing/empty/null/undefined to follow the remote prerender path
// (useful for analytics)
done();
done("");
done(null);
done(undefined);
});
`

#### blacklistPaths

Prevent paths from being prerendered. Takes a function that returns an array. It is executed before the shouldPrerender option.

The primary use case is for CDN edge node clients (CloudFront Lambda@Edge) because they don't have the ability to quickly read the origin (AWS S3) filesystem, so they have to hard-code paths that shouldn't be prerendered.

Paths you may not want prerendered are non-SPA, large pages, or pages with JavaScript that can't rehydrate prerendered DOMs.

Trailing * works as wildcard. Only works when at the end.

`javascript
const prerendercloud = require("prerendercloud");
prerendercloud.set("blacklistPaths", (req) => [
"/google-domain-verification",
"/google-domain-verification.html",
"/google-domain-verification/",
"/image-gallery/*",
]);
`

#### whitelistPaths

Limit which URLs can trigger a pre-render request to the server.

Takes a function that returns an array of strings or regexes. It is executed before the shouldPrerender option. Passing an empty array or string will do nothing (noop).

Using this option will prevent bots/scrapers from hitting random URLs and increasing your billing. Recommended for Node.js server and Lambda@Edge (can be used with our without blacklist - blacklist takes precedent).

Even better if used with whitelistQueryParams and/or removeTrailingSlash.

`javascript
const prerendercloud = require("prerendercloud");
prerendercloud.set("whitelistPaths", req => [
"/docs",
"/docs/"
/\/users\/\d{1,6}\/profile$/, // without the ending $, this is equivalent to startsWith
/\/users\/\d{1,6}\/profile\/?$/, // note the optional ending slash (\/?) and $
"/google-domain-verification.html",
"/google-domain-verification/",
]);

`

#### shouldPrerender

This is executed after the beforeRender but if present, replaces userAgent detection (it would override botsOnly).

`javascript
const prerendercloud = require("prerendercloud");
prerendercloud.set("shouldPrerender", (req) => {
return req.headers["user-agent"] === "googlebot" && someStateOnMyServer();
// return bool
});
`

#### shouldPrerenderAdditionalCheck

Runs in addition to the default user-agent check. Useful if you have your own conditions.

`javascript
// time delay
const waitUntil = new Date() + 10000;
prerendercloud.set("shouldPrerenderAdditionalCheck", (req) => {
return new Date() > waitUntil;
});

// enable flag
let isEnabled = false;
prerendercloud.set("shouldPrerenderAdditionalCheck", (req) => {
return isEnabled;
});
`

$3

#### Disable Headless-Render-API.com server cache

The servers behind service.headless-render-api.com will cache for 5 minutes as a best practice. Adding the Prerender-Disable-Cache HTTP header via this config option disables that cache entirely. Disabling the service.headless-render-api.com cache is only recommended if you have your own cache either in this middleware or your client, otherwise all of your requests are going to be slow.

`javascript
const prerendercloud = require("prerendercloud");
prerendercloud.set("disableServerCache", true);
app.use(prerendercloud);
`

#### Using the (optional) middleware cache

This middleware has a built-in LRU (drops least recently used) caching layer. It can be configured to let cache auto expire or you can manually remove entire domains from the cache. You proboably want to use this if you disabled the server cache.

`javascript
const prerendercloud = require("prerendercloud");
prerendercloud.set("enableMiddlewareCache", true);

// optionally set max bytes (defaults to 500MB)
prerendercloud.set("middlewareCacheMaxBytes", 1000000000); // 1GB

// optionally set max age (defaults to forever - implying you should manually clear it)
prerendercloud.set("middlewareCacheMaxAge", 1000 60 60); // 1 hour

app.use(prerendercloud);
`

##### Clearing the middleware cache

`javascript
// delete every page on the example.org domain
prerendercloud.cache.clear("http://example.org");

// delete every page on every domain
prerendercloud.cache.reset();
`

$3

These options map to the HTTP header options listed here: https://headless-render-api.com/docs/api

#### disableServerCache

This option disables an enabled-by-default 5-minute cache.

The servers behind service.headless-render-api.com will cache for 5 minutes as a best practice. Adding the Prerender-Disable-Cache HTTP header via this config option disables that cache entirely. Disabling the service.headless-render-api.com cache is only recommended if you have your own cache either in this middleware or your client, otherwise all of your requests are going to be slow.

`javascript
const prerendercloud = require("prerendercloud");
prerendercloud.set("disableServerCache", true);
app.use(prerendercloud);
`

#### serverCacheDurationSeconds

This option configures the duration for Headless-Render-API.com's server cache:

The servers behind service.headless-render-api.com will cache for 5 minutes as a best practice, configure that duration (in seconds):

`javascript
const prerendercloud = require("prerendercloud");
// max value: 2592000 (1 month)
prerendercloud.set("serverCacheDurationSeconds", (req) => 300);
app.use(prerendercloud);
`

#### metaOnly

This option tells the server to only prerender the