Sniff the encoding from a HTML byte stream
npm install html-encoding-snifferThis package implements the HTML Standard's encoding sniffing algorithm in all its glory. The most interesting part of this is how it pre-scans the first 1024 bytes in order to search for certain -related patterns.
``js
const htmlEncodingSniffer = require("html-encoding-sniffer");
const fs = require("fs");
const htmlBytes = fs.readFileSync("./html-page.html");
const sniffedEncoding = htmlEncodingSniffer(htmlBytes);
`
The passed bytes are given as a Uint8Array; the Node.js Buffer subclass of Uint8Array will also work, as shown above.
The returned value will be a canonical encoding name (not a label). You might then combine this with the @exodus/bytes package to decode the result:
`js`
const { TextDecoder } = require("@exodus/bytes");
const htmlString = (new TextDecoder(sniffedEncoding)).decode(htmlBytes);
You can pass the following options to htmlEncodingSniffer:
`js`
const sniffedEncoding = htmlEncodingSniffer(htmlBytes, {
xml,
transportLayerEncodingLabel,
defaultEncoding,
});
The xml option is a boolean, defaulting to false. If set to true, then we bypass the HTML encoding sniffing algorithm and compute the encoding based on the presence of a BOM, or the other options provided. (In the future, we may perform sniffing of the declaration, but for now that is not implemented.)
The transportLayerEncodingLabel is an encoding label that is obtained from the "transport layer" (probably a HTTP Content-Type header), which overrides everything but a BOM.
The defaultEncoding is the ultimate fallback encoding used if no valid encoding is supplied by the transport layer, and no encoding is sniffed from the bytes. For HTML, it defaults to "windows-1252", as recommended by the algorithm's table of suggested defaults for "All other locales" (including the en locale). For XML, it defaults to "UTF-8"`.
This package was originally based on the excellent work of @nicolashenry, in jsdom. It has since been pulled out into this separate package.