html-get
v2.21.21
Get the HTML from any website, fine-tuned for correction & speed
969/weekUpdated 1 weeks agoMITUnpacked: 31.1 KB
Published by Kiko Beats
npm install html-get
!Last version
> Get the HTML from any website, fine-tuned for correction & speed.
Features
- Get HTML markup for any URL, including images, video, audio, or pdf.
- Block ads tracker or any non-necessary network subrequest.
- Handle unreachable or timeout URLs gracefully.
- Ensure HTML markup is appropriately encoded.
html-get takes advantage of puppeteer headless technology when is needed, such as client-side apps that needs to be prerender.
Install
``bash`
$ npm install browserless puppeteer html-get --save
Usage
`js
const createBrowserless = require('browserless')
const getHTML = require('html-get')
// Spawn Chromium process once
const browserlessFactory = createBrowserless()
// Kill the process when Node.js exit
process.on('exit', () => {
console.log('closing resources!')
browserlessFactory.close()
})
const getContent = async url => {
// create a browser context inside Chromium process
const browserContext = browserlessFactory.createContext()
const getBrowserless = () => browserContext
const result = await getHTML(url, { getBrowserless })
// close the browser context after it's used
await getBrowserless((browser) => browser.destroyContext())
return result
}
getContent('https://example.com')
.then(content => {
console.log(content)
process.exit()
})
.catch(error => {
console.error(error)
process.exit(1)
})
`
$3
`
$ npx html-get https://example.com
API
$3
#### url
Required
Type: string
The target URL for getting the HTML markup.
#### options
##### encoding
Type: string
Default:
It ensures the HTML markup is encoded to the encoded value provided.
The value will be passes to html-encode
##### getBrowserless
Required
Type: function
A function that should return a browserless instance to be used for interact with puppeteer:
##### getMode
Type: function
It determines the strategy to use based on the url, being the possibles values 'fetch' or 'prerender' .
##### getTemporalFile
Type: function
It creates a temporal file.
##### gotOpts
Type: object
It passes configuration object to got under 'fetch' strategy.
##### headers
Type: object
Request headers that will be passed to fetch/prerender process.
##### mutool
Type: function|boolean
Default:
It returns a function that receives that executes mutool binary for turning PDF files into HTML markup.
It can explicitly disabled passing false.
##### prerender
Type: boolean|string
Default:
Enable or disable prerendering as mechanism for getting the HTML markup explicitly.
The value auto means that that internally use a list of websites that don't need to use prerendering by default. This list is used for speedup the process, using fetch mode for these websites.
See getMode parameter for know more.
##### puppeteerOpts
Type: object
It passes coniguration object to puppeteer under 'prerender' strategy.
##### rewriteUrls
Type: boolean
Default:
When is true, it will be rewritten CSS/HTML relatives URLs present in the HTML markup into absolutes.
##### rewriteHtml
Type: boolean
Default:
When is true, it will rewrite some common mistake related with HTML meta tags.
##### serializeHtml
It determines how HTML should be serialied before returning.
It's serialized $ => ({ html: $.html() })` by default.
License
html-get © Microlink, released under the MIT License.
Authored and maintained by Kiko Beats with help from contributors.
> microlink.io · GitHub microlinkhq · X @microlinkhq