@tanepiper/robots-webpack-plugin

Webpack Robots.txt

A webpack plugin for generating robots.txt files.

$3

When you have multiple environments such as dev, qa and production to want to ensure your non-production environments
are not exposed to search engines such as Google.

This plugin allows you to include this in your environment settings to generate a file.

$3

npm install --save-dev @tanepiper/robots-webpack-plugin

$3

In your webpack config:

``
const RobotsPlugin = require('@tanepiper/robots-webpack-plugin');

const webpackConfig = {
entry: 'src/index.js',
output: {
path: '/public',
filename: 'index.js'
},
plugins: [new RobotsPlugin()]
};
`

The default output is an Allow directive for all user agents. To configure, pass in an array to userAgents option.
Each one is an object with a name:string, disallow:array and allow:array in any combination.

There is also the sitemap options which will output a Sitemap directive at the end. Just provide a full url to the sitemap file.

`
plugins: [new RobotsPlugin({
sitemap: 'https://example.com/sitemap.xml',
userAgents: [{
name: '*',
disallow: ['/', '/cgi-bin'],
allow: ['/blog']
}, {
name: 'Googlebot',
disallow: ['/cgi-bin', '/some-path'],
allow: ['/']
}]
})]
``