The experimental astro add command-line tool automates the installation for you. Run one of the following commands in a new terminal window. (If you aren't sure which package manager you're using, run the first command.) Then, follow the prompts, and type "y" in the terminal (meaning "yes") for each one.

``sh


Using NPM

Then, restart the dev server by typing CTRL-C and then npm run astro dev in the terminal window that was running Astro.

Because this command is new, it might not properly set things up. If that happens, log an issue on Astro GitHub and try the manual installation steps below.

First, install the astro-robots-txt package using your package manager. If you're using npm or aren't sure, run this in the terminal:

`sh
npm install --save-dev astro-robots-txt
`

Then, apply this integration to your astro.config.* file using the integrations property:

__astro.config.mjs__

`js
import robotsTxt from 'astro-robots-txt';

export default {
// ...
integrations: [robotsTxt()],
}
`

Then, restart the dev server.


robots.txt

`text
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap-index.xml
`

| Type | Required | Default value |
| :-----------------------------: | :------: | :-------------: |
|Boolean / String / String[]| No | true |

If you omit the sitemap parameter or set it to true, the resulting output in a _robots.txt_ will be Sitemap: your-site-url/sitemap-index.xml.

If you want to get the _robots.txt_ file without the Sitemap: ... entry, set the sitemap parameter to false.

__astro.config.mjs__

`js
import robotsTxt from 'astro-robots-txt';

export default {
site: 'https://example.com',
integrations: [
robotsTxt({
sitemap: false,
}),
],
};
`

When the sitemap is String or String[] its values must be a valid URL. Only http or https protocols are allowed.

__astro.config.mjs__

`js
import robotsTxt from 'astro-robots-txt';

export default {
site: 'https://example.com',
integrations: [
robotsTxt({
sitemap: [
'https://example.com/first-sitemap.xml',
'http://another.com/second-sitemap.xml',
],
}),
],
};
`

| Type | Required | Default value |
| :-----: | :------: | :-------------: |
| String| No | sitemap-index |

Sitemap file name before file extension (.xml). It will be used if the sitemap parameter is true or omitted.

:grey_exclamation: @astrojs/sitemap and astro-sitemap integrations have the sitemap-index.xml as their primary output. That is why the default value of sitemapBaseFileName is set to sitemap-index.

__astro.config.mjs__

`js
import robotsTxt from 'astro-robots-txt';

export default {
site: 'https://example.com',

integrations: [
robotsTxt({
sitemapBaseFileName: 'custom-sitemap',
}),
],
};
`

| Type | Required | Default value |
| :-----------------: | :------: | :-------------: |
|Boolean / String | No | undefined |

Some crawlers (Yandex) support a Host directive, allowing websites with multiple mirrors to specify their preferred domain.

__astro.config.mjs__

`js
import robotsTxt from 'astro-robots-txt';

export default {
site: 'https://example.com',

integrations: [
robotsTxt({
host: 'your-domain-name.com',
}),
],
};
`

If the host option is set to true, the Host output will be automatically resolved using the site option from Astro config.

| Type | Required | Default value |
| :------------------------: | :------: | :-------------: |
| (content: String): String
or
(content: String): Promise | No | undefined |

Sync or async function called just before writing the text output to disk.

__astro.config.mjs__

`js
import robotsTxt from 'astro-robots-txt';

export default {
site: 'https://example.com',

integrations: [
robotsTxt({
transform(content) {
return # Some comments before the main content.\n# Second line.\n\n${content};
},
}),
],
};
`

| Type | Required | Default value |
| :--------: | :------: | :---------------------------------: |
| Policy[] | No | [{ allow: '/', userAgent: '*' }] |

List of Policy rules

Type Policy

| Name | Type | Required | Description |
| :----------: | :-------------------: | :------: | :---------------------------------------------------------------------------------------------------- |
| userAgent | String | Yes | You must provide a name of the automatic client (search engine crawler).
Wildcards are allowed.|
| disallow | String / String[] | No | Disallowed paths for crawling |
| allow | String / String[] | No | Allowed paths for crawling |
| crawlDelay | Number | No | Minimum interval (in secs) for the crawler to wait after loading one page, before starting other |
| cleanParam | String / String[] | No | Indicates that the page's URL contains parameters that should be ignored during crawling.
Maximum string length is limited to 500.|

__astro.config.mjs__

`js
import robotsTxt from 'astro-robots-txt';

export default {
site: 'https://example.com',

integrations: [
robotsTxt({
policy: [
{
userAgent: 'Googlebot',
allow: '/',
disallow: ['/search'],
crawlDelay: 2,
},
{
userAgent: 'OtherBot',
allow: ['/allow-for-all-bots', '/allow-only-for-other-bot'],
disallow: ['/admin', '/login'],
crawlDelay: 2,
},
{
userAgent: '*',
allow: '/',
disallow: '/search',
crawlDelay: 10,
cleanParam: 'ref /articles/',
},
],
}),
],
};
``