hugo/content/templates/robots.md

title, linktitle, description, date, publishdate, lastmod, categories, tags, weight, draft, aliases, toc, needsreview

title	linktitle	description	date	publishdate	lastmod	categories	tags	weight	draft	aliases	toc	needsreview
Robots.txt File	Robots.txt		2017-02-01	2017-02-01	2017-02-01	templates	robots search engines	165	false	/extras/robots-txt/	false	true

Hugo can generate a customized robots.txt in the same way as any other template.

To enable creating your robots.txt as a template, set the enableRobotsTXT value to true in your project's configuration file. By default, this option generates a robots.txt with the following content, which tells search engines that they are allowed to crawl everything:

User-agent: *

Robots.txt Template Lookup Order

The lookup order for the robots.txt template is as follows:

• /layouts/robots.txt
• /themes/<THEME>/layout/robots.txt

Robots. txt Template Example

The following is an examplerobots.txt layout:

{{% code file="layouts/robots.txt" download="robots.txt" %}}

User-agent: *

{{range .Data.Pages}}
Disallow: {{.RelPermalink}}
{{end}}

{{% /code %}}

This template disallows all the pages of the site by creating one Disallow entry for each page.