robots.md (1994B)
1 ---
2 title: Robots.txt File
3 linktitle: Robots.txt
4 description: Hugo can generate a customized robots.txt in the same way as any other template.
5 date: 2017-02-01
6 publishdate: 2017-02-01
7 lastmod: 2017-02-01
8 categories: [templates]
9 keywords: [robots,search engines]
10 menu:
11 docs:
12 parent: "templates"
13 weight: 165
14 weight: 165
15 sections_weight: 165
16 draft: false
17 aliases: [/extras/robots-txt/]
18 toc: false
19 ---
20
21 To generate a robots.txt file from a template, change the [site configuration][config]:
22
23 {{< code-toggle file="config">}}
24 enableRobotsTXT = true
25 {{< /code-toggle >}}
26
27 By default, Hugo generates robots.txt using an [internal template][internal].
28
29 ```text
30 User-agent: *
31 ```
32
33 Search engines that honor the Robots Exclusion Protocol will interpret this as permission to crawl everything on the site.
34
35 ## Robots.txt Template Lookup Order
36
37 You may overwrite the internal template with a custom template. Hugo selects the template using this lookup order:
38
39 1. `/layouts/robots.txt`
40 2. `/themes/<THEME>/layouts/robots.txt`
41
42 ## Robots.txt Template Example
43
44 {{< code file="layouts/robots.txt" download="robots.txt" >}}
45 User-agent: *
46 {{ range .Pages }}
47 Disallow: {{ .RelPermalink }}
48 {{ end }}
49 {{< /code >}}
50
51 This template creates a robots.txt file with a `Disallow` directive for each page on the site. Search engines that honor the Robots Exclusion Protocol will not crawl any page on the site.
52
53 {{% note %}}
54 To create a robots.txt file without using a template:
55
56 1. Set `enableRobotsTXT` to `false` in the [site configuration][config].
57 2. Create a robots.txt file in the `static` directory.
58
59 Remember that Hugo copies everything in the [static directory][static] to the root of `publishDir` (typically `public`) when you build your site.
60
61 [config]: /getting-started/configuration/
62 [static]: /getting-started/directory-structure/
63 {{% /note %}}
64
65 [config]: /getting-started/configuration/
66 [internal]: https://github.com/gohugoio/hugo/blob/master/tpl/tplimpl/embedded/templates/_default/robots.txt