data-templates.md (10094B)
1 ---
2 title: Data Templates
3 linktitle:
4 description: In addition to Hugo's built-in variables, you can specify your own custom data in templates or shortcodes that pull from both local and dynamic sources.
5 date: 2017-02-01
6 publishdate: 2017-02-01
7 lastmod: 2017-03-12
8 categories: [templates]
9 keywords: [data,dynamic,csv,json,toml,yaml,xml]
10 menu:
11 docs:
12 parent: "templates"
13 weight: 80
14 weight: 80
15 sections_weight: 80
16 draft: false
17 aliases: [/extras/datafiles/,/extras/datadrivencontent/,/doc/datafiles/]
18 toc: true
19 ---
20
21 <!-- begin data files -->
22
23 Hugo supports loading data from YAML, JSON, XML, and TOML files located in the `data` directory in the root of your Hugo project.
24
25 {{< youtube FyPgSuwIMWQ >}}
26
27 ## The Data Folder
28
29 The `data` folder is where you can store additional data for Hugo to use when generating your site. Data files aren't used to generate standalone pages; rather, they're meant to be supplemental to content files. This feature can extend the content in case your front matter fields grow out of control. Or perhaps you want to show a larger dataset in a template (see example below). In both cases, it's a good idea to outsource the data in their own files.
30
31 These files must be YAML, JSON, XML, or TOML files (using the `.yml`, `.yaml`, `.json`, `.xml`, or `.toml` extension). The data will be accessible as a `map` in the `.Site.Data` variable.
32
33 ## Data Files in Themes
34
35 Data Files can also be used in [Hugo themes][themes] but note that theme data files follow the same logic as other template files in the [Hugo lookup order][lookup] (i.e., given two files with the same name and relative path, the file in the root project `data` directory will override the file in the `themes/<THEME>/data` directory).
36
37 Therefore, theme authors should take care to not include data files that could be easily overwritten by a user who decides to [customize a theme][customize]. For theme-specific data items that shouldn't be overridden, it can be wise to prefix the folder structure with a namespace; e.g. `mytheme/data/<THEME>/somekey/...`. To check if any such duplicate exists, run hugo with the `-v` flag.
38
39 The keys in the map created with data templates from data files will be a dot-chained set of `path`, `filename`, and `key` in file (if applicable).
40
41 This is best explained with an example:
42
43 ## Example: Jaco Pastorius' Solo Discography
44
45 [Jaco Pastorius](https://en.wikipedia.org/wiki/Jaco_Pastorius_discography) was a great bass player, but his solo discography is short enough to use as an example. [John Patitucci](https://en.wikipedia.org/wiki/John_Patitucci) is another bass giant.
46
47 The example below is a bit contrived, but it illustrates the flexibility of data Files. This example uses TOML as its file format with the two following data files:
48
49 * `data/jazz/bass/jacopastorius.toml`
50 * `data/jazz/bass/johnpatitucci.toml`
51
52 `jacopastorius.toml` contains the content below. `johnpatitucci.toml` contains a similar list:
53
54 {{< code-toggle file="jacopastorius" >}}
55 discography = [
56 "1974 - Modern American Music … Period! The Criteria Sessions",
57 "1974 - Jaco",
58 "1976 - Jaco Pastorius",
59 "1981 - Word of Mouth",
60 "1981 - The Birthday Concert (released in 1995)",
61 "1982 - Twins I & II (released in 1999)",
62 "1983 - Invitation",
63 "1986 - Broadway Blues (released in 1998)",
64 "1986 - Honestly Solo Live (released in 1990)",
65 "1986 - Live In Italy (released in 1991)",
66 "1986 - Heavy'n Jazz (released in 1992)",
67 "1991 - Live In New York City, Volumes 1-7.",
68 "1999 - Rare Collection (compilation)",
69 "2003 - Punk Jazz: The Jaco Pastorius Anthology (compilation)",
70 "2007 - The Essential Jaco Pastorius (compilation)"
71 ]
72 {{< /code-toggle >}}
73
74 The list of bass players can be accessed via `.Site.Data.jazz.bass`, a single bass player by adding the filename without the suffix, e.g. `.Site.Data.jazz.bass.jacopastorius`.
75
76 You can now render the list of recordings for all the bass players in a template:
77
78 ```
79 {{ range $.Site.Data.jazz.bass }}
80 {{ partial "artist.html" . }}
81 {{ end }}
82 ```
83
84 And then in the `partials/artist.html`:
85
86 ```
87 <ul>
88 {{ range .discography }}
89 <li>{{ . }}</li>
90 {{ end }}
91 </ul>
92 ```
93
94 Discover a new favorite bass player? Just add another `.toml` file in the same directory.
95
96 ## Example: Accessing Named Values in a Data File
97
98 Assume you have the following data structure in your `User0123.[yml|toml|xml|json]` data file located directly in `data/`:
99
100 {{< code-toggle file="User0123" >}}
101 Name: User0123
102 "Short Description": "He is a **jolly good** fellow."
103 Achievements:
104 - "Can create a Key, Value list from Data File"
105 - "Learns Hugo"
106 - "Reads documentation"
107 {{</ code-toggle >}}
108
109 You can use the following code to render the `Short Description` in your layout:
110
111 ```
112 <div>Short Description of {{.Site.Data.User0123.Name}}: <p>{{ index .Site.Data.User0123 "Short Description" | markdownify }}</p></div>
113 ```
114
115 Note the use of the [`markdownify` template function][markdownify]. This will send the description through the Markdown rendering engine.
116
117
118 ## Get Remote Data
119
120 Use `getJSON` or `getCSV` to get remote data:
121
122 ```
123 {{ $dataJ := getJSON "url" }}
124 {{ $dataC := getCSV "separator" "url" }}
125 ```
126
127 If you use a prefix or postfix for the URL, the functions accept [variadic arguments][variadic]:
128
129 ```
130 {{ $dataJ := getJSON "url prefix" "arg1" "arg2" "arg n" }}
131 {{ $dataC := getCSV "separator" "url prefix" "arg1" "arg2" "arg n" }}
132 ```
133
134 The separator for `getCSV` must be put in the first position and can only be one character long.
135
136 All passed arguments will be joined to the final URL:
137
138 ```
139 {{ $urlPre := "https://api.github.com" }}
140 {{ $gistJ := getJSON $urlPre "/users/GITHUB_USERNAME/gists" }}
141 ```
142
143 This will resolve internally to the following:
144
145 ```
146 {{ $gistJ := getJSON "https://api.github.com/users/GITHUB_USERNAME/gists" }}
147 ```
148
149 ### Add HTTP headers
150
151 {{< new-in "0.84.0" >}} Both `getJSON` and `getCSV` takes an optional map as the last argument, e.g.:
152
153 ```
154 {{ $data := getJSON "https://example.org/api" (dict "Authorization" "Bearer abcd") }}
155 ```
156
157 If you need multiple values for the same header key, use a slice:
158
159 ```
160 {{ $data := getJSON "https://example.org/api" (dict "X-List" (slice "a" "b" "c")) }}
161 ```
162
163 ### Example for CSV files
164
165 For `getCSV`, the one-character-long separator must be placed in the first position followed by the URL. The following is an example of creating an HTML table in a [partial template][partials] from a published CSV:
166
167 {{< code file="layouts/partials/get-csv.html" >}}
168 <table>
169 <thead>
170 <tr>
171 <th>Name</th>
172 <th>Position</th>
173 <th>Salary</th>
174 </tr>
175 </thead>
176 <tbody>
177 {{ $url := "https://example.com/finance/employee-salaries.csv" }}
178 {{ $sep := "," }}
179 {{ range $i, $r := getCSV $sep $url }}
180 <tr>
181 <td>{{ index $r 0 }}</td>
182 <td>{{ index $r 1 }}</td>
183 <td>{{ index $r 2 }}</td>
184 </tr>
185 {{ end }}
186 </tbody>
187 </table>
188 {{< /code >}}
189
190 The expression `{{index $r number}}` must be used to output the nth-column from the current row.
191
192 ### Cache URLs
193
194 Each downloaded URL will be cached in the default folder `$TMPDIR/hugo_cache/`. The variable `$TMPDIR` will be resolved to your system-dependent temporary directory.
195
196 With the command-line flag `--cacheDir`, you can specify any folder on your system as a caching directory.
197
198 You can also set `cacheDir` in the [main configuration file][config].
199
200 If you don't like caching at all, you can fully disable caching with the command line flag `--ignoreCache`.
201
202 ### Authentication When Using REST URLs
203
204 Currently, you can only use those authentication methods that can be put into an URL. [OAuth][] and other authentication methods are not implemented.
205
206 ## Load Local files
207
208 To load local files with `getJSON` and `getCSV`, the source files must reside within Hugo's working directory. The file extension does not matter, but the content does.
209
210 It applies the same output logic as above in [Get Remote Data](#get-remote-data).
211
212 {{% note %}}
213 The local CSV files to be loaded using `getCSV` must be located **outside** of the `data` directory.
214 {{% /note %}}
215
216 ## LiveReload with Data Files
217
218 There is no chance to trigger a [LiveReload][] when the content of a URL changes. However, when a *local* file changes (i.e., `data/*` and `themes/<THEME>/data/*`), a LiveReload will be triggered. Symlinks are not supported. Note too that because downloading of data takes a while, Hugo stops processing your Markdown files until the data download has completed.
219
220 {{% warning "URL Data and LiveReload" %}}
221 If you change any local file and the LiveReload is triggered, Hugo will read the data-driven (URL) content from the cache. If you have disabled the cache (i.e., by running the server with `hugo server --ignoreCache`), Hugo will re-download the content every time LiveReload triggers. This can create *huge* traffic. You may reach API limits quickly.
222 {{% /warning %}}
223
224 ## Examples of Data-driven Content
225
226 - Photo gallery JSON powered: [https://github.com/pcdummy/hugo-lightslider-example](https://github.com/pcdummy/hugo-lightslider-example)
227 - GitHub Starred Repositories [in a post](https://github.com/SchumacherFM/blog-cs/blob/master/content%2Fposts%2Fgithub-starred.md) using data-driven content in a [custom short code](https://github.com/SchumacherFM/blog-cs/blob/master/layouts%2Fshortcodes%2FghStarred.html).
228
229 ## Specs for Data Formats
230
231 * [TOML Spec][toml]
232 * [YAML Spec][yaml]
233 * [JSON Spec][json]
234 * [CSV Spec][csv]
235 * [XML Spec][xml]
236
237 [config]: /getting-started/configuration/
238 [csv]: https://tools.ietf.org/html/rfc4180
239 [customize]: /themes/customizing/
240 [json]: https://www.ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf "Specification for JSON, JavaScript Object Notation"
241 [LiveReload]: /getting-started/usage/#livereload
242 [lookup]: /templates/lookup-order/
243 [markdownify]: /functions/markdownify/
244 [OAuth]: https://en.wikipedia.org/wiki/OAuth
245 [partials]: /templates/partials/
246 [themes]: /themes/
247 [toml]: https://github.com/toml-lang/toml
248 [variadic]: https://en.wikipedia.org/wiki/Variadic_function
249 [vars]: /variables/
250 [yaml]: https://yaml.org/spec/
251 [xml]: https://www.w3.org/XML/