This article is for the 23rd day of Go Advent Calendar 2020 and the 24th day of CyberAgent Developers Advent Calendar 2020.
Go 1.16, due out in February next year will have many interesting new features as well as the usual release. One of the most notable features is the go: embed directive, which embeds a read-only static file in Go's pre-built binaries. Various approaches to embedding static files in binary have been proposed [^ 1], but the embedding method is different for each tool and OS, and it is unavoidable to learn how to use each and to depend on a specific tool. .. With the introduction of this directive this time, it will be unified as the Go formula.
The reason why the introduction of this directive is attracting attention is, of course, the historical background mentioned above, but I think there are also the following questions.
――What kind of function does it have? ――How to use it? (How can I use it on the contrary?) --Where is the file embedded? ――What format is the file embedded in? ――What kind of process do you follow from the file embedding source to the caller?
Therefore, it is divided into the first part and the second part of the ** usage part ** that explains how to use go: embed and the ** specification part ** that deepens how it is implemented. In this article, how to use the first part. I will focus on the explanation. I will release the second part at a later date because I am looking forward to it.
Please note that the version of Go used in this article is go1.16beta1 darwin/amd64, and Go1.16 has not yet been officially released at the time of writing, so it may be updated from the content of this article. ..
[^ 1]: There are many just written in Draft.
Before we dive into go: embed, let's talk about the io.fs package, which will also be introduced in Go1.16.
The hierarchical file system is Dennis Ritchie and Ken Thompson's Major Achievements of UNIX Research at Bell Labs, and today it includes operating systems, as well as Web URLs and ZIPs. It's used everywhere, even in archive files (which you've probably opened in Vim about once;)). Accordingly, Go also implements standard packages that read files placed in a hierarchical file system. It is the os package that manipulates operating system files, the archive/zip package for ZIP, and the html/template package that dynamically generates HTML from static file templates, URLs. The http package (the File/FileSystem structure) directly returns the static assets for. Although these can be treated in the same way as a hierarchical structure, their implementations were not unified, so some kind of bridging was necessary. Despite trying to handle resources expressed as a hierarchical file system in Plan 9, which was also developed by Bell Labs as the next-generation UNIX, without depending on the protocol or architecture. is there.
Meanwhile, the io/fs package appeared like a meteor. Inheriting the intention of Plan 9 (?), Go can finally handle the file system with a unified interface.
For the explanation and usage of io/fs, @ spiegel_2007's" Preparing for (maybe) io/fs package to be introduced in the next Go language "is almost the same. It will be released as it is, so please refer to that for details.
Now, let's get into the main subject from here.
First of all, import the embed package to enable the go: embed directive. If you do not use the embed package directly, import it blank.
import _ "embed"
Next, regarding how to embed a file, the go: embed directive can be embedded in an uninitialized variable declared with var. As with other directives, if you put a space between // and go: embed, it will be treated as a normal comment, so be careful. The files that can be specified are the files under the current directory, and are specified with a relative path. Even when building a binary for a platform that represents a hierarchical structure with the half-width yen symbol \ such as Windows, it is represented by the half-width slash / as in the * nix series.
//go:embed hello.txt
var hello string
//go:embed hello/world.txt
var world []byte
There are three types of variables that can be embedded with the go: embed directive: string, [] byte, and embed.FS, but there is a difference in the embedding method between the former two and the last one. ..
string and [] byte can be treated as variables that have been initialized as usual by reading the file with a single go: embed directive. Intuitively, defining multiple go: embeds will result in a compilation error.
On the other hand, embed.FS is a structure [^ 3] that implements the io/fs.FS interface, which allows you to embed files embedded with go: embed as a hierarchical file system. It is possible to embed one or more files or directories. To specify multiple files or directories, use the wildcard * or separate them on multiple lines.
[^ 3]: The io/fs.FS interface only has a Open function to open a file, which is minimal as a file system. In addition to the io/fs.FS interface, the embed.FS structure also implements io/fs.ReadFileFS to read files and io/fs.ReadDirFS to read directories.
//go:embed image/* template/*
//go:embed style/*.css
//go:embed html/index.html
var assets embed.FS
Finally, regarding the scope of embedding the go: embed directive, it is possible to embed it not only globally but also locally, that is, within a function. Embedding locally feels a little strange, but it's quite natural to think of it in the same way as normal variable initialization [^ local].
[^ local]: Embedding by directive in a local variable is removed from the implementation in Go1.16 due to problems such as sharing the embedded file between functions or separating the embedded file for each function. Going in the direction. https://github.com/golang/go/issues/43216
//go:embed global.txt
var global string
func f() {
//go:embed local.txt
var local string
...
}
This concludes the brief explanation of how to use the go: embed directive.
Now, from here on, I will give some detailed notes, more specifically, examples such as "Unexpectedly, a compile error does not occur even if you use it in this way" and conversely, "It seems that a compile error does not occur and actually a compile error occurs". ..
Files and directories that are read in duplicate are ignored. Therefore, specifying multiple go: embed directives for the string type does not result in a compile error as follows.
//go:embed hello.txt
//go:embed hello.txt
var hello string
It is possible to embed it literally in test.
go: embed directive before embed.FSIt is simply recognized as an empty directory.
var fs embed.FS
fmt.Println(fs)
// {<nil>}
go: embed directive and the variable that embeds the fileIn cgo (not a directive), the C language code is not recognized if there is a space between the comment description of the C language code and import" C ", whereas in go: embed, there is a space. Even if there is, it is recognized properly. However, if the space spans multiple lines, it will be formatted as one line by gofmt.
//go:embed hello.txt
var hello.txt
go: embed directive and the variable that embeds the fileAs with the space example above, it's okay to have comments.
//go:embed hello.txt
// --- comment ---
var hello.txt
I think it is possible to embed an empty directory, like when I declare a variable in embed.FS without specifying the go: embed directive, and I get a compile error. I mentioned above that ** embed a directory **, but in reality, the unit to embed with the go: embed directive is a ** file **, not the directory itself. The variable of embed.FS declared without the go: embed directive can be seen by looking at the original structure, but it has a slice of type file as a member and simply for this slice. Since it is not explicitly initialized, it will have 0 files, but if you embed an empty directory, you will expect one or more files at the end of the directory, resulting in a compile error. It's easy to imagine that Git can't commit an empty directory. As we'll see in more detail in the second part, the embedded file is treated like a package and is empty, just as you get a compile error when you import a package that is an empty directory (in this case you can call it a package anymore). I get a compile error when I embed a directory.
/This also causes a compile error because it is a file, not a directory, that is embedded. If you want to embed all the files in the directory, specify it with a wildcard.
go: embed directive without importing the embedSo at compile time, for reasons other than the often blank imports when using SQL Drivers like github.com/go-sql-driver/mysql and github.com/mattn/go-sqlite3 This is because it is defined.
This doesn't need any special explanation.
The io/fs package does not allow the path to include .., which makes the parent directory visible, because it expects the dependent files to be entirely on its underlying hierarchical filesystem. [^ 4]. The embed package also implements a hierarchical filesystem structure that follows the interface provided by the io/fs package, so the go: embed directive also prohibits the use of ... ..
Surprisingly, . pointing to the current directory is also forbidden in the hierarchical file system of the io/fs package. Therefore, the embed package is also omitted below. No literature was found to provide an explicit reason for the ban on this.
Some files, such as symbolic links and device files, cannot be embedded with the go: embed directive. Specifically, when you display the file list with ls -l, the file mode will be displayed in the form of -rw-r--r-- etc., but the first character is the normal file Files other than - cannot be embedded. Other files that cannot be embedded are described in here.
There are some other patterns that you can't specify with the go: embed directive, but let's save some fun for yourself reading this.
In this article, I have described how to use go: embed and points to note when specifying the file path. In the next sequel, we will look into the contents of go: embed.