possible inconsistency in the embed package documentation

[Manlio Perillo]

I'm reading the https://tip.golang.org/pkg/embed/ package documentation and I found a possible inconsistency.

At the end of https://tip.golang.org/pkg/embed/#hdr-Directives:

"Patterns must not match files outside the package's module, such as ‘.git/*’ or symbolic links"

and

"Patterns must not contain ‘.’ or ‘..’ path elements nor begin with a leading slash"

It seems to me that the first phrase is not necessary, since the second phrase prevents matching files outside the package module.

Thanks

Manlio Perillo

[Axel Wagner]

I don't think they do. There are two examples in the first phrase, which are not excluded by the second - the ".git" directory and a symbolic link (pointing outside of the module).

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/5e1d688e-cdb4-4b32-a06a-086a6b097064n%40googlegroups.com.

[Dmitri Shuralyov]

I think both are needed, they don't overlap. Note that the second phrase says "must not contain '.' or '..' path elements", emphasis them being a complete path element. So "./git" is disallowed by the second phrase, but ".git" is not.

[Axel Wagner]

To put it another way:

The second phrase is a lexical requirement about the pattern. It must not contain a . or .. element - whether or not the result is included in the module (e.g. "foo/../foo/bar" is not allowed either, even though it's equivalent to "foo/bar").

But, a lexical path *in* the module might still refer to a file not included in it it - either by a symlink, or by being in the .git directory (and maybe other cases I'm unaware of). So, the first phrase excludes any case where the file is not included the module, whether or not the name you refer it by lexically contains . or '..'.

Both phrases are necessary.

[Manlio Perillo]

Thanks. I was only considering the parent of the  module's root directory.

Is the concept of "outside the module" defined somewhere? 

Manlio Perillo

[Manlio Perillo]

As an example: is testdata outside the package's module?

Thanks

Manlio

[Dmitri Shuralyov]

Directories named testdata are included in the module; they're needed for tests to run. The most important thing that's left out are subdirectories that contain a go.mod file, since the content of such directories is a different module.

Some good places to look for full details include https://golang.org/ref/mod#zip-path-size-constraints and https://pkg.go.dev/golang.org/x/mod/zip.

[Axel Wagner]

I think this is the best doc about what is included in a module:

https://golang.org/ref/mod#zip-path-size-constraints

Everything not in that list is "outside" that module.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/37f4c78b-ee26-45d1-8f61-f45e2acd978dn%40googlegroups.com.

[Manlio Perillo]

https://golang.org/ref/mod#zip-path-size-constraints prevents directories that begin with a dot, but only because the directory is interpreted as a package.

It is not clear, to me, if `.git` is ignored by the `embed` directive because it is the private directory of the VCS or because it starts with a dot.

Thanks

Manlio Perillo

[Dmitri Shuralyov]

It gets pretty subtle. The ".git" directories aren't included in module zips by the go command (I don't know if this is documented anywhere, but it's very sensible behavior), but they aren't disallowed. A custom module zip may include a ".foo", "_foo", or even ".git" directory with files.

In the the phrase you mentioned:

> Patterns must not match files outside the package's module, such as ‘.git/*’ or symbolic links

Symbolic links are neither included not allowed.

.git/* files aren't included by the go tool.

As I understand, the "such as ‘.git/*’ or symbolic links" part is just an example of some common types of files that aren't included in modules. The important part of that phrase is "Patterns must not match files outside the package's module". For example, if you have this tree:

$ tree .

.

├── LICENSE

├── go.mod // module example.com/m1

├── p.go

├── p_test.go

└── nested

    ├── go.mod // example.com/m1/nested

    ├── foo.txt

    └── ...

Then p.go can't embed "nested/foo.txt", because nested/foo.txt is going to be outside of the m1 module.

If you're looking to improve package embed documentation, I suggest filing an issue. If your goal to understand this better for your own interests, I hope you find the nuanced details above interesting. :)

[Axel Wagner]

In general, embedding files from directories starting with dot ("hidden directories") works fine. But you must take care, to either mention the hidden directory explicitly, or the file you want to exclude, as otherwise, the hidden directory will be skipped by embed (see https://github.com/golang/go/issues/42328).

.git is thus special. As https://pkg.go.dev/golang.org/x/mod/zip#CreateFromDir mentions, .git and similar directories are skipped when creating the zip file of a module, because they are not deemed "part of the module" (which, I think, makes a lot of sense), so they can't be embedded based on the rule that embedded files must be part of the module (i.e. they must be included in the zip file).

It might be reasonable to spell the skippage of .git etc. out more specifically in the docs.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/d3604650-ae87-4376-9209-b1a75f6263a0n%40googlegroups.com.

[Manlio Perillo]

Thanks for the clarification.  The goal was to understand the embedding better.

I have another question.

Consider the case where example.com/m1/nested is not a separate module.

Due to the rule of "Patterns must not contain ‘.’ or ‘..’ path elements", the nested package can not embed, as an example, the LICENSE file from the example.com/m1 package.

The LICENSE file is not outside the nested package module, so I'm interested to know why it is not allowed, since this can prevent the sharing of files between packages in the same module.

Was the alternate design of assuming embedded file paths relative to the module root directory considered?

Thanks

Manlio

[Dmitri Shuralyov]

> Consider the case where example.com/m1/nested is not a separate module.

> Due to the rule of "Patterns must not contain ‘.’ or ‘..’ path elements", the nested package can not embed, as an example, the LICENSE file from the example.com/m1 package.

> The LICENSE file is not outside the nested package module, so I'm interested to know why it is not allowed, since this can prevent the sharing of files between packages in the same module.

That's a good question. I'm interested in knowing this too.

In https://go.googlesource.com/proposal/+/master/design/draft-embed.md#dot_dot_module-boundaries_and-file-name-restrictions, it says:

> In order to build files embedded in a dependency, the raw files themselves must be included in module zip files. This implies that any embedded file must be in the module’s own file tree. It cannot be in a parent directory above the module root (like ../../../etc/passwd), it cannot be in a subdirectory that contains a different module, and it cannot be in a directory that would be left out of the module (like .git).

That paragraph doesn't mention files in a parent directory within the module root.

[Manlio Perillo]

Il giorno venerdì 22 gennaio 2021 alle 20:02:02 UTC+1 dmit...@golang.org ha scritto:

> Consider the case where example.com/m1/nested is not a separate module.

> Due to the rule of "Patterns must not contain ‘.’ or ‘..’ path elements", the nested package can not embed, as an example, the LICENSE file from the example.com/m1 package.

> The LICENSE file is not outside the nested package module, so I'm interested to know why it is not allowed, since this can prevent the sharing of files between packages in the same module.

That's a good question. I'm interested in knowing this too.

In https://go.googlesource.com/proposal/+/master/design/draft-embed.md#dot_dot_module-boundaries_and-file-name-restrictions, it says:

> In order to build files embedded in a dependency, the raw files themselves must be included in module zip files. This implies that any embedded file must be in the module’s own file tree. It cannot be in a parent directory above the module root (like ../../../etc/passwd), it cannot be in a subdirectory that contains a different module, and it cannot be in a directory that would be left out of the module (like .git).

That paragraph doesn't mention files in a parent directory within the module root.

It is mentioned in:

"Because embed.Files implements fs.FS, it cannot provide access to files with names beginning with .., so files in parent directories are also disallowed entirely, even when the parent directory named by .. does happen to be in the same module."

Thinking about it, the parent package can export the embedded data variable, so it is probably not necessary for sub packages to access the files of the parent package.

Manlio Perillo