Modules and internal packages

[Michael Ellis]

I recently coded up a skeleton app to use as a starting point for projects that include a server component and a wasm (WebAssembly) client that does local work in a user's browser while communicating with the server.  

The skeleton is hosted  on Github (in a private repo until I think it's ready to publish).  

Today I decided to clone it and start an actual project.  I've run into a problem with internal package imports.  

Briefly,  the skeleton has multiple source files  that import from internal packages, e.g.

import  "github.com/Michael-F-Ellis/skeleton/internal/common"

The purpose of the internal/common package is to define structs that need to be known by both the server and the wasm component.  As such, the package will be different in each application built from the skeleton and, hence, the import statements all need to be modified to reference the new project's module name, e.g.,

import  "gitlab.com/SomeOneElse/someproject/internal/common"

Is there now no way to simply refer to "internal/common" in the import statement and have the toolchain treat it as a local import.  I've tried using the various argument to a replace directive in go.mod with no luck so far.

What's the right way to handle this use case?

[Axel Wagner]

On Thu, Mar 4, 2021 at 12:02 AM Michael Ellis <michael...@gmail.com> wrote:

What's the right way to handle this use case?

I think the right way to handle it is to modify the file. In the final code, the import path should unambiguously point to where the code can be found - regardless of whether it was using your skeleton to get started. You might write a tool to do that modification automatically, if you want to simplify things.

 

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/669b426e-5d0a-43a4-b086-e9e0c7213a9en%40googlegroups.com.

[Michael Ellis]

On Wednesday, March 3, 2021 at 6:11:38 PM UTC-5 axel.wa...@googlemail.com wrote:

On Thu, Mar 4, 2021 at 12:02 AM Michael Ellis <michael...@gmail.com> wrote:

What's the right way to handle this use case?

I think the right way to handle it is to modify the file. In the final code, the import path should unambiguously point to where the code can be found - regardless of whether it was using your skeleton to get started. You might write a tool to do that modification automatically, if you want to simplify things.

Thanks even though it's not the answer I was hoping for.  Seems to me that since the Go Authors have accorded special status to directories named "internal"  the module mechanism should recognize references to it and not require a globally unique path string.  

[Bryan C. Mills]

You should be able to use a `replace` directive to replace the full path "github.com/Michael-F-Ellis/skeleton/internal/common" or similar. If you take that approach, the skeleton module itself must not contain that package, or else the `import` statements that refer to that package would be ambiguous.

You can think of that path as specifying “the identity of the hook package” rather than “the location at which the source code for the hook package can be found”. From that perspective, the path "gitlab.com/SomeOneElse/someproject/internal/common" wouldn't make sense: it may be a completely different package with its own identity and meaning.

[Axel Wagner]

On Thu, Mar 4, 2021 at 12:55 AM Michael Ellis <michael...@gmail.com> wrote:

Thanks even though it's not the answer I was hoping for.  Seems to me that since the Go Authors have accorded special status to directories named "internal"  the module mechanism should recognize references to it and not require a globally unique path string.  

Maybe. There seem to be few downsides. Then again, your case is also fairly special.

On Thu, Mar 4, 2021 at 3:34 AM 'Bryan C. Mills' via golang-nuts <golan...@googlegroups.com> wrote:

You should be able to use a `replace` directive to replace the full path "github.com/Michael-F-Ellis/skeleton/internal/common" or similar.

Using replace would still mean that a module doesn't work, when used as a library. Generally, I feel that `replace` is being used for more and more things it wasn't designed for, leading to more and more friction and problems. I don't think we should encourage more off-label uses. 

[Michael Ellis]

On Thursday, March 4, 2021 at 2:24:11 AM UTC-5 axel.wa...@googlemail.com wrote:

On Thu, Mar 4, 2021 at 12:55 AM Michael Ellis <michael...@gmail.com> wrote:

Thanks even though it's not the answer I was hoping for.  Seems to me that since the Go Authors have accorded special status to directories named "internal"  the module mechanism should recognize references to it and not require a globally unique path string. 

 

 

Maybe. There seem to be few downsides. Then again, your case is also fairly special.

Not sure if my case is all that special.  Seems like the requirement for a full path to an internal package breaks the concept of "internal" because it gives everyone import access to whatever the internal package exports.  It would be good to have Russ and/or Ian weigh in on this.  My feeling at this point is that either "internal" should be deprecated or the module mechanism needs to honor it.

 

On Thu, Mar 4, 2021 at 3:34 AM 'Bryan C. Mills' via golang-nuts <golan...@googlegroups.com> wrote:

You should be able to use a `replace` directive to replace the full path "github.com/Michael-F-Ellis/skeleton/internal/common" or similar.

Using replace would still mean that a module doesn't work, when used as a library. Generally, I feel that `replace` is being used for more and more things it wasn't designed for, leading to more and more friction and problems. I don't think we should encourage more off-label uses. 

Agree. Replace in this instance feels like a hack. Don't get me wrong, I like the concept behind go.mod and think it's generally a strong and useful design. That being said, I also agree with the sentiment expressed in a couple of other recent thread that the present implementation imposes a burden on coders who want to quickly sketch and test ideas in a local directory.  It ought to be possible to progress at least a little way beyond Hello World before needing an external repo.

[Bryan C. Mills]

On Thu, Mar 4, 2021 at 9:54 AM Michael Ellis <michael...@gmail.com> wrote:

On Thursday, March 4, 2021 at 2:24:11 AM UTC-5 axel.wa...@googlemail.com wrote:

On Thu, Mar 4, 2021 at 12:55 AM Michael Ellis <michael...@gmail.com> wrote:

Thanks even though it's not the answer I was hoping for.  Seems to me that since the Go Authors have accorded special status to directories named "internal"  the module mechanism should recognize references to it and not require a globally unique path string. 

 

 

Maybe. There seem to be few downsides. Then again, your case is also fairly special.

Not sure if my case is all that special.  Seems like the requirement for a full path to an internal package breaks the concept of "internal" because it gives everyone import access to whatever the internal package exports.  It would be good to have Russ and/or Ian weigh in on this.  My feeling at this point is that either "internal" should be deprecated or the module mechanism needs to honor it.

I don't understand what this even means.

You're saying that you have some API for constructing a paired server component and WASM client, and the way that projects use that API is by swapping in their own "internal/common" package. That already fundamentally breaks the concept of “internal”: the "internal/common" package is a public feature of the API, not an internal detail.

 

On Thu, Mar 4, 2021 at 3:34 AM 'Bryan C. Mills' via golang-nuts <golan...@googlegroups.com> wrote:

You should be able to use a `replace` directive to replace the full path "github.com/Michael-F-Ellis/skeleton/internal/common" or similar.

Using replace would still mean that a module doesn't work, when used as a library. Generally, I feel that `replace` is being used for more and more things it wasn't designed for, leading to more and more friction and problems. I don't think we should encourage more off-label uses. 

Agree. Replace in this instance feels like a hack. Don't get me wrong, I like the concept behind go.mod and think it's generally a strong and useful design. That being said, I also agree with the sentiment expressed in a couple of other recent thread that the present implementation imposes a burden on coders who want to quickly sketch and test ideas in a local directory.  It ought to be possible to progress at least a little way beyond Hello World before needing an external repo.

I would argue that the “hack” in this case is the approach of defining the API in terms of copying in an entire tree of packages, rather than having users of the API make function calls into a specific set of (unmodified, un-relocated) packages with stable import paths.

If the API is defined in terms or reflection over a set of types, why substitute a package rather than (say) having the caller pass in a set of instances of `reflect.Type` or similar?

[Axel Wagner]

On Thu, Mar 4, 2021 at 3:54 PM Michael Ellis <michael...@gmail.com> wrote:

Not sure if my case is all that special.  Seems like the requirement for a full path to an internal package breaks the concept of "internal" because it gives everyone import access to whatever the internal package exports.

Really? I can't reproduce that. If I create a minimal module importing "github.com/Merovius/srvfb/internal/fb" and try to build that, I get an error message:

 foo.go:3:8: use of internal package github.com/Merovius/srvfb/internal/fb not allowed

So, the internal mechanism seems to work just fine, to me? Note that "every importer needs to mention the full path" is not the same as "everyone mentioning the full path can import".

 

It would be good to have Russ and/or Ian weigh in on this.  My feeling at this point is that either "internal" should be deprecated or the module mechanism needs to honor it.

 

On Thu, Mar 4, 2021 at 3:34 AM 'Bryan C. Mills' via golang-nuts <golan...@googlegroups.com> wrote:

You should be able to use a `replace` directive to replace the full path "github.com/Michael-F-Ellis/skeleton/internal/common" or similar.

Using replace would still mean that a module doesn't work, when used as a library. Generally, I feel that `replace` is being used for more and more things it wasn't designed for, leading to more and more friction and problems. I don't think we should encourage more off-label uses. 

Agree. Replace in this instance feels like a hack. Don't get me wrong, I like the concept behind go.mod and think it's generally a strong and useful design. That being said, I also agree with the sentiment expressed in a couple of other recent thread that the present implementation imposes a burden on coders who want to quickly sketch and test ideas in a local directory.  It ought to be possible to progress at least a little way beyond Hello World before needing an external repo.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/c7f87f45-b944-4f00-a6a6-ad3a8d1c5d4bn%40googlegroups.com.

[Michael Ellis]

On Thursday, March 4, 2021 at 10:14:03 AM UTC-5 axel.wa...@googlemail.com wrote:

On Thu, Mar 4, 2021 at 3:54 PM Michael Ellis <michael...@gmail.com> wrote:

Not sure if my case is all that special.  Seems like the requirement for a full path to an internal package breaks the concept of "internal" because it gives everyone import access to whatever the internal package exports.

Really? I can't reproduce that. If I create a minimal module importing "github.com/Merovius/srvfb/internal/fb" and try to build that, I get an error message:

 foo.go:3:8: use of internal package github.com/Merovius/srvfb/internal/fb not allowed

So, the internal mechanism seems to work just fine, to me? Note that "every importer needs to mention the full path" is not the same as "everyone mentioning the full path can import".

My bad. I should have tested before writing that.  Thanks for checking.  Good to know the tools are enforcing the distinction.  Still, the import path requirement does get in the way of being able to create a new application by cloning and revising an existing one without doing a recursive sed (or equivalent thereof).

[Axel Wagner]

On Thu, Mar 4, 2021 at 4:51 PM Michael Ellis <michael...@gmail.com> wrote:

My bad. I should have tested before writing that.  Thanks for checking.  Good to know the tools are enforcing the distinction.  Still, the import path requirement does get in the way of being able to create a new application by cloning and revising an existing one without doing a recursive sed (or equivalent thereof).

I agree :) And as I said, we could probably make relative imports work. But currently, the mapping from import paths to packages in a single go binary is 1-1. If we would allow you to use relative imports as well, that would be lost. Or we would have to force you to do one or the other per module. Either way, it seems like a non-trivial and potentially confusing transition. At which point we get back to "your usecase seems fairly special". Not "using internal packages", but the entire "cloning an existing project/skeleton and expect to have that just work as the jumping-off point for a new one". Note that *some* search/replace like stuff is still going to be needed anyway - at the very least, `go.mod` needs to contain a user-chosen module path.

That's why I really don't think it's worth changing. Overall, your use-case seems much better addressed by writing a tool that generates your skeleton, replacing paths as needed, instead of expecting `git clone` to serve that purpose.

 

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/35e46b0e-a730-4b27-9ca0-313ddfccf855n%40googlegroups.com.

[Michael Ellis]

On Thursday, March 4, 2021 at 10:06:01 AM UTC-5 Bryan C. Mills wrote:

I would argue that the “hack” in this case is the approach of defining the API in terms of copying in an entire tree of packages, rather than having users of the API make function calls into a specific set of (unmodified, un-relocated) packages with stable import paths.

If the API is defined in terms or reflection over a set of types, why substitute a package rather than (say) having the caller pass in a set of instances of `reflect.Type` or similar?

I'm not trying to create an API.  My initial goal is to provide a working skeleton app for my own use that incorporates:

• a server,
• a wasm client,
• a web page that loads the client,
• a common struct that represents information to be displayed in the page, changed by controls therein, and propagated back to the server through the wasm client.
• magefiles and templates that generate code in the server, client and web page such that changes in the common struct are updated in all of the above at build time.

The above functionality is working pretty well. I can modify, add or subtract members in the common struct, rebuild and run it with no other manual changes.  I posted here because of the problems I've encountered with Go modules when trying to create new app from a clone of the skeleton.

If I can make the skeleton broadly useful and make it play nice with Go modules,  I'll happily share it.  

 

[Michael Ellis]

On Thursday, March 4, 2021 at 11:01:06 AM UTC-5 axel.wa...@googlemail.com wrote:

On Thu, Mar 4, 2021 at 4:51 PM Michael Ellis <michael...@gmail.com> wrote:

My bad. I should have tested before writing that.  Thanks for checking.  Good to know the tools are enforcing the distinction.  Still, the import path requirement does get in the way of being able to create a new application by cloning and revising an existing one without doing a recursive sed (or equivalent thereof).

I agree :) And as I said, we could probably make relative imports work. But currently, the mapping from import paths to packages in a single go binary is 1-1. If we would allow you to use relative imports as well, that would be lost. Or we would have to force you to do one or the other per module. Either way, it seems like a non-trivial and potentially confusing transition. At which point we get back to "your usecase seems fairly special". Not "using internal packages", but the entire "cloning an existing project/skeleton and expect to have that just work as the jumping-off point for a new one". Note that *some* search/replace like stuff is still going to be needed anyway - at the very least, `go.mod` needs to contain a user-chosen module path.

That's why I really don't think it's worth changing. Overall, your use-case seems much better addressed by writing a tool that generates your skeleton, replacing paths as needed, instead of expecting `git clone` to serve that purpose.

 

Ok. Thanks for coherent explanation of why it may be hard to support relative imports.  The skeleton code already has a fair amount of code generation at build time, so it's not unreasonable to add an "Init" target to the magefile that modifies go.mod.  I'll look into ways (build tags?) to avoid using internal if possible.

I disagree that cloning and modifying existing projects is special.  Good lord! I (along with about a million other engineers) have been doing that for decades:-) My clients really don't want and shouldn't have to pay me to start from scratch every time.

[Michael Ellis]

Just to close the loop, I've got a solution that's working well enough for my personal use.  

It takes advantage of GitHub's Template repo feature and the gh CLI tool to simplify the process of creating a new repository with a clone of the skeleton and a local copy thereof.  Using Axel's suggestion (thx!), I added an Init section to the magefile to walk the project tree changing any import references to match the new repo before building the app for the first time.

Thanks again for the help and explanations.