My trouble of local package referencing

[Ally Dale]

Hi all,

I was confused that why golang do not support referencing local package.

eg: import "./local2" //error: local import "./local2" in non-local package

I have upload a test project here:

https://github.com/vipally/localpackage

Here is my trouble:

My project path is: github.com/vipally/localpackage The local package reference relation is: main <- local2 <- local1 In "main.go" use such to reference local package local2: import "github.com/VIPALLY/localpackage/local2" Someone who forked this projcet as "github.com/someone/localpackage". But how can his project working by avoid following change? import "github.com/SOMEONE/localpackage/local2"

Here maybe one solution:

1. use package comment to specify root of local project in projcetroot

package main // import "#"

2. use someway to reference local package

import "#/local2"

REFERENCE: https://github.com/vipally/localpackage

Thanks for reading!

Ally

[Axel Wagner]

Hi,

there are two things, people might mean, when they say "fork":

a) The original meaning: Taking an open source project and adopting it, maintaining your own changes. In this case, the forked package becomes a different one. It has different code, maintained by different people so it makes total sense, for it to have a different name. In this case, changing the import path is the correct and only thing to do; you want this other, new package, not the old one.

b) The meaning coopted by github: Creating a clone, which contains largely the same code and is meant to track the upstream repository. Only used, to publish changes, so upstream can pull them (hence "pull request"). In this case, this new package isn't really supposed to have an identity at all; it's just a publicly hosted copy of the original for convenience. 

This second case, is what people often are concerned with, when they complain about this issue. But it seems very reasonable to me, that this notion of fork does *not* support go-getting in the same way. After all, it's not a package to be distributed, it only exists, so that upstream can pull the changes to check them. And upstream can still do that, for example:

$ cd $GOPATH/src/github.com/upstream/package

$ git remote add fork g...@github.com/fork/package

$ git fetch fork

$ git checkout fork/master

$ go test ./...

So, the gist is: If you are doing the second kind of fork, you can't use go-get, but that's a good thing; it's not a distributable package. If you are doing the first kind of fork, you have to change the import paths to the new identity, but that's a good thing: You are clearly saying, which of the two, now distinct packages, you need.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ally Dale]

Hi,

I've followed your idea, it seems that go team doing this purposely.

But I don't think this is "completely reasonable".

It seems like forcing project to put an assertion "Where I am".

As our consensus, a good project is surely with "high cohesion", but never care "Where I am".

Imagine that if "go.exe" has an assertion [assert(GoRoot == "c:\go")] in function main.

If we want to install golang in "d:\go", sorry, it doesn't work. You have to fork another copy of golang src, and make the change [assert(GoRoot == "d:\go")] as another project.

It looks strange, and nobody wanna doing like this.

在 2017年11月16日星期四 UTC+8下午5:00:15，Axel Wagner写道：

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

[Albert Tedja]

Yes, it's a debatable subject within the Go community. See this blog for example:

https://medium.com/@c9s/golang-the-annoying-remote-import-path-c6c7e76517e5

There are some tricks you can do using a different git remote, `git remote add myfork github.com/yourfork/project`

Then when you push, you call `git push myfork master`

I am not too keen of using this trick. It's kind of like banging your computer case to fix it. Go paths and dependency management has been broken if you use it in the manner not intended by its original design, and they are fixing it right now.

[Volker Dobler]

On Friday, 17 November 2017 02:43:48 UTC+1, Ally Dale wrote:

[...]

It seems like forcing project to put an assertion "Where I am".

As our consensus, a good project is surely with "high cohesion", but never care "Where I am".

That's true and still the case.

The "Where I am?" arises for go get where it is an obvious requirement.

For go build et al. a certain notion of "where is the stuff" on the filesystem

is still needed and unarguable reasonable.

That different projects (and a github-style fork _is_ a different project)

might or even should have different notions of "Where I am" was explained

very well by Axel.

V.  

[Axel Wagner]

What kind of confuses me around these discussions is, that there are rarely complains about having to do the same thing with, say, python libraries. If I press "fork" on a python library on github, people can't just "pip install" it and get crackin'. They have to manually clone it and put it in the correct spot to be found by the interpreter, or I have to choose a new name for it and upload it to PyPI, so people can find it.

Well, same difference, just that a) in Go, the name/identity of a package is conventionally scoped via a domain name, so that we don't need to give a central authority the power over all our package names, to avoid conflicts and b) this enables a cute, additional mechanism for discovering the code-location, downloading it and storing it in the right place for the compiler to discover it. But these are extras. Go isn't really doing anything special or weird or different from anyone else, except that it added a couple of conventions on top of it, to make it more convenient than in other languages to distribute packages.

Stop thinking of Go import paths as some weird and strange concept screwing up your life. It's just a name. And the whole purpose of a name is to identify; so if you change the identity, change the name too and if you don't want to change the name, don't be surprised that you can't just coopt some other packages identity.

--

You received this message because you are subscribed to the Google Groups "golang-nuts" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.

[Ally Dale]

Hi,

Here are two examples of "hello world" main packages:
<ProjectRoot>/withoutlocal
<ProjectRoot>/withlocal <- github.com/vipally/localpackage/local

"withoutlocal" works well anywhere <ProjectRoot> is, even out of GoPath.
"withlocal" works only when "<ProjectRoot> = <GoPath>/github.com/vipally/localpackage"
How does go team think about this difference?

It makes "withlocal" packages non-independent due to reference "LOCAL" packages as "GLOBAL" style.
If I want my package works well anywhere, I have to write all code into one "LARGE-package".
Just like: all.Printf/all.OpenFile/all.GOROOT
Does this go team recommended?

We must explicit followed priorty of go package find process:
<ProjectRoot>: with highest-priorty path to find local packages.
<Vendor>     : with second-priorty path to find explicit-version of local-referenced third-party packages.
<GoRoot>     : with third-priorty path to find standard packages.
<GoPath>     : with lowest-priorty path to find third-party packages.

Think about that not every go-project is wrote for open-souce(aim to share with others).
Thousands of private go-projects(eg:gameservers) focus their own particular logic-flow only 
and never shared private-packages.
We just called these projects "independent-projects".
Because they have hundreds-of private-packages but no one is wrote for share.

That is to say, they never care "where I am", but "what I need".

Unfortunately, these kind of projects are always "huge". 
Maybe millions-of lines or thousands-of private-packages reference inside?

In this case, change project name or source control server become heavy work, 
because the working path changes and thousands-of private-packages reference code have to be update.

But if local-packages are referenced by "#/modules/module1" style, 
everything is change the name of project root only then.

How do you think about the difference between such styles of referencing local-packages then?
"#/modules/module1"
"<GoRoot>/server/user/project/modules/module1"

Details: https://github.com/vipally/localpackage#examples-of-withwithout-local-package-reference

在 2017年11月17日星期五 UTC+8下午3:31:31，Axel Wagner写道：

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

[Axel Wagner]

On Sat, Nov 18, 2017 at 12:19 PM, Ally Dale <vip...@gmail.com> wrote:

Hi,

Here are two examples of "hello world" main packages:
<ProjectRoot>/withoutlocal
<ProjectRoot>/withlocal <- github.com/vipally/localpackage/local

"withoutlocal" works well anywhere <ProjectRoot> is, even out of GoPath.
"withlocal" works only when "<ProjectRoot> = <GoPath>/github.com/vipally/localpackage"
How does go team think about this difference?

Disclaimer: I'm not on the go team. This is how I view these problems, personally.

There is none. There are no "local" packages (a better, more correct phrasing, would be that there *shouldn't be* any local packages; they are still in the spec, but discouraged). There are two packages

github.com/vipally/localpackage

github.com/vipally/localpackage/local

Forking the repository creates two new packages (which might or might not be intended, see below):

github.com/forker/localpackage

github.com/forker/localpackage/local

where

github.com/forker/localpackage imports github.com/vipally/localpackage/local

Now, my message above applies. There are several semantic intentions possible from the fork (and the github overloading of the term "forking" is, what muddies them):

1. forker intended this to be a "soft-fork", only used to publish a branch to be pulled by vipally. In that case, they intend it to be a copy of the repository github.com/vipally/localpackage, the creation of new, public Go packages is an unintended side-effect of the way go-get and github forks interact. It is WAI (and working as in every other language) that they have to manually make sure, the compiler finds it under the name github.com/vipally/localpackage; usually by adding their soft fork as a remote to the directory in $GOPATH and checking out their own branch.
   
2. forker intended this to be a "hard-fork" of vipally/localpackage and not a fork of vipally/localpackage/local (i.e. they want to continue to use upstream). In that case, the creation of the github.com/forker/localpackage Go package is an intended effect, the forker/localpackage/local is an unintended side-effect. It is WAI (and as in every other language) that the soft-forked subdirectory isn't used by anyone. It should be deleted, to avoid anyone accidentally using it. No import paths have to change.
3. forker intended this to be not be a fork of vipally/localpackage and a "hard-fork" of vipally/localpackage/local. In that case, creation of github.com/forker/localpackage was an uninteded side-effect, remedied by renaming forker/localpackage/local to forker/local and publishing that. It is WAI (and as in every other language), that anyone wanting to use the hard-forked new package needs to change their import paths.
4. forker intended this to be a "hard-fork" of vipally/localpackage and all its subrepositories. In that case, the creation of any new packages is not a side-effect, but an intended state. It is WAI that forker now has to change the name of the package they want to use in forker/localpackage - if nothing else, then to distinguish it from the former cases.
   

So, the distinction with other languages comes from two things:

a) Go doesn't have "subpackages". Every package has an identity completely disconnected from any other package, even though source control might choose to distribute them together.

b) Go does not require you to push to a centralized package-registry to publish your package, but adds a discovery mechanism based on the import path instead.

Arguably, most of the confusion comes from the fact, that Go has "magic" to allow github.com/user/repo/package to be go-get'ed (instead of requiring what today is called a "vanity import") in conjunction with github's decision to publish soft-forks implicitly under the forking username. It muddies the distinction between an intentional, hard fork and republishing of a package and a soft fork, that is supposed to just present a temporary proxy to an upstream repo.

But there still isn't an actual diference between Go and Non-Go, here. Once you realize a) the import path is the identifier of the package and b) creating a go-gettable github-repository is the conceptual equivalence of publishing something to Hackage/PyPI/npm/CPAN… under that ID. You would not expect "pip install django" to fetch your django-fork and you wouldn't use "pip install django" to start hacking on it. You would clone the repository locally, tell your python interpreter where to find the code and then add your fork as a remote. Do the same in Go.

 

(Note: There is a valid point in that Go makes it harder than languages with subpackages, to maintain a "hard fork" of a package, as you have to continually rebase the import path change to point to your hard fork. That is true, but 1) not usually the complaint brought forth, 2) IMO a rare enough case that it's a small price to pay for a good packaging/discovery mechanism in general and 3) I'm not even sure it's a bad thing. Hard forks most of the time suck for everyone involved)

It makes "withlocal" packages non-independent due to reference "LOCAL" packages as "GLOBAL" style.
If I want my package works well anywhere, I have to write all code into one "LARGE-package".
Just like: all.Printf/all.OpenFile/all.GOROOT
Does this go team recommended?

We must explicit followed priorty of go package find process:
<ProjectRoot>: with highest-priorty path to find local packages.
<Vendor>     : with second-priorty path to find explicit-version of local-referenced third-party packages.
<GoRoot>     : with third-priorty path to find standard packages.
<GoPath>     : with lowest-priorty path to find third-party packages.

Think about that not every go-project is wrote for open-souce(aim to share with others).
Thousands of private go-projects(eg:gameservers) focus their own particular logic-flow only 
and never shared private-packages.
We just called these projects "independent-projects".
Because they have hundreds-of private-packages but no one is wrote for share.

That is to say, they never care "where I am", but "what I need".

Unfortunately, these kind of projects are always "huge". 
Maybe millions-of lines or thousands-of private-packages reference inside?

In this case, change project name or source control server become heavy work, 
because the working path changes and thousands-of private-packages reference code have to be update.

Project name, sure, that's a huge amount of work. But it's a huge amount of hard to automate work *anyway*, as usually a project name appears in more places than just an import path. Adding a trivially automated, tiny amount of work to rewrite import paths, for such a rare case, seems like an okay tradeoff to me.

Changing the source control server is not harder in Go. Use your own Domain for your package paths, then simply swap out the meta-tag. Easy as pie.

But also, if you are not publishing your repositories, what does it matter, whether they are go-gettable?

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.

[jake...@gmail.com]

In a large, multi developer, project I worked on we used gb. It was a "large executable" style project, with may local sub packages that were intended to provide code organization, not to be reused by other projects. In addition to vendoring support, using gb makes each project a completely independent go universe. You can git clone any version of your source to any location on your hard drive, `cd` there, and it will build. You don't need to change any enthronement variable or anything ... just `cd` and build. All of your packages go right under /src and the vendored ones go under /vendor/src. Your packages can even have simple paths like /src/myapp.

This essentially accomplishes what you are looking for, albeit in a pretty heavy handed way. A few things to note:
* We made the decision to use gb before go had built in vendoring support. I'm not sure what its future look like now.
* Gb requires you to vendor all your dependencies. I would only go this route if that sounds like a win, rather than a hassle.
* If you need to run a go tool that gb doers not support, it is possible to do so by carefully constructing your GOPATH.

[Ally Dale]

Thanks for replying all above.

The main point of this topic is probably to discuss "what's the right way to manage local-only packages".

1. Axel has point out that "why local-packages is not necessary in golang"

2. Jake has given an effective way to use "vendor" path to fix local-refering problem.

But this is with a liitle defective that it's depart from vendor's original goal--manage explicit-version of third-party packages.

3. Goalng has a local package refer solution with "./xxx" "../xxx" style. But this is not recommeded by go team.

a)It may cause out-of-control problems(“../.../xxx” maybe refer to a path out of my project).

b)It doesn't work when working path is under GoPath(error: local import "./local2" in non-local package) .

c)local and global package reference style cannot working well together.

4. "#/xxx" style reference seems a best solution like "vendor" to fix local-package reference problem.

Without any hard code "where I am" exists in any go files.

And it will works well together with current global-style, because "#" will be replaced with <ProjectRoot> by compiler automatically.

在 2017年11月18日星期六 UTC+8下午7:19:33，Ally Dale写道：

[Axel Wagner]

On Mon, Nov 20, 2017 at 5:03 AM, Ally Dale <vip...@gmail.com> wrote:

4. "#/xxx" style reference seems a best solution like "vendor" to fix local-package reference problem.

Without any hard code "where I am" exists in any go files. 

And it will works well together with current global-style, because "#" will be replaced with <ProjectRoot> by compiler automatically.

What is a "Project"? This is not a concept that exists in Go (and shouldn't exist). Note, among other things, that GOPATH does not necessarily contain any information about repositories, in case that should be the answer. Especially monorepos might vendor their dependencies in a way that strips this info or makes it invisible to the go tool.

Again, it is important to note, that "local packages", or "subpackages" are not a thing in Go. Making up a notation for referencing something that doesn't exist seems pretty futile.

[Ally Dale]

Hi all,

I have made a patch of golang to support [import "#/foo"] style reference.

Golang https://github.com/vipally/go

TestProject https://github.com/vipally/localpackage/tree/master/localroot/src

Manage private-only golang-projects by replacing GoPath with LocalRoot.

Details: https://github.com/vipally/localpackage/tree/master/localroot/src#localroot

LocalRoot is a <root> directory that contains such patten of sub-tree "<root>/src/vendor/" up from current path. A LocalRoot has the same tree structure with GoPath and GoRoot. Actually, a LocalRoot is a private GoPath that is accessible by sub-packages only.

This is the minimal state of a valid LocalRoot: LocalRoot │ └─src ├─vendor │ ... └─...

Refer local-only packages with [import "#/x/y/z"] style.

I have made a test project here:

https://github.com/vipally/localpackage/blob/master/localroot/src/main/main.go#L1

As my expectation, it works well wherever LocalRoot is, even out of GoPath.

Actually, a LocalRoot is a private GoPath that is accessible by sub-packages only.

If a private project use local and vendor third-party packages only,

it will have nothing to do with GoPath.

1. LocalRoot under GoPath

Gopath is : E:\gocode\src

ProjectRoot is : E:\gocode\src\github.com\vipally\localpackage\localroot

ThisPackagePath: E:\gocode\src\github.com\vipally\localpackage\localroot\src\main

ReleatGopath is: github.com\vipally\localpackage\localroot\src\main

**********************************************************

main import [#/locals/local2]

local2 import [#/locals/local1]

local1 import [#/publics/public1]

public1 import noting

main import [golang.org/x/debug/macho(local vendor)]:

macho.CpuAmd64

**********************************************************

2. LocalRoot out of GoPath

Gopath is : E:\gocode\src

ProjectRoot is : E:\localpackage\localroot

ThisPackagePath: E:\localpackage\localroot\src\main

ReleatGopath is: ..\..\localpackage\localroot\src\main

**********************************************************

main import [#/locals/local2]

local2 import [#/locals/local1]

local1 import [#/publics/public1]

public1 import noting

main import [golang.org/x/debug/macho(local vendor)]:

macho.CpuAmd64

**********************************************************

Details: https://github.com/vipally/localpackage/tree/master/localroot/src#localroot

在 2017年11月20日星期一 UTC+8下午2:16:48，Axel Wagner写道：