Proposal for go get/import of remote versions and branches

[Erik Unger]

I really enjoy how "go get" fetches dependencies via remote package
import without any further configuration files, but there is one big
catch: there is no way to define the version of a remote package
dependency. This can be real show stopper for big and complex projects
where the assumption that "everything is just fine with the latest
version of all packages" doesn't hold. Especially not if you have to
check out an older version of the project and have to find and check
out the corresponding versions of dependencies. There is no built in
documentation of what version of a package depends on what version of
another package, so this has to be done by hand for every package
which obviously kills the whole concept of automated dependency
checkouts via go get.

This problem could be solved by allowing optional version/tag/branch
information at remote import statements.

Proposal for a repo with a sub package (the version is always specific
to the repo not the sub-package):

import "github.com/ungerik/go-start/views@06c43ac7fe"

The string after @ could be anything that identifies a version or a
date.
It should also be possible to specify a branch. Here a format proposal
for a branch at a given date and time:

import "github.com/ungerik/go-start@mybranch:2012-04-01T23:00"

go get github.com/ungerik/go-start@mybranch:2012-04-01T23:00

[minux]

'go get' don't always download the most recent version, but choose the version that

best matches your 'go version'.

For example, if you are using go1, it will prefer packages tagged or branch go1.

if you are using weekly.YYYY-MM-DD, it will prefer go.weekly.YYYY-MM-DD.

On Mon, Apr 2, 2012 at 11:18 PM, Erik Unger <ung...@gmail.com> wrote:

Proposal for a repo with a sub package (the version is always specific
to the repo not the sub-package):

import "github.com/ungerik/go-start/views@06c43ac7fe"

The string after @ could be anything that identifies a version or a
date.
It should also be possible to specify a branch. Here a format proposal
for a branch at a given date and time:

import "github.com/ungerik/go-start@mybranch:2012-04-01T23:00"
go get github.com/ungerik/go-start@mybranch:2012-04-01T23:00

What if you also want to set specific version for go-start's dependency but

go-start's author doesn't? Or consider how 'go get' can choose version for

dependencies of a non-current version of some package (you can't force

every package author to use specific versions for imports in his package).

Package versioning is so complicated that I don't think 'go get' could manage it.

[Krzysztof Kowalik]

As I once mentioned on my blog, go encoureges (or forces) to keep master or any go version related branches green... but we all know in what world we're living... because of maintainer's fault, or because of simple bugs in the code master can get broken. Anyway, inspired by discussion about goven (this app for using local copies of the dependencies) which imho is totally against the concept of go dependencies management i started experimenting with few ways of "remembering" the package's dependencies environment. Will publish all the ideas this week on my blog as well - stay tuned.

-nu7
http://areyoufuckingcoding.me

2012/4/2 minux <minu...@gmail.com>

[André Moraes]

> import "github.com/ungerik/go-start/views@06c43ac7fe"

>
> import "github.com/ungerik/go-start@mybranch:2012-04-01T23:00"
>
> go get github.com/ungerik/go-start@mybranch:2012-04-01T23:00

I really don't think that having those import statments in the code
would be usefull.

I think that the master should always be green and compatible with
previous versions, if you need to break that it's better to have a
separated repo for the incompatible changes.

That way old code will always run and new code can use the new
features of the given library.

What we have in Go is better than what we could have with something
similar to maven.

--
André Moraes
http://andredevchannel.blogspot.com/

[Erik Unger]

On Apr 2, 5:30 pm, minux <minux...@gmail.com> wrote:
> 'go get' don't always download the most recent version, but choose the
> version that
> best matches your 'go version'.

Yes, but this manages only dependencies on Go releases not version of
other remote packages.

> Package versioning is so complicated that I don't think 'go get' could
> manage it.

I think we are pretty close the optimal system, IMHO go get/import is
better than 99% of all other source code package systems out there.

[André Moraes]

> I think we are pretty close the optimal system, IMHO go get/import is
> better than 99% of all other source code package systems out there.

The 1% is the hardest one and changing so many things to reach the
100% is a difficult decision to make.

[Erik Unger]

Of course one compilation unit can only import one version of a remote
packages. Checking this intelligently with dates could require an
online lookup of date/version relationships in centralized VCS, which
would significantly slow down the build process. Thus, only one exact
import string can be used per remote repo/sub-package.

But different versions of the same package could be installed in a
GOPATH by adding the version information to the directory name of the
package. This would require a slightly different format from my
proposal due to file naming restrictions.

[André Moraes]

Sure, but compare these:

Use a version number inside every single import on every single file
Use a version number inside a file where you really need the specific
version and leave the rest to the default master
Use a different import path

Using a different import path looks easier than the others two,
off-course it has drawbacks:

The maintainer need to be carefull with the master branch.
API changes must be backward compatible
If a new API is required, then the user MUST start a new project under
a different path.

[DisposaBoy]

To keep this post on topic I won't comment about versioned dependencies, however, I was curious why the need to go through all this trouble when you can simply checkout the version, branch, tag, whatever. you want and then do a `go install` on that.

[Krzysztof Kowalik]

The other way i already mentioned many times is to fork your dependency and import the forked repository... even easier.

-nu7

2012/4/2 DisposaBoy <dispo...@dby.me>

[ara...@gmail.com]

The go get is definitely a place where I think additional options are needed.  The usage of pulling branches related to go version doesn't solve the need of needing to lock into a known version, then decide when to upgrade.  The minimalist approach on many of the code related requests I can see, this one imho is needed.

go get github.com/ungerik/go-start@mybranch:2012-04-01T23:00

+1        I think the go get needs something similar to this (although i would:    /go-start@identifier     identifier drops through branch/tag/hash )

I would also love to see:

go get github.com/araddon/go-start  --as=github.com/ungerik/go-start  

[minux]

2012/4/3 André Moraes <and...@gmail.com>

Sure, but compare these:

Use a version number inside every single import on every single file
Use a version number inside a file where you really need the specific
version and leave the rest to the default master
Use a different import path

Another problem:

what if pkgA/fileA.go import version1 of pkgB, and pkgA/fileB.go doesn't say which version it

imports? What if pkgA/fileB.go imports a different version2 of pkgB?

[André Moraes]

That's why i suggest using different import paths, they solve the
problem by simply giving different names to different things.

Also, imports are local to the file, so, A.go import (pkg/ver1) and
B.go import (pkg/ver2) they have access to different things

[minux]

2012/4/3 André Moraes <and...@gmail.com>

> Another problem:
> what if pkgA/fileA.go import version1 of pkgB, and pkgA/fileB.go doesn't say
> which version it
> imports? What if pkgA/fileB.go imports a different version2 of pkgB?

That's why i suggest using different import paths, they solve the
problem by simply giving different names to different things.

Also, imports are local to the file, so, A.go import (pkg/ver1) and
B.go import (pkg/ver2) they have access to different things

And then pkgB's init() will run twice? I think it's hardly acceptable,

because you can't know how many times a pkg's init() will be run

simply by reading one source file.

The problem become harder when you import pkgA and pkgB without knowing

that pkgA also imports a different version of pkgB and your import of pkgB and

pkgA's can't interoperate (because they are different instance).

[André Moraes]

>> > Another problem:
>> > what if pkgA/fileA.go import version1 of pkgB, and pkgA/fileB.go doesn't
>> > say
>> > which version it
>> > imports? What if pkgA/fileB.go imports a different version2 of pkgB?

> And then pkgB's init() will run twice? I think it's hardly acceptable,
> because you can't know how many times a pkg's init() will be run
> simply by reading one source file.

Yes, and this is valid, since if they came from different places they
should be different things.

For example, if both versions tries to register a database driver in
database/sql, then if they had the same name they would conflict and
panic.

>
> The problem become harder when you import pkgA and pkgB without knowing
> that pkgA also imports a different version of pkgB and your import of pkgB
> and
> pkgA's can't interoperate (because they are different instance).

This is true, but if the user MUST use different versions of the same
package, then he should be aware of those problems.

I don't advocate for using multiple versions of one package, but if
this is required than the current behavior of go is explicty and safe
(as much as possible).

[Liigo Zhuang]

+1

[minux]

2012/4/3 André Moraes <and...@gmail.com>

> The problem become harder when you import pkgA and pkgB without knowing
> that pkgA also imports a different version of pkgB and your import of pkgB
> and
> pkgA's can't interoperate (because they are different instance).

This is true, but if the user MUST use different versions of the same
package, then he should be aware of those problems.

The user might not be aware of it. He might simply want to import "pkgA" and "pkgB",

and is totally unaware the fact that pkgA uses a specific version of pkgB, and so pkgA

can't interoperate with user's import of pkgB (and what's worse, it might even result in

runtime panic as you've said), that would be quite a surprise to the user (who presumably

don't know pkgA's internals).

If you really want a specific version of some package, just control it yourself, I don't

think 'go get' could manage all these complexities.

[yy]

2012/4/4 minux <minu...@gmail.com>

I don't understand what the problem is.

Let's say you are importing "repo.com/usera/pkgA@2.0" and "repo.com/userb/pkgB", which imports "repo.com/usera/pkgA@1.0". Then, the go tool would build pkg/.../repo.com/usera/pk...@2.0.a and pkg/.../repo.com/usera/pk...@1.0.a checking different versions of the repository, but they would be different packages.

The current solution to the problem a package which breaks compatibility with previous versions (or one of them), would be to make a new package, let's say repo.com/userb/pkgA, and import that one from pkgB. This would perfectly work, because both pkgAs would be different packages. I don't see any reason why it should not work in the case of different versions of the same repository, as long as different versions are identified by an unique name.

All this said, I think it may be too early to implement this in the go tool. This is a difficult to solve problem and careful thinking is needed.

--
- yiyus || JGL .

[minux]

On Wed, Apr 4, 2012 at 7:13 PM, yy <yiyu...@gmail.com> wrote:

2012/4/4 minux <minu...@gmail.com>

2012/4/3 André Moraes <and...@gmail.com>

> The problem become harder when you import pkgA and pkgB without knowing
> that pkgA also imports a different version of pkgB and your import of pkgB
> and
> pkgA's can't interoperate (because they are different instance).

This is true, but if the user MUST use different versions of the same
package, then he should be aware of those problems.

The user might not be aware of it. He might simply want to import "pkgA" and "pkgB",

and is totally unaware the fact that pkgA uses a specific version of pkgB, and so pkgA

can't interoperate with user's import of pkgB (and what's worse, it might even result in

runtime panic as you've said), that would be quite a surprise to the user (who presumably

don't know pkgA's internals).

If you really want a specific version of some package, just control it yourself, I don't

think 'go get' could manage all these complexities.

I don't understand what the problem is.

Let's say you are importing "repo.com/usera/pkgA@2.0" and "repo.com/userb/pkgB", which imports "repo.com/usera/pkgA@1.0". Then, the go tool would build pkg/.../repo.com/usera/pk...@2.0.a and pkg/.../repo.com/usera/pk...@1.0.a checking different versions of the repository, but they would be different packages.

The user imports "repo.com/usera/pkgA" and "repo.com/userb/pkgB", but he doesn't know that usera/pkgA also imports

"repo.com/userb/pkgB@XXX", so he end up with two copies of pkgB, this might conflict, but the user won't see anything

wrong from his imports, he must dig into the implementation of pkgA to find it out. (Put it another way, should cmd/go

automatically append "@XXX" to all import of "repo.com/userb/pkgB"?)

Also, I think supporting "pkgName@Revision" will need major revision to the compiler, not just cmd/go alone.

[André Moraes]

> If you really want a specific version of some package, just control it
> yourself, I don't
> think 'go get' could manage all these complexities.

Mee to, that's why I am against that change.

Most people misunderstand the idea behind the "go" tool, especially "go get",
Like you said "go get" couldn't handle all those variations and the
best way to do that is simply to force the person to know about what
is installed.

Probably, the best way to see "go get" is: A tool to checkout
repositories under GOPATH and nothing more.

Making "go get" aware of versions would make monster.

[André Moraes]

And just to be clear, my position is to force the user to have
different import paths when he need different things.
And just using a version number on the import path isn't good enough.

2012/4/4 André Moraes <and...@gmail.com>:

[Zippoxer]

Just goven libraries you don't trust to stay compatible.

[Jan Mercl]

Have read the dozen or so previous threads about this topic?

-j

[Daniel Skinner]

Discussing a bug entering tip and avoiding this by trying to stick to a particular version is only touching on the start of a problem. There are so many other things that could come up, especially when working with an older commit.

Example: a major feature was removed b/c it was bugged in 1% of its use cases. Your case doesn't involve that 1% but centers around that feature. There's also a bug affecting a different part of the pkg where said feature is still available but a patch needs to be backported.

The cases can obviously get more complex and varied. Trying to address the first issue, using a previous version, opens the door to everything that follows. Yet all of this can be solved with a fork or depending on the environment, a bootstrap script for creating a go workspace specific to an app.

I think perhaps the better issue to address here is how to identify what version of a pkg the app was built and tested against which doesn't necessarily involve extending any of the current tools.

I've always liked how this is handled in mgo with v1/mgo, v2/mgo though I'd probably find the code a bit curious by v9/mgo.

So to wrap up my thoughts, I think it's important for someone to be able to identify what the target version of a pkg was but I don't necessarily think it has to be done automagically or with an extended syntax.

On Friday, May 10, 2013 5:15:10 PM UTC-5, travis...@gmail.com wrote:

Of course it would be wonderful if all dependency developers everywhere were good, perfect citizens, and never made breaking changes or had a bug in the repo tip.

It would be wonderful for this to be impossible: http://forums.thedailywtf.com/forums/t/27755.aspx

But we all know this isn't the world we live in. Dependency versions have to be able to be handled, or using this just won't work. At all.

[Henrik Johansson]

I get that we can't and shouldn't try to solve the transitive dependency problem but one thing strikes me as odd in the discussion about backwards incompatible changes. Why should we need to create a new directory or even a new repository for this? This is precisely what _version_ control systems was made for. You tag a certain version of the software and you know that you or anyone else can use that tag if they need it. It seems really counter intuitive to be forced (ok not really forced) to duplicate the vcs's function especially when it does it so well.

And also having an @ syntax or something else simple (?tag=v1.2 to keep url syntax) and default to todays behaviour makes for a quite nice way to at least  be able to control this if needed. I am fine with build fail if a transitive clash is detected because then we have bigger problems that have to be handled manually which is much more preferable to today when me, my colleauge and our CI server all can have different "versions" of a dependency.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Daniel Skinner]

no clue why i thought v1, v2 were inside of the mgo repo, those are branches and launchpad/bzr facilitates cloning a branch by url alone (correct me if I'm wrong, dont use bzr). This happens to work out well for `go get` where you can setup your vanity url such as labix.org/v2/mgo which actually points to the bzr mgo/v2 branch, now you have a local checkout tracking that branch.

I don't think there's any trivial way to do this with git since a single branch checkout requires additional arguments and the meta tag `go get` reads only wants a url to clone.

It almost seems worthwhile to just allow an additional param to be specified in the meta tag stating the branch/tag. Then that wouldn't entail any additional syntax for an import and cmd/go/get.go already has the rest done to check out the branch/tag.

This doesn't help much for working directly with a service like github, though I suppose one could use the gh-pages to setup accordingly, assuming you could achieve a url like github.io/pkg/tag/foo as I've never used gh-pages. This also would seem to interfere with how get.go is anticipating branches specific to go versions.

[Johannes Löthberg]

The simplest way to do this would probably be to allow the import path to also contain arguments that the go tool uses when getting the dependencies.

[Frank Schröder]

As someone who is currently in the transitive dependency hell imposed by maven in Java where pkgA imports pkgB@v1 but my code need pkgB@v2 which is incompatible with v1 I can see a clear benefit from *not* having this flexibility and being able to compile everything from a single source tree. What you suggest will create problems with transitive dependencies which are far worse than managing some dependencies yourself.

If you need a specific version of a library just fork it and manage that dependency yourself. It may sound like more work but in the end it isn't.

Frank

[Daniel Skinner]

then its not really an import path anymore but something else.

Since a vanity path is translated to the source location, it probably wouldn't hurt to allow a branch/tag to be specified when querying this and that might help a team environment where that can be maintained separately. This just follows along where a fork is unnecessary but go-get can still pull multiples tracking different parts of the same repo. Effectively, no different then a manual clone and checkout to a different directory in $GOPATH.

Everything continues to have its own path, no weird dependency questions. Such a thing would ideally be done in cmd/go/get.go to work with each supported vcs but I'd imagine a clever person could manage it without any changes even, noting the issues I previously mentioned with git.

--
You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/0avuiWURSQk/unsubscribe?hl=en-US.
To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.

[Henrik Johansson]

Is the script used standalone or as part of a fabric invocation? Fabric noob here. Looks straight forward but i lack fabric knowledge.

On Jun 5, 2013 11:16 AM, "William Zhang" <jolly...@gmail.com> wrote:

We use Fabric(a Python library and command-line tool) to handle that;

code is the best comment :)

DEPENDENCIES=(
    'github.com/kr/pretty@195524255c32890818bd298d996e3dbe353fcb57',
    'github.com/bmizerany/assert@e17e99893cb6509f428e1728281c2ad60a6b31e3',
    'github.com/axw/gocov@bcb74f1fa6b05c843cb7888551bc3ca4096ec473',
    'github.com/alecthomas/gozmq@9e15b37d7d6fe08a1a31b678d4e7eaa488986a2e',
)

def init():
    '''Updateing dependencies'''
    for dep in DEPENDENCIES:
        _go_get(dep)

def _go_get(pkg):
    if '@' in pkg:
        git_repo, git_hash = pkg.split('@')
    pkgpath = os.path.join(GOPATH, 'src', git_repo)

    _git_pull(git_repo, pkgpath)
    _git_update(pkgpath, git_hash)

def _git_pull(pkg, pkgpath):
    if os.path.isdir(os.path.join(pkgpath, '.git')):
        title('pulling %s from origin master' % pkg)
        with lcd(pkgpath):
            local('git pull origin master')
    else:
        title('cloning %s to %s' % (pkg, pkgpath))
        scheme = 'http'
        if pkg.startswith('code.google.com') or pkg.startswith('github.com'):
            scheme = 'https'
        local('git clone %s://%s.git %s' %(scheme, pkg, pkgpath))

def _git_update(pkgpath, revision):
    with lcd(pkgpath):
        local('pwd')
        local('git checkout -q %s' %revision)

[ma...@dealyze.com]

As someone who has spent over a year writing code in go, this proposal seems like a no brainer to me. The lack of versioning is the single biggest issue I've encountered in go's otherwise perfect dependency management system. In my opinion this is the biggest thing holding go back as a language for wider server side adoption. It is the only thing that prevents me from enthusiastically recommending this language to friends and peers.

I'm really confused as to why this is so controversial. I understand that it will complicate 'go get' slightly and make it download more files (potentially), but it seems completely doable if you just prepend the tag name to the package name in GOPATH. The initial go get on a project will take longer, but all subsequent times the package versions will be cached.

Are there other objections that I'm missing here or is that really the only reason go's dependency management is doomed to be practically unusable for ever?

[Frank Schröder]

The reason this isn't possible IMO is that pinning versions does not solve the transitive dependency problem since this isn't solvable automatically. (A -> C1, B -> C2, which Cx do you import, which Cx.init() do you run, ...)

By not providing this feature Go forces the developer to solve this problem himself and this forces you to think about whether you actually need a library or not since now adding a library has a cost. This also puts (positive) pressure on library authors to require as little external dependencies as possible - preferably zero - which again hopefully leads to better libraries.

Go is designed to simplify software development for large teams and codebases and there this is a real problem.

This is by intention and not because it would be difficult to implement. Allowing this feature would lead to version and dependency hell.

Frank
--
Frank Schröder
Amsterdam, NL
PGP - DDA53977

> --
> You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/0avuiWURSQk/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.