Need for official package manager

[Kevin P]

Go is getting to a point that it really needs official package manager. I think gopm https://github.com/gpmgo/gopm has the right idea and hopefully can be built into the go command.

A simple way to handle this would be.

"go pkg install" would read the .gopkg file that lists the remote repos and install everything listed in this file to .pkg/ directory.

Then all the imports can reference back to this .pkg directory with the packages. e.g (import "../.pkg/github.com/bitly/go-simplejson").

Just a thought.

[omarsharif...@gmail.com]

Whats wrong with go get/install?

[egon]

The recommended way is vendoring:

http://nathany.com/go-packages/

https://code.google.com/p/go-wiki/wiki/PackageManagementTools

But there are tools for alternative use cases.

[Kelsey Hightower]

I've been enjoying the vendoring approach using Godep https://github.com/kr/godep. I'm starting to feel that vendoring is the right approach to this problem, mainly because of the ability to checkin all the dependancies in the same repo.

[Gustavo Niemeyer]

On Tue, Mar 18, 2014 at 12:36 PM, egon <egon...@gmail.com> wrote:
> The recommended way is vendoring:

And if you're the author of a package people use in the wild, please
consider adding the major version of the package to the URL using
gopkg.in or anything else of your preference, and change that major
version once the API is broken, so that dependent packages can
continue working.


gustavo @ http://niemeyer.net

[Gustavo Niemeyer]

Vendoring is great for having a completely controlled environment and
reproducible builds, but we don't have to stop there. Adopting stable
API practices would improve the package ecosystem at large. It allows:

- "go get" to continue working for packages that have transitive dependencies
- vendored packages to be more easily updated
- different package API versions to receive important updates (security, etc)

etc.

We don't need to change "go get" to get these sorted. A simple major
version number in the package URL, using gopkg.in or anything else,
already improves the scene.

> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--

gustavo @ http://niemeyer.net

[Jan Mercl]

Version numbers, in the general case, do not enable a solution to the
versioning problem which would be both automatic and correct (pick
one). This has been discussed ad nausea in this ML and this post is
not intended to to revive any such discussion.

-j

[Gustavo Niemeyer]

The whole standard library has an API being held stable by the Go 1.X
promise, and mgo has been using versioned stable APIs as
labix.org/v2/mgo with very good results, and explicit appreciation
from developers, for years as well.

Such personal dislikes for version numbers are irrelevant.

[Ross Light]

I remember there being a ML that was created just to discuss this issue in general.  Don't know if anything came of it, but worth discussing there:

https://groups.google.com/forum/#!forum/go-package-management

Ross Light | Software Engineer | li...@google.com                       

--

[Jan Mercl]

On Tue, Mar 18, 2014 at 5:27 PM, Gustavo Niemeyer <gus...@niemeyer.net> wrote:
> Such personal dislikes for version numbers are irrelevant.

That's silly. Math is not a person and has no dislikes nor likes. Once
it's proven that something is non computable, the question is answered
once and forever.

-j

[Gustavo Niemeyer]

Yeah, math is great.

Now please excuse me as I go back to stabilizing another API.

[Gerard]

I am just a noob with all the versioning stuff. And I like it that way. It's too complex for my simple mind.

What I would like to see, and I think that's just a matter of time, is that the OS designers start to adopt Go packages. I think that's the easiest way to solve the depency issues.

[Gustavo Niemeyer]

On Tue, Mar 18, 2014 at 3:03 PM, Gerard <gvds...@gmail.com> wrote:
> I am just a noob with all the versioning stuff. And I like it that way. It's
> too complex for my simple mind.

You can ignore most of what "versioning" means to people. The
important idea is trivial to understand: don't handle revision A and
revision B as compatible if revision B would break people that were
using revision A. This allows everything that depends on the software
sitting at revision A to continue working, and allows you to break
away into revision B to evolve your API. One way to do that is to call
things that are <= revision A as version 1 and things that are >=
revision B as version 2 of your API.

gopkg.in is a handy way to do that kind of thing, but you can
implement that same underlying concept in many other ways.


gustavo @ http://niemeyer.net

[Gerard]

Sorry, I didn't make myself clear. I am aware of the package management issues.

But that's not what I wanted to say. I just think it's better if "we" don't reinvent the wheel. Package management is IMO best handled by OS designers.

[Konstantin Khomoutov]

On Tue, 18 Mar 2014 11:14:45 -0700 (PDT)
Gerard <gvds...@gmail.com> wrote:

> Sorry, I didn't make myself clear. I am aware of the package
> management issues.
>
> But that's not what I wanted to say. I just think it's better if "we"
> don't reinvent the wheel. Package management is IMO best handled by
> OS designers.

True. Until you *have to* deliver something custom to a third-party.
Then Go's approach of building a single executable suitable for "xcopy
deployment" wins big time. And then to have reproducible builds you
have to have a versioned environment. Otherwise you will be unable to
reproduce the bug they report (and possibly introduce new bugs just by
compiling with untested versions of 3rd-party libraries).

(Being a Debian maintainer I appreciate system approach to
managing libraries, but I as well appreciate an ability to deliver an
application which has zero dependencies.)

[Gerard]

The problem of delivering an application with zero dependencies is easy solved with just embedding the specific dependencies. e.g. in "app/external/lib".

Still no need for reinventing the wheel (IMO).

But talking to a Debian maintainer: Are you aware of Go packages being introduced in Debian? I mean on a serious scale?

[Gustavo Niemeyer]

On Tue, Mar 18, 2014 at 3:14 PM, Gerard <gvds...@gmail.com> wrote:
> But that's not what I wanted to say. I just think it's better if "we" don't
> reinvent the wheel. Package management is IMO best handled by OS designers.

That doesn't work for a number of reasons. I won't dive into them, but
I'm completely certain that "OS designers" would love to hear that the
Go community is behind some sort of API stability concept.


gustavo @ http://niemeyer.net

[Jan Mercl]

On Tue, Mar 18, 2014 at 6:59 PM, Gustavo Niemeyer <gus...@niemeyer.net> wrote:
> Yeah, math is great.

Yep, one of the great things in math is that a particular case is not
sufficient to prove the mechanism works (or not) in the more general
case.

The proof of the mechanism not working in the general case, by
counterexample(s) provided, was presented in this ML more than once.
That can be ignored; however that's never enough to disprove anything.

-j

[Gustavo Niemeyer]

On Tue, Mar 18, 2014 at 3:37 PM, Jan Mercl <0xj...@gmail.com> wrote:
> Yep, one of the great things in math is that a particular case is not
> sufficient to prove the mechanism works (or not) in the more general
> case.

Another great thing about math is that some dude saying things on a
mailing list proves nothing.



gustavo @ http://niemeyer.net

[Henrik Johansson]

Of course versioning works. Nowhere on this list has it been proven that versioning does not work.

It does not work with the Go tools as they are today and the plethora of solutions out there has a lot to be desired to say the least.

In general however the concept of versions works but the implementations are hard and fraught with traps.

[Gerard]

You are certainly right about the API stability.

But maybe it's best if I don't get involved in this discussion. I think I have said already more than I should have.

[Aram Hăvărneanu]

On Tue, Mar 18, 2014 at 4:06 PM, Kevin P <dco...@gmail.com> wrote:
> Go is getting to a point that it really needs official package manager.

No.

--
Aram Hăvărneanu

[Gustavo Niemeyer]

On Tue, Mar 18, 2014 at 3:49 PM, Henrik Johansson <dahan...@gmail.com> wrote:
> Of course versioning works. Nowhere on this list has it been proven that
> versioning does not work.

The problem is that people read too much in what "versioning" means,
and that gets people all excited about arguing with each other about
how everything is broken.

To avoid these bikesheds, we should focus on API stability. This is a
much simpler and more fundamental concept in software development. If
we improve on that alone, it's already a major win. We can also safely
ignore anyone that bluntly disagrees as living in a different world.


gustavo @ http://niemeyer.net

[Jan Mercl]

On Tue, Mar 18, 2014 at 7:42 PM, Gustavo Niemeyer <gus...@niemeyer.net> wrote:
> Another great thing about math is that some dude saying things on a
> mailing list proves nothing.

Either a disprovement an existing proof exists or the proof holds.
It's that simple.

-j

PS: Would you mind stopping the continuous attempts to divert the
discussed math in a way which involves any persons?

[Jan Mercl]

On Tue, Mar 18, 2014 at 7:49 PM, Henrik Johansson <dahan...@gmail.com> wrote:
> Of course versioning works. Nowhere on this list has it been proven that
> versioning does not work.

Of course versioning cannot work (in the general case and in the both
correct and automatic way [like using version numbers in URL combined
with the go get mechanism]) and it has been proven in this ML before.

-j

[Gustavo Niemeyer]

On Tue, Mar 18, 2014 at 4:09 PM, Jan Mercl <0xj...@gmail.com> wrote:
> Either a disprovement an existing proof exists or the proof holds.
> It's that simple.

Evolving ideas requires conversation, Jan. Your presence is not
required, though.


gustavo @ http://niemeyer.net

[Gustavo Niemeyer]

Not only it can work, but it does work in practice. The hundred of packages
depending on mgo in any way [1] will not face an API breakage. When that
becomes necessary, v3 will be introduced.

If you have "mathematical proofs" that this is not true, please
provide them, and include proper context for why any explanations are
relevant to this context, math style.

[1] http://go-search.org/search?q=mgo


gustavo @ http://niemeyer.net

[Jan Mercl]

On Tue, Mar 18, 2014 at 8:22 PM, Gustavo Niemeyer <gus...@niemeyer.net> wrote:
> On Tue, Mar 18, 2014 at 4:09 PM, Jan Mercl <0xj...@gmail.com> wrote:
>> Either a disprovement an existing proof exists or the proof holds.
>> It's that simple.
>
> Evolving ideas requires conversation, Jan. Your presence is not
> required, though.

NIce try. However, proving math facts doesn't require discussing
persons. Nor does hand waving disprove proofs.

Let's make it really simple. There is a publicly available proof that
version numbers do not enable a solution to the versioning problem
that is both correct and automatic (just search the ML for it). The
mentioned hand waving seems to attempt to indicate that a disprovement
of the said proof is available. Congrats in that case! Just one small
question remains - why don't simply post the disprovement here for
everyone to see and check (as is the case of the existing proof)?

Is there anything wrong nowadays with peer review? ;-)

-j

PS: Would you mind stopping the senseless attempts to divert the

[Gustavo Niemeyer]

On Tue, Mar 18, 2014 at 4:44 PM, Jan Mercl <0xj...@gmail.com> wrote:
> On Tue, Mar 18, 2014 at 8:22 PM, Gustavo Niemeyer <gus...@niemeyer.net> wrote:
>> On Tue, Mar 18, 2014 at 4:09 PM, Jan Mercl <0xj...@gmail.com> wrote:
>>> Either a disprovement an existing proof exists or the proof holds.
>>> It's that simple.
>>
>> Evolving ideas requires conversation, Jan. Your presence is not
>> required, though.
>
> NIce try. However, proving math facts doesn't require discussing
> persons. Nor does hand waving disprove proofs.

That's what I thought. Now please allow me to have sane conversations.

Thank you.


gustavo @ http://niemeyer.net

[Jan Mercl]

On Tue, Mar 18, 2014 at 8:31 PM, Gustavo Niemeyer <gus...@niemeyer.net> wrote:
> Not only it can work, but it does work in practice. The hundred of packages
> depending on mgo in any way [1] will not face an API breakage. When that
> becomes necessary, v3 will be introduced.

Is that a claim that a particular case prove the general one? If so,
then I can rest my case. Thank you ;-)

> If you have "mathematical proofs" that this is not true, please
> provide them, and include proper context for why any explanations are
> relevant to this context, math style.

It has been said so many times before: it's in this ML. I think the
time frame was like last autumn (or so) and it was a long(ish) thread,
IIRC.

-j

[Jan Mercl]

On Tue, Mar 18, 2014 at 8:45 PM, Gustavo Niemeyer <gus...@niemeyer.net> wrote:
> That's what I thought. Now please allow me to have sane conversations.

You're welcome to provide the disprovement of the existing proof in
any future time. Unfortunately, it seems to be not yet publicly
available as of now.

No problem, math is known to be quite patient.

-j

[Henrik Johansson]

One example is all that is needed to invalidate "it does not work". True it does not extend to the general case but it shows that it can work.

I think only generics inspire more heat on this list. ;-)

[roger peppe]

Personally I think there is a need for both stable APIs *and* repeatable builds. Gustavo's efforts are a great step towards the former direction, and I like what Keith Rarick's godep tool contributes towards the latter.

[Dan Kortschak]

I think the issue here is that Jan is including a requirement for 'automatic' in the system, in which case it is the case that it's not solvable. The approach that has been proposed in having version numbering in the package is not automatic, but works for some level of "well enough" for many cases for many people.

Can we move on.

[Gustavo Niemeyer]

Yeah, godep is great!

[chris dollin]

Much heat, excessively personal. Lay it off, chaps.

It's true that there's no fully-general solution to "the versioning

problem".

Gustavo isn't claiming to offer such a solution. He's offering

a partial but useful approach to part of that problem.

Chris

--
Chris "allusive" Dollin

[DV]

I must be completely missing the point of what "versioning" means here, because it doesn't seem like a difficult problem to "solve", as it's been solved many times before, or am I missing something? 

In .NET land, there are what's called NuGet packages which have versions on them and you can say that package A needs version 2.1 (or greater, for example) of package B, and *exactly* version 1.1 of package C, or any combination thereof. And it's all resolved automatically and recursively. Conflicts can arise, yes, but it's rare, and can even be sometimes resolved with clever binding redirects and other techniques at run-time. 

Forget about .NET land for now, what about Clojure + Leiningen (which is fantastic), or Maven, or Linux package management?

Naturally, these systems aren't perfect (or 100% "correct", I guess, whatever that means), but why is 100% mathematical correctness suddenly important? Should we also not use IEEE floating point numbers because they're not 100% mathematically correct as well? This doesn't have to be a quasi-existential, mathematically-derived solution to a problem that's been mostly solved for years now, by various OS's and package managers. 

I think "go get" is great, when we're dealing with source code directly, and Gustavo's suggestion to just tag the version to the URL solves most pain points in an obvious and elegant fashion. 

The problem with go get, the way I see it, and it's not a huge problem yet, is that it doesn't allow me to just download pre-compiled, versioned packages I can just link against, ala Maven/Leiningen/NuGet and the rest of those systems. A public repo of pre-compiled, versioned Go packages would be awesome. It seems to work just fine for at least 2-3 programming communities out there. 

Again, I must be missing something key here. Either way, I'm happy(-ish) with Go get, for now

[Rob Pike]

I know what versioning is, it's "vendoring" that I don't recognize as a term.

-rob

[flat...@users.sourceforge.net]

On Wednesday, March 19, 2014 12:42:48 AM UTC+4, DV wrote:

[...]

> yet, is that it doesn't allow me to just download pre-compiled, versioned packages I can just link against, ala Maven/Leiningen/NuGet and the rest of those systems. A public repo of pre-compiled, versioned Go packages would be awesome. It seems to work just fine for at least 2-3 programming communities out there. 

This can't work for Go because it's a compiled language, in the sense it's compiled to the
machine code of the target H/W architecture, which is in contrast with languages
targeting a VM (.NET and Java) which are compiled to an intermediate bytecode
which is architecture-agnostic.

[...]

[Kelsey Hightower]

I use the term "vendoring" to describe checking in or shipping external source code that your project depends on.

[Artem Kovardin]

https://github.com/pote/gpm + https://github.com/pote/gvp

[Frank Schröder]

I'm fine with vendoring in code I need manually. Takes a couple of minutes, I'm in control, the code is stable, no mess.

I've spent way too much time tracking down transitive dependency issues via maven, scala versions and log libraries in the java world.

Try it first.

Frank

[Gustavo Niemeyer]

I don't know the exact origin of the term. First I heard about it was
back in the cvs days, to describe a branch that was taken from its
original repository and maintained locally, generally to track work
that didn't make much sense to upstream.

From the CVS book (http://goo.gl/GWBiuO):

"""
Sometimes a site will make local changes to a piece of software
received from an outside source. If the outside source does not
incorporate the local changes (and there might be many legitimate
reasons why it can't), the site has to maintain its changes in each
received upgrade of the software.

CVS can help with this task, via a feature known as vendor branches.
In fact, vendor branches are the explanation behind the puzzling
(until now) final two arguments to cvs import: the vendor tag and
release tag that I glossed over in An Overview of CVS.

"""


On Tue, Mar 18, 2014 at 5:44 PM, Rob Pike <r...@golang.org> wrote:
> I know what versioning is, it's "vendoring" that I don't recognize as a term.
>
> -rob
>

> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[DV]

Not sure what you mean here. 

I don't understand the difference between a "compiled" language like Go that compiles to the native instruction set directly and a "compiled" language that targets the CLR/JVM that is later JIT'ed. Both are "compiled" - just different compile targets. 

Furthermore, nothing in the Go spec tells me I couldn't write a Go code to JVM/CLR bytecode compiler and compile Go code that runs on top of the CLR/JVM. 

"go build" produces compiled output. Combine that output with an XML/JSON/whatever manifest file that lists - description/author/version/target/etc - zip them together into a ".gopkg" file - put it online, make it searchable, installable, etc.....done? 

It almost sounds too simple, hence I'm thinking I must be missing something else. 

[Aram Hăvărneanu]

You just assume everyone wants to have the VM/JIT compiler installed
and you just assume that relying on complex software won't create
deployment issues, or other systemic issues that appear only in large
environments.

--
Aram Hăvărneanu

[Péter Szilágyi]

Hi,

  I don't really want to get into this debate as I don't really feel qualified to do so, but I thought I'd share a big issue with your solution.

  Go in it's current state has no notion of "project file" or "dependency file" or "manifests" or any such things. Which is brilliant. The source code alone is enough to do everything from dependency management to recursive builds to etc. This is a very powerful feature as I only have to care about keeping my code working, and all other things are sorted out automatically. The moment you add any single file beside code, you start a spiral of tooling issues (current tools only care about code).

  C++ dependency management is a library hell, Java dependency management is a configuration hell, Go currently works perfectly without *any* configuration. Why oh why ruin it? There are shortcoming, yes, but there are also seamless solutions being worked out (see Gustavo's gopkg.in). Imho, simplicity and clarity will *always* remain better than adding all possible bells and whistles.

  Btw, current go packages are

• Online: github, bitbucket, google code, etc.
• Searchable: either at individual sites or through godoc
• Installable: go get / go install

  Where's the issue?

Cheers,

  Peter

--

[DV]

I think in principle I agree with you, things work the way they are, but I can definitely see two potential issues happening. 

Issue 1:

when, in your project, you have:

import "github.com/woo"

what does that really mean? That, at the time you originally wrote your code, "woo" was in a certain state that worked for you? But how do you guarantee that when I import your code, which will cause me to import "woo", that it will still work? Maybe your code works with "woo" as it was on 1/22/2013, but I need "woo" as it is in 2/12/2014, and lo and behold - there's an API incompatibility - how do we resolve this? 

I like Gustavo's idea of package authors making the URL be something like "github.com/woo/v1" and then "github.com/woo/v2" or whatever as the need arises, but that *does* imply a certain discipline is required from library authors. Which leads to:

Issue 2:

Is it a good idea for me to pull in your latest HEAD? What state is your source tree in now? Is it stable? Bleeding edge, experimental, full of bugs? Beta? Alpha? Did the API break? I don't know. All I can do is "import github.com/foo/bar" and cross my fingers and hope for the best. 

So what I probably need to do is "save" a snapshot of your tree that I know works on my local machine and then import that. And don't freshen that from your bleeding edge unless needed. Which again, is a manual process, required discipline, etc...I can definitely see some way to make it better, but not without adding complexity. 

Again, so far, most libraries out there seem to not have these issues, but you have to admit that these issues could happen. 

[Péter Szilágyi]

I agree with the concerns you raised, but I think the solution needs to be solved at library developer side, not client side. Meaning that if I develop a popular library, rest assured that I will go out of my way to make it stable, as any API breakage will drive my users away. And the same holds from a client standpoint: if I use a library, and it breaks it's own API, rest assured that I will bin it at the first opportunity.

The goal ihmo is not to make a fool proof system, but one which backs responsible programmers whilst standing as much out of the way as possible. I.e: Make it trivial to use correctly, not hard to abuse :)

If you destabilize your own library, you kill your own library. Natural selection. Call it Darwin's law of library survival :D

[Kelsey Hightower]

On Wednesday, March 19, 2014 8:21:27 AM UTC-7, DV wrote:

I think in principle I agree with you, things work the way they are, but I can definitely see two potential issues happening. 

Issue 1:

when, in your project, you have:

import "github.com/woo"

what does that really mean? That, at the time you originally wrote your code, "woo" was in a certain state that worked for you? But how do you guarantee that when I import your code, which will cause me to import "woo", that it will still work? Maybe your code works with "woo" as it was on 1/22/2013, but I need "woo" as it is in 2/12/2014, and lo and behold - there's an API incompatibility - how do we resolve this? 

I like Gustavo's idea of package authors making the URL be something like "github.com/woo/v1" and then "github.com/woo/v2" or whatever as the need arises, but that *does* imply a certain discipline is required from library authors. Which leads to:

Issue 2:

Is it a good idea for me to pull in your latest HEAD? What state is your source tree in now? Is it stable? Bleeding edge, experimental, full of bugs? Beta? Alpha? Did the API break? I don't know. All I can do is "import github.com/foo/bar" and cross my fingers and hope for the best. 

So what I probably need to do is "save" a snapshot of your tree that I know works on my local machine and then import that. And don't freshen that from your bleeding edge unless needed. Which again, is a manual process, required discipline, etc...I can definitely see some way to make it better, but not without adding complexity. 

This is what I was referring to when I say "vendoring". Tools like Godep allow me to do exactly this, but in a semi-automated way. I can call the godep save command to take a snapshot of all my external dependancies with the option to check in those dependancies (source files) into a subdirectory in my project. Future updates of external dependancies are up to me, this is the manual part. The process is pretty standard. I update packages in my GOPATH using go get -u $package, make sure go build works and all tests pass, then I call godep save to update my "vendored" tree. The drawback being I need to use godep build instead of go build going forward.

While this approach does not rely solely on the native tooling (go get), it does place nicely:

    GOPATH=`godep path`:$GOPATH go build .

[DV]

After some more thought, I agree. "Github Darwinism" all the way! 

I guess unless it becomes endemic, where libraries (even popular ones) start breaking left-and-right, and start damaging the entire Go eco-system and development experience, there's no need to come up with any "solutions". 

[Henrik Johansson]

I am slowly but surely adapting to the idea of taking backwards compatibility as far as it can go.

When I get to the point where backwards compatibility cannot be maintained without immense energy I will try to see it as a new thing entirely and then it might not hurt so much (quite the opposite in fact) to create a new repo with the new breaking stuff.

[Nate Finch]

Note that with gopkg.in, you don't need a new repo, but can just make a new v2 branch from the same repo.  

so, gopkg.in/me/v2/foobar   will redirect to github.com/me/foobar (using the branch named v2)

This is a really great way to handle versioning, so you don't have to make a new repo with a new name, you just make a branch.  And, you can make the v2 branch the default one on github when it's ready, etc. 

There's talk of adding tooling support so that you can warn people if they import your code without using the redirector, too.

I think this will be the wave of the future with go, with everyone versioning their packages through redirectors like gopkg.in.   It's not a perfect solution, there certainly can be breakages that happen even without changing the external API, but it should be a lot less likely. 

[Lars Seipel]

On Wed, Mar 19, 2014 at 07:44:33AM +1100, Rob Pike wrote:
> I know what versioning is, it's "vendoring" that I don't recognize as a term.

You do know vendoring machines though, don't you?

Lars

[Konstantin Khomoutov]

On Wed, 19 Mar 2014 06:25:06 -0700 (PDT)
DV <dimiter....@gmail.com> wrote:

> > > yet, is that it doesn't allow me to just download pre-compiled,
> > versioned packages I can just link against, ala
> > Maven/Leiningen/NuGet and the rest of those systems. A public repo
> > of pre-compiled, versioned Go packages would be awesome. It seems
> > to work just fine for at least 2-3 programming communities out
> > there.
> >
> > This can't work for Go because it's a compiled language, in the
> > sense it's compiled to the
> > machine code of the target H/W architecture, which is in contrast
> > with languages
> > targeting a VM (.NET and Java) which are compiled to an
> > intermediate bytecode
> > which is architecture-agnostic.

> Not sure what you mean here.
>
> I don't understand the difference between a "compiled" language like
> Go that compiles to the native instruction set directly and a
> "compiled" language that targets the CLR/JVM that is later JIT'ed.
> Both are "compiled"

> - just different compile *targets*.

This is correct. The problem that while with CLR and JVM there is the
single compile target (for each class of VMs, that is), for Go there are
several targets, and this number is potentially unbounded -- see below.

> Furthermore, nothing in the Go spec tells me I couldn't write a Go
> code to JVM/CLR bytecode compiler and compile Go code that runs on
> top of the CLR/JVM.

This is correct but is largely orthogonal to the problem I've tried to
point out.

> "go build" produces compiled output. Combine that output with an
> XML/JSON/whatever manifest file that lists -
> description/author/version/target/etc - zip them together into a
> ".gopkg" file - put it online, make it searchable, installable,
> etc.....done?

Yes, but not terribly useful -- again, please see below for
explanations.

> It almost sounds too simple, hence I'm thinking I must be missing
> something else.

What you're missing is that when you compile to bytecode the produced
result will be able to run on a compliant VM on any target platform
(well, in the real world this may not be 100% true as projects often
use platform-specific libraries, say, which work only with Windows, but
I digress). In contrast to this, Go compiles down to machine code and
system calls of a specific platform -- which is a combination of the
hardware platform (say, amd64) and the OS kernel running on it, so that
we can talk about amd64/windows build vs i386/linux build vs
armhf/netbsd build or whatever. The binary targetting a speficic
platform typically has no chances to run on another. That's because
machine instructions differ, system calls and ABIs differ, executable
module formats differ and so on. The only exception I'm aware of is
that binaries for i386/$GOOS will run on amd64/$GOOS (but not
vice-versa) because the latter has been specifically designed to be
backwards-compatible with the former.

To look at it from another angle, a VM performs the so-called
"Just-In-Time" (JIT) compilation of its bytecode to the machine code
while Go's compiler (at least the compiler from the GC suite and
the GCC backend) perform the so-called "Ahead-Of-Time" (AOT)
compilation and hence produce the machine code right away.
This actually presents the well-known JIT vs AOT tradeoffs: running the
code on a VM requires the presence of the compiler at run time, has
potentially huge process startup cost but also has a possibility for the
hot-spot optimization, while AOT does not require any compiler present
at run time, has zero startup cost but hot-spot optimization isn't
possible (at least for AOT-compiled languages I'm aware of).

There are only two ways of producing a binary for a specific
$GOARCH/$GOOS combo: build it on that platform or cross-compile it for
that platform on another platform.

This means that to deploy compiled Go code into some repository you're
advocating, the author of that code has these choices:
1. Upload the compiled code only for the author's native platform.
Users of other platforms are required to fetch the source and
compile it theirselves. Possibly they might upload the results
of their compilation as well. This would require some sort of
"blessing" from the original author (who is supposedly somehow
trusted).
2. (1) plus the compiled code for whatever platform the author
managed to cross-compile their project.
3. Have a farm of autobuilders in the repo which would (also) accept a
source package and build it for all the supported platforms --
incidentially, that's what binary-oriented GNU/Linux distributions
are routinely doing, and you may beleive me that this is *hard*.

TL;DR
Having a repository like the one you're proposing is hard. It's
simpler than having it for C but still hard. The only working solution
I'm aware of is an already existing binary-centric GNU/Linux
distribution or another (free) OS.

[Konstantin Khomoutov]

On Tue, 18 Mar 2014 11:33:27 -0700 (PDT)
Gerard <gvds...@gmail.com> wrote:

> The problem of delivering an application with zero dependencies is
> easy solved with just embedding the specific dependencies. e.g. in
> "app/external/lib".
>
> Still no need for reinventing the wheel (IMO).
>
> But talking to a Debian maintainer: Are you aware of Go packages
> being introduced in Debian? I mean on a serious scale?

I'm not sure how to measure.
You can see already packaged stuff (in unstable) here [1] and the ITP
("intend to package") stuff here [2] (not filterable -- just look for
packages whose names start with "golang-" or "go-".

Note that packages are not somehow introduced into a community-driven OS
by themselves -- people who want/need stuff in the OS package it
themselves. Hence if no one steps forward, nothing is packaged.
It's DIY for the most part.

1. https://packages.debian.org/search?keywords=golang&searchon=sourcenames&suite=unstable&section=all
2. https://www.debian.org/devel/wnpp/being_packaged

[DV]

Right, I understand how JIT compilers work. My example was just to show that just because something is "pre-compiled" is not in itself an insurmountable hurdle or a unique feature to Go that precludes the existence of binary package management. 

So, how about this:

Foo.gopkg. Contains:

1. darwin_amd64/foo/bar/gopkg.a

2. darwin_386/foo/bar/gopkg.a

3. linux_386/foo/bar/gopkg.a

manifest xml:

<Name>FooBar</Name>

<Version>1.0.0</Version>

<GORUNTIME>1.2.1</GORUNTIME>

<Description>...</Description>

<Author>...</Author>

Heck, since it's so trivial to cross-compile in Go, this imagined repo could contain both a "super" package and individual "GOOS_GOARCH" package permutations. 

Like you said, this repo may be difficult, but it's been done multiple times, unless my FreeBSD experience with pkginstall from 5-6 years ago was just a dream :-)

Then again, it's so trivial *and* fast to compile on Go, unlike C/C++, that compiling from source is not a burden at all, so why even bother with a binary distribution of anything? 

I guess the only outstanding issue for me is that when I create a package that has "import github.com/foo/bar" and I compile it today and you download and build *my* package tomorrow, we end up with potentially radically different executables. And there's nothing I can do about it. 

[Gerard]

Thanks for the answer. Well, let me start than with the statistics package that I rewrote in Go some time ago ;-)

[Benjamin Measures]

On Tuesday, 18 March 2014 20:42:48 UTC, DV wrote:

packages which have versions on them and you can say that package A needs version 2.1 (or greater, for example) of package B, and *exactly* version 1.1 of package C, or any combination thereof. And it's all resolved automatically and recursively.

Unfortunately, version 2.6 of package B now requires version 1.4 of package C. And it turns out that package A would actually work with version 1.4 of package C but is disallowed for what is now no good reason. 

Things have broken and you're no better off (worse, in fact) than if you were without [versioned dependency resolution].

Conflicts can arise, yes, but it's rare

It's exactly as "rare" as in the case without, only this time you've random requirements on old versions (which may or may not be necessary).

Automatic versioned dependency resolution is no substitute for vendoring, and once you have the latter the former is entirely unnecessary.

[Sebastien Douche]

On Tue, Mar 18, 2014 at 4:06 PM, Kevin P <dco...@gmail.com> wrote:
> Go is getting to a point that it really needs official package manager. I
> think gopm https://github.com/gpmgo/gopm has the right idea and hopefully
> can be built into the go command.

Cargo, the next official Rust package manager:
https://mail.mozilla.org/pipermail/rust-dev/2014-March/009087.html

If it's work for Rust, it's a good candidate for Go :)



--
Sebastien Douche <sdo...@gmail.com>
Twitter: @sdouche / G+: +sdouche

[Dave Cheney]

I'm eager to see what wycats comes up with, but one email does not a package management ecosystem make. 

[Sebastien Douche]

On Fri, Mar 21, 2014 at 4:04 AM, Dave Cheney <da...@cheney.net> wrote:
> I'm eager to see what wycats comes up with

Me too

> but one email does not a package management ecosystem make.

Right but Cargo will be the "official" tool (used by all Rust coders).
It's a big difference with godep, gogpm... It will enhance the
attractiveness of Rust.

[Gustavo Niemeyer]

On Fri, Mar 21, 2014 at 12:47 AM, Sebastien Douche <sdo...@gmail.com> wrote:
> Right but Cargo will be the "official" tool (used by all Rust coders).
> It's a big difference with godep, gogpm... It will enhance the
> attractiveness of Rust.

"go" is the official tool, and most developers use it. godep is built
on top of it.


gustavo @ http://niemeyer.net

[Alex Skinner]

I think this depends on how you view it.  As far as dependency management goes, "go" is a rather "dumb" tool in that it doesn't manage anything, it just grabs what you ask it to.  There are a lot of directions to go from here, and I have no opinion on which is the right one.  On one hand, I really like the idea of gopkg.in in the way that it works, and think this may be the right path.  If so, "go" is adequate.  On the other hand, in 3 years, if github isn't around anymore, we're stuck with tons of code that just doesn't work anymore.  

I guess in a perfect world we'd have a site that handled versioning, the repo, etc ( a la pypi, as bad as python package management can be).  Or better, something that by default redirected as before, but if the target didn't exist, used its own repo perhaps?

Good package management is hard.  Forking your own repo is a good solution for now, but doesn't seem like a permanent or reliable solution.

Ultimately, do you feel it would be better if we had some type of moderated, controlled central repo?

[Dave Cheney]

Ultimately, do you feel it would be better if we had some type of moderated, controlled central repo?

No. I cite as my example npm and the unfolding drama around the ownership and reliability of their central repo.

[Alex Skinner]

I can see your point.  Are you of the opinion that go provides the necessary tools, and package management should be ultimately up to the user?  

I ask only because I think package management, for the most part, should be something agreed upon by the community.  If most feel that what we have is fine, and that there are good enough ways for users to use it, so be it.  I don't disagree with it, I only feel that the community(including myself) should help guide people in this direction.  What's worse than any current schemes are a hundred different proprietary implementations, when none were really need all along.

So if what is provided is generally good enough for the users, let's simply guide people into this direction and how to best use it, similar to the post earlier about forking your own copies.

Thanks,

Alex 

[Dave Cheney]

I can't speak for a community, only for myself. I wrote about my thoughts on this last year, http://dave.cheney.net/2013/10/10/why-i-think-go-package-management-is-important.

Sadly, I was not able to contribute to the go-pm group as much as I wanted to for various reasons that have nothing to do with writing code.

--

You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/PLTY792AVzc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.

[Gerard]

This is an endless discussion. It's all about control. You don't control 3rd party libraries. That's how it is.

So, how to get control?

1) trusting a maintained environment.

1.1) Such as CPAN. But creating a community, a policy, a proper website etc. is hard. I can see dozens of bears. And legal stuff too.

1.2) FOSS Operating System package management. I think that this is the way to go. And because Go doesn't have shared libs, you only have the "_dev" packages which contain only source code. Of course this doesn't work on Windows, but in Windows you are screwed anyhow because you don't control anything.

2) control it yourself

2.1) Use tarballs. Not fancy, but it works!

2.2) Embed the third party packages. E.g. "yourapp/external/3rdparty_package"

[Taru Karttunen]

On 20.03 08:57, DV wrote:
> Right, I understand how JIT compilers work. My example was just to show
> that just because something is "pre-compiled" is not in itself an
> insurmountable hurdle or a unique feature to Go that precludes the
> existence of binary package management.
>
> So, how about this:
>
> Foo.gopkg. Contains:
> 1. darwin_amd64/foo/bar/gopkg.a
> 2. darwin_386/foo/bar/gopkg.a
> 3. linux_386/foo/bar/gopkg.a

6 * Windows (386, 386+387, amd64), cgo on, cgo off
6 * Darwin (386, 386+387, amd64), cgo on, cgo off
12 * Linux (386, 386+387, amd64, arm5, arm6, arm7), cgo on cgo off
...

- Taru Karttunen

[Jan Mercl]

On Tue, Mar 18, 2014 at 9:05 PM, Dan Kortschak
<dan.ko...@adelaide.edu.au> wrote:
> I think the issue here is that Jan is including a requirement for
> 'automatic' in the system, in which case it is the case that it's not
> solvable.

'go get' is the 'automatic' part - it walks/gets the dependency graph
automatically. That's why using a version number in the import path
doesn't and cannot work.

-j

[Dan Kortschak]

identification of annotated conflicts is solvable. correction of those conflicts automatically is not.

[Nicolas Grilly]

On Friday, March 21, 2014 1:52:14 AM UTC+1, Sébastien Douche wrote:

Cargo, the next official Rust package manager:
https://mail.mozilla.org/pipermail/rust-dev/2014-March/009087.html

If it's work for Rust, it's a good candidate for Go :) 

According to the Rust mailing list, it looks like Cargo's authors are leaning toward a centralized repository solution. According to my experience with pip, Bundler and npm, it is a bad idea.

I'm really happy with the approach promoted by Go with no centralized repository. I'm perfectly okay with vendoring to freeze dependencies versions and have reproducible builds. The only issue I have with this approach is the little burden of having to rewrite import paths.

But if I had to choose between two pleas, having a centralized repository or having to rewrite import paths, I would choose the latter always.

[This is reposting of the same message on go-package-management]

[Henrik Johansson]

Can't Bundler fetch from github? I think it can and the plans for Cargo also includes such functionality.

--

You received this message because you are subscribed to the Google Groups "golang-nuts" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

[Jan Mercl]

On Fri, Mar 21, 2014 at 10:46 AM, Dan Kortschak
<dan.ko...@adelaide.edu.au> wrote:
> identification of annotated conflicts is solvable. correction of those conflicts automatically is not.

The conflict is enabled as soon as different versions of the same
package get involved in the same project. So the only guaranteed
non-conflicting case is when no differing versions are imported. But
that's the current state of art - without using version numbers.

Let me emphasize "enabled". The detection of the conflict actually
present is, again, non computable. So version numbers (or any other
scheme with the same effect, including a renamed fork, for example) do
not solve the problem. They enable the problem.

-j

[Nicolas Grilly]

On Tuesday, March 18, 2014 9:44:33 PM UTC+1, Rob 'Commander' Pike wrote:

I know what versioning is, it's "vendoring" that I don't recognize as a term.

It looks like "vendoring" is a neologism derived from the concept of "vendor branches" popularized by Subversion:

http://svnbook.red-bean.com/en/1.7/svn.advanced.vendorbr.html

[Jan Mercl]

On Thu, Mar 20, 2014 at 10:59 PM, Benjamin Measures
<saint....@gmail.com> wrote:
> On Tuesday, 18 March 2014 20:42:48 UTC, DV wrote:
>>
>> packages which have versions on them and you can say that package A needs
>> version 2.1 (or greater, for example) of package B, and *exactly* version
>> 1.1 of package C, or any combination thereof. And it's all resolved
>> automatically and recursively.
>
>
> Unfortunately, version 2.6 of package B now requires version 1.4 of package
> C. And it turns out that package A would actually work with version 1.4 of
> package C but is disallowed for what is now no good reason.
>
> Things have broken and you're no better off (worse, in fact) than if you
> were without [versioned dependency resolution].

This is math done right! And yes, encouraging people to use a broken
solution, which only makes things worse for them is just wrong.

>> Conflicts can arise, yes, but it's rare
>
>
> It's exactly as "rare" as in the case without, only this time you've random
> requirements on old versions (which may or may not be necessary).

They arise easily, discussed, including the trivial "rare" counter
examples, here:
https://groups.google.com/d/topic/golang-nuts/sfshThQ_wrA/discussion

> Automatic versioned dependency resolution is no substitute for vendoring,
> and once you have the latter the former is entirely unnecessary.

Of course. Just wondering why _facts_ have to be discussed again and
again and ... ;-)

-j

[Nicolas Grilly]

On Friday, March 21, 2014 10:55:43 AM UTC+1, Henrik Johansson wrote:

Can't Bundler fetch from github? I think it can and the plans for Cargo also includes such functionality.

That's correct: Bundler can fetch from github. And according to Rust mailing list, Cargo will be able to fetch from github too. But in my experience, the centralized repository is still used in most cases, and github is only used for repositories directly managed by the project team. The centralized repository creates a lot of issues: someone has to manage it, how to manage naming conflicts, security issues, availability issues, etc.

[Henrik Johansson]

What a about a p2p based solution?

Bitdeps here we go! :)

--

[Dan Kortschak]

On 21/03/2014, at 8:27 PM, "Jan Mercl" <0xj...@gmail.com> wrote:

> The conflict is enabled as soon as different versions of the same
> package get involved in the same project.

Not necessarily, but it simplifies things to assume that it does.

> So the only guaranteed
> non-conflicting case is when no differing versions are imported. But
> that's the current state of art - without using version numbers.
>
> Let me emphasize "enabled". The detection of the conflict actually
> present is, again, non computable. So version numbers (or any other
> scheme with the same effect, including a renamed fork, for example) do
> not solve the problem. They enable the problem.

I don't believe that's true. You can do a simple traversal of the dep graph annotating nodes with dependency requirements and looking for inconsistencies in the resulting node annotations. This finds problems, but will not fix them.

[Jan Mercl]

On Fri, Mar 21, 2014 at 11:22 AM, Dan Kortschak
<dan.ko...@adelaide.edu.au> wrote:
> I don't believe that's true. You can do a simple traversal of the dep graph annotating nodes with dependency requirements and looking for inconsistencies in the resulting node annotations. This finds problems, but will not fix them.

The space in which the conflicting states exist is not finite. If it
would be only that the versions cannot compete for the same named
resource (as an example) then it would be a finite space and it would
be computable. Unfortunately, that's not the case. Only intelligent
systems, aka humans, can detect some of the conflicts. End even humans
can and will fail at it regularly for non trivial projects (lot of
packages, long evolution history).

Better formulated here:
https://groups.google.com/d/msg/golang-nuts/sfshThQ_wrA/6QVvQ5GlctEJ

-j

[Dan Kortschak]

Yes, absolutely. That's why it's important to make the assumption (as I did) that any conflicting version is going to cause an actual conflict.

I'm curious though how you justify the claim that the state space is not finite.

[Jan Mercl]

On Fri, Mar 21, 2014 at 11:46 AM, Dan Kortschak
<dan.ko...@adelaide.edu.au> wrote:
> Yes, absolutely. That's why it's important to make the assumption (as I did) that any conflicting version is going to cause an actual conflict.
>
> I'm curious though how you justify the claim that the state space is not finite.

Sufficient is to consider that it necessarily includes the time domain.

-j

[Ian Lance Taylor]

On Fri, Mar 21, 2014 at 3:03 AM, Nicolas Grilly
<nic...@vocationcity.com> wrote:
> On Tuesday, March 18, 2014 9:44:33 PM UTC+1, Rob 'Commander' Pike wrote:
>>
>> I know what versioning is, it's "vendoring" that I don't recognize as a
>> term.
>
>
> It looks like "vendoring" is a neologism derived from the concept of "vendor
> branches" popularized by Subversion:

Minor trivia note: Subversion took the idea of vendor branches from
CVS, which had them long before.

Ian

[Gustavo Niemeyer]

On Fri, Mar 21, 2014 at 7:04 AM, Jan Mercl <0xj...@gmail.com> wrote:
> They arise easily, discussed, including the trivial "rare" counter
> examples, here:
> https://groups.google.com/d/topic/golang-nuts/sfshThQ_wrA/discussion

A wise guy said in that thread: "I am not convinced the dependency
problem can be solved. I believe that in general it is intractable,
although in practice the pathological cases might not arise often."

Even airplanes with hundreds of people on board do not fly with a
mathematical guarantee of arrival.

> Of course. Just wondering why _facts_ have to be discussed again and
> again and ... ;-)

People's code commonly break when dependencies do not follow any kind
of API stability strategy. This is a fact with years of backing
evidence.

The labix.org/v2/mgo convention works, and developers like the fact
that it won't break tomorrow. Another fact with years of backing
evidence.

The reason this specific discussion continues is pedantry.


gustavo @ http://niemeyer.net

[Nicolas Grilly]

On Fri, Mar 21, 2014 at 2:52 PM, Ian Lance Taylor <ia...@golang.org> wrote:

Minor trivia note: Subversion took the idea of vendor branches from
CVS, which had them long before.

I didn't know that. Everybody loves trivia :)

[Frank Schröder]

People's code commonly break when dependencies do not follow any kind

of API stability strategy. This is a fact with years of backing
evidence.

Why shouldn't the API be allowed to change even within the same version? How often is this going to happen once the driver or lib has stabilized?

I can see the benefit of enforced backwards compatibility on the Go platform itself but not necessarily for the libs out there. If some API is broken or no longer suitable then it should be changed. Libraries with better desgin will have less amounts of changes and will be favored by the community which will drive their adoption.

With the /version approach you somewhat lock in the library maintainer because it sets the expectation that your code will not break Period.

As a lib maintainer what I'd prefer to say is "I will not break your code unless I have to and then I'm going to explain why and how you can fix it". Then you don't need the /version approach anymore.

Frank

[Gustavo Niemeyer]

On Fri, Mar 21, 2014 at 11:56 AM, Frank Schröder
<frank.s...@gmail.com> wrote:
>> People's code commonly break when dependencies do not follow any kind
>> of API stability strategy. This is a fact with years of backing
>> evidence.
>
> Why shouldn't the API be allowed to change even within the same version? How
> often is this going to happen once the driver or lib has stabilized?

Because it breaks everything that depends on it. It's that simple. How
relevant that breakage is will be directly proportional to how
relevant your package is.

> I can see the benefit of enforced backwards compatibility on the Go platform
> itself but not necessarily for the libs out there. If some API is broken or
> no longer suitable then it should be changed. Libraries with better desgin
> will have less amounts of changes and will be favored by the community which
> will drive their adoption.

This assumes the suggestion is to never change your API, which is not
the case. The suggestion is to offer versioned stable APIs.

> With the /version approach you somewhat lock in the library maintainer
> because it sets the expectation that your code will not break Period.

It's the opposite: it gives you the freedom to change your API
whenever you have to, without worrying about breaking people that
depend on the current version of the API.

> As a lib maintainer what I'd prefer to say is "I will not break your code
> unless I have to and then I'm going to explain why and how you can fix it".
> Then you don't need the /version approach anymore.

You should do that even if you do offer versioned stable APIs, but
doing so doesn't change the fact this will break existent code that
depended on the current version of your API.

Please ponder about this for a moment.. imagine there are hundreds of
packages depending on a given package out here. The moment you push a
breaking change, all of those packages are suddenly broken. It doesn't
matter if you explain why you're breaking, and how great your
intention is. That body of working code is now broken for everybody,
until people go there and fix it.

If a package is very young, or used by two people.. then sure, who cares.


gustavo @ http://niemeyer.net

[Andy Balholm]

On Friday, March 21, 2014 1:19:57 AM UTC-7, Taru Karttunen wrote:

6 * Darwin (386, 386+387, amd64), cgo on, cgo off

Let me pick a little nit here: I don't think there are any Darwin/386 systems running on CPUs too old to support SSE. SSE was introduced on the Pentium III, Darwin/386 was developed on Pentium 4 and released on Core Solo and Core Duo.

But your point still stands that there are a lot of GOOS/GOARCH combinations to support.

[Chris Hines]

I did some research on the origin of the term when the question came up in the Go Package Management group. Here is a link to my post on the topic: https://groups.google.com/d/msg/go-package-management/bTWi4KanW8M/ZFVUqNcqkE0J

[Nate Finch]

On Friday, March 21, 2014 10:56:51 AM UTC-4, Frank Schröder wrote:

With the /version approach you somewhat lock in the library maintainer because it sets the expectation that your code will not break Period.

Yes.  Exactly. That's a good thing.  If I'm using your library, I don't want it to break, period.  Possibly if there is a critical security patch that absolutely must be applied, and no one should ever ever ever run the non-patched version *then* you might be able to break the old API.  But otherwise? No.  Don't break stuff.

 

As a lib maintainer what I'd prefer to say is "I will not break your code unless I have to and then I'm going to explain why and how you can fix it". Then you don't need the /version approach anymore.

Explain to whom?  You don't know who's using your code.   And most of the time the people who are using your code, aren't using "your" code. They're using someone else's code that happens to reference your code.  So all those people who try to go get that other package, now have broken code, and complain to that other organization... and that other organization has to scramble to figure out what broke and how, and how they can update their code so that it still works....   That's totally not workable. This is why other languages use libraries that have versions that don't just change out from under people.  You may say "hey, v2.1 is out and fixes some security vulnerabilities", but you don't change 2.0 out from under people, especially not anything that you *know* will break people's code.

[Gustavo Niemeyer]

CVS 1.0 from Brian Berliner, from 1989 [1], already had the term "vendor
branch" in it. The CVS shell scripts dating back from 1986, published
by Dick Grune [2], did not mention the term. So it was Brian or someone
around him that brought it up. I don't know if that's the *origin* of
the term, though. It may well date back further.

[1] http://ftp.isc.org/usenet/comp.sources.unix/volume22/cvs-berliner/
[2] http://ftp.isc.org/usenet/comp.sources.unix/volume6/cvs/

> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--

gustavo @ http://niemeyer.net

[Frank Schröder]

If you don't want my lib to break then copy the version you're happy with and forget about it. If it needs other libs then copy them, too. When you need to upgrade it then get a new copy. This way you know why you're getting it and what has changed.

I think library maintainers will out of pride and self-defense try to design their libraries in a way that don't break other peoples code since they want them to use their code, want to keep the number of questions low and the praise high. Call it evolution.

Dependency management is difficult as soon as you're building non-trivial applications and I don't see how versioning (git tags, /version, central repo, smarter tools) solve any of the transitive dependency problems you're describing. You're just moving the problem and give the developer the false illusion that the package management will solve the problems for him.

If you use code via github urls then you get random versions since you depend on my tip/branch. How can you make any guarantees of your functionality? Even if the API is stable the behavior may still have changed.

With Go pulling everything from tip/branch that is what happens. There is zero control over what you actually get at any given time and the way I understand it this is intentional.

That's why IMHO vendoring is a saner approach to this problem. It makes it explicit what you import and you decide when to upgrade a certain library.

Frank
--
Frank Schröder
Amsterdam, NL
PGP - DDA53977

> --
> You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/PLTY792AVzc/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.

[Henrik Johansson]

This is precisely why versioning is better than vendoring. Only the library maintainer has a reasonable chance to decide if some state of the code is stable.

With vendoring not only I but you and everyone else using a library have to test and verify the stability of all their dependencies and the dependencies dependencies...

[Frank Schröder]

So you use A which uses C1 and B which uses C2. Both authors of A and B have tested their lib with their respective version of Cx. What does versioning solve here? Which version do you choose? Does A work with C2 and B with C1? How do you know?

Try Java + maven and importing spring or commons-* via transitive dependencies and you get 3.0.x and 3.1.x packages which are somewhat incompatible but sorta-kinda work with each other - until they don't. You can "solve" this by pinning the versions of all packages you're importing.

You don't get the "Get out of jail free" card with a package manager.

Frank
--
Frank Schröder
Amsterdam, NL
PGP - DDA53977

[Henrik Johansson]

I work with maven and spring daily and if you get different versions of dependencies then you are doing something wrong.

Let me put it this way: Try to vendor a spring project...

[Frank Schröder]

Yes, you use the <dependencyManagement> section and pin the versions of transitive dependencies. This way you make sure that you get the same versions. But these are not the versions the authors of the library I'm using have used when they tested and packaged their library.

In fact I am trying to vendor in spring as we speak since I want to try to build our Java projects the way I'm building Go projects.

Any help is greatly appreciated - but better offline. This is golang-nuts after all :)

[yy]

On 21 March 2014 21:21, Frank Schröder <frank.s...@gmail.com> wrote:
> So you use A which uses C1 and B which uses C2. Both authors of A and B have tested their lib with their respective version of Cx. What does versioning solve here? Which version do you choose? Does A work with C2 and B with C1? How do you know?

I don't think anybody has claimed to have solved this problem (neither
"vendoring" will). However, a version system might at least detect the
error.

Back to your example:

- If A and B import C from github: go get clones the repositories A, B
and C and you cannot compile A or B or your package, or it fails
(maybe silently) because A, B or both are using a package C which does
not work as expected. The problem is not only not solved, but also
hard to diagnose.

- If A and B import C1 and C2 from gopkg.in: when your package doesn't
compile or doesn't work you can easily see that A and B are importing
different versions of C. You can even detect this conflict very easily
in some automatic way, before or as part of the build process of your
package. What is so bad about this?

> You don't get the "Get out of jail free" card with a package manager.

There are many things that don't get me that card and yet I find useful.



--
- yiyus || JGL .

[Frank Schröder]

On 21 Mar 2014, at 22:43 , yy <yiyu...@gmail.com> wrote:

> On 21 March 2014 21:21, Frank Schröder <frank.s...@gmail.com> wrote:
>> So you use A which uses C1 and B which uses C2. Both authors of A and B have tested their lib with their respective version of Cx. What does versioning solve here? Which version do you choose? Does A work with C2 and B with C1? How do you know?
>
> I don't think anybody has claimed to have solved this problem (neither
> "vendoring" will). However, a version system might at least detect the
> error.

"vendoring" doesn't solve that problem since the developer has to
solve it. But "vendoring" puts the developer in control and doesn't
hide transitive imports as well as a packaging tool would.

>
> Back to your example:
>
> - If A and B import C from github: go get clones the repositories A, B
> and C and you cannot compile A or B or your package, or it fails
> (maybe silently) because A, B or both are using a package C which does
> not work as expected. The problem is not only not solved, but also
> hard to diagnose.
>
> - If A and B import C1 and C2 from gopkg.in: when your package doesn't
> compile or doesn't work you can easily see that A and B are importing
> different versions of C. You can even detect this conflict very easily
> in some automatic way, before or as part of the build process of your
> package. What is so bad about this?

You're assuming C1a and C1b to behave the same but C1a may have a bug
that maintainer A worked around but that doesn't work with C1b. He
only tested with C1a. Same with lib B which was tested only with C1b.

Yes, it mostly works if your projects aren't too complex.
If the number of dependencies and versions and transitive dependencies
reach a certain threshold then this becomes a problem.

With vendoring I'm in control of the entire source code that makes up
my program. Nothing shows up unexpectedly, nothing gets upgraded
unless I copy it into the source tree. Also, I need only one
checkout and not many.

The build machines get exactly the 3rd party code I've checked
in and not the version the author just pushed 3 min after I've committed.

If I refer to gopkg.in/v1.2/foo then this may resolve to v1.2.0,
v1.2.1, v1.2.5 depending on the time of day.

In small projects you may not care but if you want reproducible
builds then you cannot rely on some external source repository.

[Gustavo Niemeyer]

On Fri, Mar 21, 2014 at 7:14 PM, Frank Schröder
<frank.s...@gmail.com> wrote:
> With vendoring I'm in control of the entire source code that makes up

(...)

> If I refer to gopkg.in/v1.2/foo then this may resolve to v1.2.0,
> v1.2.1, v1.2.5 depending on the time of day.
>
> In small projects you may not care but if you want reproducible
> builds then you cannot rely on some external source repository.

Sure, and that's fine. gopkg.in is not meant to replace godep and
friends, but rather allow people to handle stable API versions more
easily. Whether you pick 1.2.1 or 1.2.5 or 1.3, the build should not
break unless the developer has screwed up something not intended.


gustavo @ http://niemeyer.net

[Sebastien Douche]

On Fri, Mar 21, 2014 at 1:52 AM, Sebastien Douche <sdo...@gmail.com> wrote:
> Cargo, the next official Rust package manager:
> https://mail.mozilla.org/pipermail/rust-dev/2014-March/009087.html

Cargo Status Update (week of March 17, 2014):
https://mail.mozilla.org/pipermail/rust-dev/2014-March/009118.html

--
Sebastien Douche <sdo...@gmail.com>
Twitter: @sdouche / G+: +sdouche

[afriz...@gmail.com]

On Wednesday, March 19, 2014 6:27:26 AM UTC+8, Artem Kovardin wrote:

https://github.com/pote/gpm + https://github.com/pote/gvp

Some descriptions of the problem and how to solve it using GPM & GVP

http://java.dzone.com/articles/why-gpm-right-go-package