Dependency package management idea

[Casey Marshall]

The kickstarter fiasco on DailyWTF that made the rounds last week, while pretty ridiculous and misinformed, got me to thinking about how dependencies *should* be managed. Could it be better automated?

Dependency package management has been on my mind for awhile. I deal with it in my own project, Hockeypuck (https://launchpad.net/hockeypuck), which uses an ad-hoc Makefile and builder scripts for freezing these into a Debian source package. Sort-of works, could be much better, not very reusable.

I put some slides together on the dream tool I'd like to use for managing dependencies, if it existed:

http://cmars.github.io/dpm/

Feedback most welcome.

Thanks,
Casey

[Paulo Pinto]

You also add the D and Rust's current solutions to your package
manager's list.

https://github.com/mozilla/rust/wiki/Rustpkg

http://registry.vibed.org/about (still quite new)

What I miss from Go is a story for managing dependencies among
commercial packages
without source code available.

With most package managers I am able to install binary libraries and
make them easily
available to any developer on the team, even if we need to set up a
private repository.

Go seems to require everyone to distribute source code still.

--
Paulo

[Jan Mercl]

On Sat, May 25, 2013 at 9:36 AM, Paulo Pinto <paulo....@gmail.com> wrote:
>
> Go seems to require everyone to distribute source code still.

Can you please point out where in the specs is it like that?

Please don't freely interchange Go and a specific tool chain.
Additionally distributing a package w/o it's sources in a way which
_is_ compatible with the existing tool chain is well known technique
(search the list please). Moreover, the tool chain is completely open
source (so you can adjust it to your needs) and completely free (as in
free beer).

Now, _this explicitly *is not* about you_, but some people on this
list show an attitude that open sourced and zero costing is not enough
and expect others to fulfil their personal/specific additional needs
just like it would be almost a right they magically acquired. And for
free, of course.

-j

[Paulo Pinto]

On 25 Mai, 11:34, Jan Mercl <0xj...@gmail.com> wrote:

> On Sat, May 25, 2013 at 9:36 AM, Paulo Pinto <paulo.jpi...@gmail.com> wrote:
>
> > Go seems to require everyone to distribute source code still.
>
> Can you please point out where in the specs is it like that?

It is not specified anywhere. I just read somewhere that the binary
packages
are still coupled to specific compiler versions, even minor versions.

>
> Please don't freely interchange Go and a specific tool chain.
> Additionally distributing a package w/o it's sources in a way which
> _is_ compatible with the existing tool chain is well known technique
> (search the list please). Moreover, the tool chain is completely open
> source (so you can adjust it to your needs) and completely free (as in
> free beer).

What I expect from an enterprise point of view, is to be able to
distribute Go
binary code the same way I was able to distribute Turbo Pascal and
Delphi
units as an example.

>
> Now, _this explicitly *is not* about you_, but some people on this
> list show an attitude that open sourced and zero costing is not enough
> and expect others to fulfil their personal/specific additional needs
> just like it would be almost a right they magically acquired. And for
> free, of course.
>
> -j

I do understand that no problem.

As I mentioned in another posts, for reasons which many of you know I
do tend to
spend more time in D nowadays and follow Rust as well.

So I don't have any special right to ask for anything regarding Go,
just keep coming
here because I like the community and I as someone with compiler
design background I tend
to keep an eye on programming languages. And although I am sometimes a
bit vocal, which
I probably should stop doing, the Oberon roots in Go make me find the
language attractive.

My remark was just a kind of heads up for what the enterprise world
expects, as it is the
way I work and I thought the input might be valuable.

You know these big corporations with outsourcing scattered around the
globe with development
teams of 60+ developers, with many third parties involved.

--
Paulo

[Jan Mercl]

On Sat, May 25, 2013 at 12:02 PM, Paulo Pinto <paulo....@gmail.com> wrote:
>
> It is not specified anywhere. I just read somewhere that the binary
> packages
> are still coupled to specific compiler versions, even minor versions.
>

>> ...

>
> What I expect from an enterprise point of view, is to be able to
> distribute Go
> binary code the same way I was able to distribute Turbo Pascal and
> Delphi
> units as an example.

You can and in the very same way Delphi does that. (Actually the
concepts are nearly exactly the same). To distribute a binary
(program), just ship the binary (program), both for Delphi and Go.

To distribute a package(unit), ship the respective .a(.dcu) for Go and
Delphi. _Both_ use versioning and both do not accept lower than
implementation specific limits numbers. In both cases the version
numbers are (I believe) changed iff the new format is no more
compatible with the older one (format, features, semantics, ...
changed).

Really, there's next to no difference in this between your example
language (Delphi) and Go.

-j

[Paulo Pinto]

Ah ok, thanks for the clarification.

On 25 Mai, 12:18, Jan Mercl <0xj...@gmail.com> wrote:

[Benoît Amiaux]

I believe shipping binary objects without sources does not work with either go 1.0 and go 1.1

See related issue here https://code.google.com/p/go/issues/detail?id=2775

I would be glad to be proven wrong though, but all my tests have failed under windows.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Jan Mercl]

On Sat, May 25, 2013 at 4:53 PM, Benoît Amiaux <benoit...@gmail.com> wrote:
> I believe shipping binary objects without sources does not work with either
> go 1.0 and go 1.1
> See related issue here https://code.google.com/p/go/issues/detail?id=2775
> I would be glad to be proven wrong though, but all my tests have failed
> under windows.

Considering package foo, having import path 'example.com/bar/foo':

$ #1 Create a fake foo's source:
$ echo "package foo" > $GOPATH/src/example.com/bar/foo.go
$ #2 make foo.a younger (!older) than its (fake) source
$ touch foo.a
$ #3 Install foo.a into its respective compiled package cache place:
$ mkdir -p $GOPATH/pkg/$GOOS_$GOARCH/example.com/bar/
$ cp foo.a $GOPATH/pkg/$GOOS_$GOARCH/example.com/bar/

Now you should be good to go. Ie. to use the go build system for
project importing 'example.com/bar/foo'. Modulo obvious no-nos as -a.

Not tested, never actually done by me. IIRC, this was discussed some
time before as a working solution. However, looking at #2775 which I
was not aware of before, there must be something wrong somewhere.

If you, by chance, are going to test the above steps, could you please
report back where it fails? TIA. I assume that the build system will
find foo.go precedes foo.a and thus just use it w/o compiling foo.go.
Would like to know where is this hypothesis failing and why. (?)

-j

[Rob Pike]

The go tool requires source, since the source contains the dependency
information it uses. But the compiler and linker do not need source
code for the packages a program depends on, so it is possible to
distribute Go libraries without source, at the cost of requiring
Makefiles or some such.

-rob

[Casey Marshall]

On Saturday, May 25, 2013 2:36:23 AM UTC-5, Paulo Pinto wrote:

You also add the D and Rust's current solutions to your package
manager's list.

https://github.com/mozilla/rust/wiki/Rustpkg

Rust sounds very interesting, but I've been waiting for it to become more stable. I'm a huge fan of Mozilla.
 

http://registry.vibed.org/about (still quite new)

What I miss from Go is a story for managing dependencies among
commercial packages
without source code available.

With most package managers I am able to install binary libraries and
make them easily
available to any developer on the team, even if we need to set up a
private repository.

Personally, I don't miss this for development or repeatable builds, but I can see the value in it. It would be useful for Debian packaging of Go packages, with dependencies managed by apt, for example.

So, if dpm provides a recipe for repeatable materializing and building of source code, one could create a new tool that builds and derives binary packages from dpm.yml and $GOPATH, for a binary distribution.

[John Asmuth]

https://github.com/skelterjohn/go-gb will do it. I stopped working on it a long time ago, but thanks to the go1 contract it still works (just tested to make sure).

If the .a file exists in the right place, gb won't demand the source exist as well. It will just be ignorant of any further deps. This ignorance can cause issues when the sourceless package imports something that is imported elsewhere, and the elsewhere is a newer version. I don't know exactly what would happen, but it would probably not work as intended if data was moved from one version to the other.

[suharik]

There is also a nut tool.

There is a problem that libraries can be renamed or moved, and a project won't build. But your solution doesn't require centralized repo and changing import paths, so you don't need to wait when libraries will support the dependency management tool.

Note: YAML is quite complicated, what about json, ljson or toml?

[Casey Marshall]

I used YAML for the example out of a recent familiarity with it elsewhere (Ansible), but I'm not terribly attached to it.

JSON is ok, but of the three, I really like TOML. Thanks for the suggestion!

-Casey