Go package/version management, vendoring, dep, gb, etc...

123 views
Skip to first unread message

jeffk...@gmail.com

unread,
Feb 4, 2017, 6:08:48 PM2/4/17
to Go Package Management, jeff....@jibeconsulting.com
I can’t seem to converge on a decent package/version management scheme. I asked Dave Cheney, and he recommended that I email this list.

I’ve written a Go project. I started with a directory tree like this:

$GOPATH/src/project/objects/agent_v1/agent_v1.go (import “project/objects/agent_v1”)
$GOPATH/src/project/objects/customer_v1/customer_v1.go (import “project/objects/customer_v1”)
Etc.

I know that I needed to be able to support different versions of my objects, so that is the directory structure I started using.

( I am a newbie at using VCS tools. Currently using GitHub, to clone a repository, add, commit, and push. No version numbers, releases, branches, forks, tags, etc., because I am a newbie at Git.)

Then, I started using one of the Go lint programs, and it complained about the underscores in my package names. That made me reconsider how I was versioning my packages. Also, how would I mature in my usage of GitHub as a VCS (with releases, branches, forks, tags, etc.). It also prompted me to ask “What is the Go way to deal with this?” There must be a recommended way. Little did I know, it seems like there are lots of options, with no clear winner.

Alternatives:
1) Change “agent_v1” to “agent-v1”.
Suggested by: http://zduck.com/2014/go-and-package-versioning/
Opposed by: https://dave.cheney.net/2014/12/01/five-suggestions-for-setting-up-a-go-project
2) Change “agent_v1” to “agentV1”. Ugly. The only thing that distinguishes the version number is the uppercase “V”.
Opposed: https://dave.cheney.net/2014/12/01/five-suggestions-for-setting-up-a-go-project
3) Using http://labix.org/gopkg.in.
I don’t like this because it forces the “go-pkg/pkg” naming scheme.
4) “dep”: looks like it will be the official tool, but not yet ready for prime time.
5) gb: Replaces some Go tooling. Will it eventually be replaced by “dep”?
6) Vendoring. I don’t understand it well enough to tell if it would help with versioning.
7) https://github.com/golang/go/wiki/PackageManagementTools
I’ve looked at most of these, and I don’t see a clear winner. Also, it is unclear if I adopt any one of them, how much work it will be to move to “dep” (once it becomes available).

I’d love to adopt SemVer 2.0.0, as Dave Cheney suggests (in https://dave.cheney.net/2016/06/24/gophers-please-tag-your-releases), but I can’t see how that will work with the Go tooling.

What I’m looking for: A solution that will:
a) make builds reproducible.
b) use SemVer 2.0.0 versioning.
c) use the full power of GitHub (releases, branches, forks, tags, etc.)
d) use GitHub tags as a versioning mechanism, instead of version names embedded in directory paths.
e) use the standard Go tooling as much as possible. Only use other tooling where the standard Go tooling won’t work for package/version management.
f) minimize work needed to adopt “dep”, or whatever package/version management the Go community eventually converges on and adopts.

It seems like this is an area under flux. I can’t see it converging. (Maybe that is my fault; I’m a newbie).

Dave's advice:

Use gb and semver, https://github.com/golang/go/issues/12302. If gb isn't your thin, I recommend github.com/dpw/vendetta.

Anyone else have a recommendation?

Jeff Kayser
Jibe Consulting | Managing Principal Consultant
5000 Meadows Rd. Suite 300
Lake Oswego, OR 97035
O: 503-517-3266 | C: 503.901.5021
jeff....@jibeconsulting.com

jeffk...@gmail.com

unread,
Feb 4, 2017, 7:35:10 PM2/4/17
to Go Package Management, jeff....@jibeconsulting.com, jeffk...@gmail.com
Would also be nice if it worked with Delve.
I'm using Delve inside Atom.

Peter Bourgon

unread,
Feb 4, 2017, 11:50:43 PM2/4/17
to jeffk...@gmail.com, Go Package Management, jeff....@jibeconsulting.com
Do you need to support different versions of your objects in the same
running program?

How do those object schemas evolve — independently, or as a unit?
> --
> You received this message because you are subscribed to the Google Groups "Go Package Management" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to go-package-manag...@googlegroups.com.
> To post to this group, send email to go-package...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

Jeff Kayser

unread,
Feb 5, 2017, 5:17:25 PM2/5/17
to Peter Bourgon, jeffk...@gmail.com, Go Package Management
Hi, Peter.

Thanks for your reply. GREAT questions!

Normally, I wouldn't need to support multiple versions of the same object. I plan to code newer APIs so they are backwards compatible with the corresponding older APIs, so newer code could deal with older objects. However, I can see a use case where it would be helpful to have both object formats in the same running program: migrating older objects to newer, backward-incompatible formats. You could read objects using the old format, morph old objects to become new objects, then write new objects to the database.

Different objects types would probably evolve at different rates, depending on need, so it would make more sense to have them evolve independently. But, I guess I could change some objects, and then retag *everything* with a new version number.

Jeff Kayser
Jibe Consulting | Managing Principal Consultant
5000 Meadows Rd. Suite 300
Lake Oswego, OR 97035
O: 503-517-3266 | C: 503.901.5021
jeff....@jibeconsulting.com


Disclaimer: This electronic message may contain information that is Confidential or legally privileged. It is intended only for the use of the individual(s) and entity named in the message. If you are not an intended recipient of this message, please notify the sender immediately and delete the material from your computer. Do not deliver, distribute or copy this message and do not disclose its contents or take any action in reliance on the information it contains.

Peter Bourgon

unread,
Feb 5, 2017, 9:11:14 PM2/5/17
to Jeff Kayser, jeffk...@gmail.com, Go Package Management
As you've no doubt surmised, there are lots of reasonably effective
ways of accomplishing your goals. I don't think it will be possible to
pick a clear winner without a great deal of domain knowledge. I think
the best advice for you at this stage is to pick a design that
requires the least structure and commitment, and to revisit it as your
project progresses. To generalize a bit, Go prefers you design for the
problems of today, rather than planning for the problems of tomorrow.
Go is designed to be maintained, so it's uncomplicated to refactor
when new, concrete use-cases arise.

With all of that said, and based on your stated requirements, my
intuition would be to avoid using package versioning altogether, and
to lift the versioning into the types directly. That is,

package user

type UserV1 struct{ ... }
type UserV2 struct{ ... }

Eric Johnson

unread,
Feb 6, 2017, 1:22:39 PM2/6/17
to Peter Bourgon, Go Package Management, Jeff Kayser, jeffk...@gmail.com
Indeed, the approach of versioning the structures makes more sense to me, based on the limited knowledge of the problem domain.

Presumably, in some future version of the project, you will want to add to your domain objects. At which point, you can just extend the existing structures - that doesn't invalidate prior data structures.

Also, difficult to tell without understanding your application - why do you need to even version your domain objects? Are you sharing your domain objects across multiple Go applications?

I've found, for example, when I'm building a JSON front-end, that the data going into the JSON result comes from different structures than the domain objects stored in the database.

Eric.


>> To unsubscribe from this group and stop receiving emails from it, send an email to go-package-management+unsub...@googlegroups.com.
>> To post to this group, send email to go-package-management@googlegroups.com.

>> For more options, visit https://groups.google.com/d/optout.
>
> Disclaimer: This electronic message may contain information that is Confidential or legally privileged. It is intended only for the use of the individual(s) and entity named in the message. If you are not an intended recipient of this message, please notify the sender immediately and delete the material from your computer. Do not deliver, distribute or copy this message and do not disclose its contents or take any action in reliance on the information it contains.

--
You received this message because you are subscribed to the Google Groups "Go Package Management" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-package-management+unsub...@googlegroups.com.
To post to this group, send email to go-package-management@googlegroups.com.

Jeff Kayser

unread,
Feb 6, 2017, 2:19:27 PM2/6/17
to Peter Bourgon, jeffk...@gmail.com, Go Package Management, Jeff Kayser
Hi, Peter.

Thank you so much for your reply. I hadn't thought of that option. I like it! It makes it quite obvious which version of the object you are dealing with. With a little front-end detection work (to tell which version of the object you are dealing with), you can restore the object from storage, and re-create an object of the appropriate version. And, it will be easy to convert one version of an object to another version; just put a function in the package to upgrade the object.

I also like it because you just keep including the same package, without having to worry about which version of the package to include. The package can evolve as needed, and you can incrementally add new features.

I'm wondering if I can embed the version number inside the object, have methods on the object detect the object version, and implement the appropriate behavior based on the object version.

What do you think of this architecture?

package user

const (
latestVersion string = "v3"
)

type User struct {
VersionNumber string
Properties map[string]interface{}
}

// for functionality backward compatible to all versions
func (u *User) SomeAction (inputs...) (outputs..., error) {
switch u.VersionNumber {
case "v1": {code for v1 object}
case "v2": {code for v2 object}
case "v3": {code for v2 object}
}
}

// for old functionality that won’t work with the latest version
func (u *User) SomeOldAction (inputs...) (outputs..., error) {
switch u.VersionNumber {
case "v1": {code for v1 object}
case "v2": {code for v2 object}
case "v3": return errors.New("v3 not implemented.")
}
}

// for new functionality that won't work with the oldest version
func (u *User) SomeNewAction (inputs...) (outputs..., error) {
switch u.VersionNumber {
case "v1": return errors.New("v1 not implemented.")
case "v2": {code for v2 object}
case "v3": {code for v3 object}
}
}

func (u *User) UpgradeToVersion (newVersion string) (outputs..., error) }
if u.VersionNumber == latestVersion { return ... }
switch u.VersionNumber + ":" + newVersion {
case "v1:v2": {code to convert v1 to v2}
case "v2:v3": {code to convert v2 to v3}
}
}


The main issue I see is that it is more complicated, and complex is bad. Simple is good.

Another issue I see is that the use of maps makes it not safe for concurrency.
A possible solution would be to create a goroutine for each object, and serialize access via a channel.
If the goroutine needs to return an object to the caller, make a local copy of the official object, and return the local copy to the caller.
That way, the official object remains under the control of the goroutine.
Cons:
One goroutine per object would be a lot of goroutines and channels.
No obvious way to decide when to terminate a goroutine.
If you don't already know the correct channel for your object, finding the correct channel would be challenging.

Does anyone have any other ideas for making it safe for concurrency?

Does anyone see any other potential issues with it?

Jeff Kayser

unread,
Feb 6, 2017, 2:50:29 PM2/6/17
to Eric Johnson, Peter Bourgon, Go Package Management, jeffk...@gmail.com, Jeff Kayser
Hi, Eric.

Thank you so much for your reply.

I want to be able to version objects because if I can handle object versioning well, extending the objects will be easier. My object store (currently flat files) may have old objects (compared to the Go code), so I need to be able to handle old objects.

(In my work life, I am an Oracle DBA, so upgrading/versioning database objects is buried deep in my DNA).

I’ve settled on maps of properties to store object state, because it is flexible enough to contain multiple versions of the same object. Still working out an appropriate solution to have maps be safe for concurrency.

Interesting comment about JSON. Good point. Thanks for the reminder. I haven’t started work on the web UI yet. Currently, just a Go API.

As an aside, I tried to serialize to JSON, but ran into a problem where JSON was not able to serialize a specific structure. I don’t remember the details. XML worked OK, so my V1 used XML to serialize. Since my app will be written entirely in Go, in my next version, I think I’ll probably use encoding/gob to serialize object state.

Jeff Kayser
Jibe Consulting | Managing Principal Consultant
5000 Meadows Rd. Suite 300
Lake Oswego, OR 97035
O: 503-517-3266 | C: 503.901.5021
jeff....@jibeconsulting.com


>> To unsubscribe from this group and stop receiving emails from it, send an email to go-package-manag...@googlegroups.com.
>> To post to this group, send email to go-package...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>
> Disclaimer: This electronic message may contain information that is Confidential or legally privileged. It is intended only for the use of the individual(s) and entity named in the message. If you are not an intended recipient of this message, please notify the sender immediately and delete the material from your computer. Do not deliver, distribute or copy this message and do not disclose its contents or take any action in reliance on the information it contains.

--
You received this message because you are subscribed to the Google Groups "Go Package Management" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-package-manag...@googlegroups.com.
To post to this group, send email to go-package...@googlegroups.com.

Eric Johnson

unread,
Feb 6, 2017, 4:22:20 PM2/6/17
to Jeff Kayser, Peter Bourgon, Go Package Management, jeffk...@gmail.com
At this point, seems like a discussion for golang-nuts group.

As a wrap-up of the discussion here, I think it is worth making a distinction between the serialized form of objects, and the working model that you have as a struct in Go. You don't ever want to actually version the package that contains the serialized forms of your models, because you potentially always need to be able to read the older formats. Especially so, since you're not talking about a database schema tied to a version of your application, but about a file-format that can be copied without your application around.

It sounds to me like you need to think through the end-to-end use cases more - what are you going to do with a JSON API? - before you can figure out what structures might possibly need to be versioned.

Eric.

>> To unsubscribe from this group and stop receiving emails from it, send an email to go-package-management+unsub...@googlegroups.com.
>> To post to this group, send email to go-package-management@googlegroups.com.

>> For more options, visit https://groups.google.com/d/optout.
>
> Disclaimer: This electronic message may contain information that is Confidential or legally privileged. It is intended only for the use of the individual(s) and entity named in the message. If you are not an intended recipient of this message, please notify the sender immediately and delete the material from your computer. Do not deliver, distribute or copy this message and do not disclose its contents or take any action in reliance on the information it contains.

--
You received this message because you are subscribed to the Google Groups "Go Package Management" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-package-management+unsub...@googlegroups.com.
To post to this group, send email to go-package-management@googlegroups.com.

Jeff Kayser

unread,
Feb 6, 2017, 4:35:30 PM2/6/17
to Eric Johnson, Peter Bourgon, Go Package Management, jeffk...@gmail.com, Jeff Kayser
Hi, Eric.

Thank you for your thoughts about this. I appreciate your distinction between the serialized form of the object, and the internal Go form of the object. Until now, I was thinking they were the same, but I appreciate your comment about not versioning the serialization part. I can see the wisdom in that.

Thank you for the reminder about use cases. My app is a job scheduler. I haven’t thought about the web UI yet – probably a mistake. I’ve been focusing on getting the engine working.

I’ll continue the discussion in the go-nuts group.

Jeff Kayser
Jibe Consulting | Managing Principal Consultant
5000 Meadows Rd. Suite 300
Lake Oswego, OR 97035
O: 503-517-3266 | C: 503.901.5021
jeff....@jibeconsulting.com



From: Eric Johnson [mailto:er...@tibco.com]
Sent: Monday, February 06, 2017 1:22 PM
To: Jeff Kayser <jeff....@jibeconsulting.com>
To: Peter Bourgon <pe...@bourgon.org>; Go Package Management <go-package...@googlegroups.com>
>> To unsubscribe from this group and stop receiving emails from it, send an email to go-package-manag...@googlegroups.com.
>> To post to this group, send email to go-package...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>
> Disclaimer: This electronic message may contain information that is Confidential or legally privileged. It is intended only for the use of the individual(s) and entity named in the message. If you are not an intended recipient of this message, please notify the sender immediately and delete the material from your computer. Do not deliver, distribute or copy this message and do not disclose its contents or take any action in reliance on the information it contains.

--
You received this message because you are subscribed to the Google Groups "Go Package Management" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-package-manag...@googlegroups.com.
To post to this group, send email to go-package...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages