blog post on go.mod and go.sum misconceptions
1,476 views
Skip to first unread message
Daniel Martí
Aug 3, 2021, 7:17:47 PM8/3/21
to golang-dev
Hi all,
Modules have come a long way in the past few years, and I think a lot of
the initial confusion and common questions around them are now fairly
well understood in general.
From what I've been seeing on the issue tracker and forums, as well as
from colleagues at work, it seems like we have one major common
misconception left: what the go.mod and go.sum files actually mean.
go.sum is the easier one. Many users fall into the initial assumption
that go.sum is a form of lock file, as it seems to list versions for all
direct and indirect dependencies. And they might even use modules for
years without realising that that's not the case.
And the misunderstanding is reasonable, especially given that many other
package managers also use two files: one to declare the package and list
its dependencies, and another to lock their versions. And since go.mod
declares the module, it's only natural to expect go.sum to be a form of
lock file.
go.mod is a trickier one. I think many users think they can directly
consume and edit the go.mod file, when in practice trying to do that has
many nuances. Here are some examples of confusion I've seen recently:
* I want to upgrade a dependency version.
When I add a line to go.mod, "go mod tidy" removes it.
* I only want to upgrade dependency A in my go.mod.
"go mod tidy" is then upgrading or removing other dependencies too.
* I want to downgrade a dependency version in my go.mod.
"go mod tidy" undoes the downgrade.
* I want to see my deps and their versions, including indirect deps.
Reading go.mod mostly works, but some deps are missing.
The root causes of these unexpected behaviors can vary depending on the
situation. But at the core, the issue is always the same: the user is
attempting to directly understand or edit go.mod.
And that's tricky: to do so, you at least need to fully understand SIV.
Not only that, the rules that "go mod tidy" follows for pruning go.mod
aren't easy to grasp, and they'll get trickier for 1.17.
It's worth noting that the aging Modules wiki article, now largely
superseded by https://golang.org/ref/mod, tackles some of these topics:
https://github.com/golang/go/wiki/Modules#faqs--gomod-and-gosum
And I think that, if a user sat down and spent two hours digesting
/ref/mod, they would probably get a better grasp.
That said, I think it's naive to assume that users will carefully read
that doc, let alone stay up to date with it and perfectly understand how
the go.mod and go.sum files work at a given moment.
I think our best bet is to actively encourage people to use modules
through cmd/go. For example, using "go list -m" to see the list of
currently used modules and their versions, and "go get" to update
dependencies.
One example of a greatly under-utilised command is "go list -m all".
Far too many users are filling this need by reading go.mod directly.
(Worth noting that looking at go.mod diffs in PRs/commits can still be
useful to get a sense of what's changed, even without understanding MVS
and go.mod tidying. I don't think we need to fight against that.)
As much as I like canonical documentation pages like /ref/mod, I think
this kind of usability outreach best matches a post on blog.golang.org.
Think like https://blog.golang.org/go1.13-errors, which brought many
gophers up to date with new ways to use errors, but for go.mod and
go.sum files. That blog post is still useful as an introduction today.
I think it would be fine for the content to be written once and mostly
left static. The advice to generally not try to consume or edit go.mod
and go.sum directly should be timeless for the foreseeable future.
Perhaps the way we use "go get" or "go mod" will change over time, and
for that reason the blog post should remain brief and link to /ref/mod
for details and examples.
I'm sure there are more plans for content on modules. This is just my
current impression and a hopefully useful take on a solution :)
Modules have come a long way in the past few years, and I think a lot of
the initial confusion and common questions around them are now fairly
well understood in general.
From what I've been seeing on the issue tracker and forums, as well as
from colleagues at work, it seems like we have one major common
misconception left: what the go.mod and go.sum files actually mean.
go.sum is the easier one. Many users fall into the initial assumption
that go.sum is a form of lock file, as it seems to list versions for all
direct and indirect dependencies. And they might even use modules for
years without realising that that's not the case.
And the misunderstanding is reasonable, especially given that many other
package managers also use two files: one to declare the package and list
its dependencies, and another to lock their versions. And since go.mod
declares the module, it's only natural to expect go.sum to be a form of
lock file.
go.mod is a trickier one. I think many users think they can directly
consume and edit the go.mod file, when in practice trying to do that has
many nuances. Here are some examples of confusion I've seen recently:
* I want to upgrade a dependency version.
When I add a line to go.mod, "go mod tidy" removes it.
* I only want to upgrade dependency A in my go.mod.
"go mod tidy" is then upgrading or removing other dependencies too.
* I want to downgrade a dependency version in my go.mod.
"go mod tidy" undoes the downgrade.
* I want to see my deps and their versions, including indirect deps.
Reading go.mod mostly works, but some deps are missing.
The root causes of these unexpected behaviors can vary depending on the
situation. But at the core, the issue is always the same: the user is
attempting to directly understand or edit go.mod.
And that's tricky: to do so, you at least need to fully understand SIV.
Not only that, the rules that "go mod tidy" follows for pruning go.mod
aren't easy to grasp, and they'll get trickier for 1.17.
It's worth noting that the aging Modules wiki article, now largely
superseded by https://golang.org/ref/mod, tackles some of these topics:
https://github.com/golang/go/wiki/Modules#faqs--gomod-and-gosum
And I think that, if a user sat down and spent two hours digesting
/ref/mod, they would probably get a better grasp.
That said, I think it's naive to assume that users will carefully read
that doc, let alone stay up to date with it and perfectly understand how
the go.mod and go.sum files work at a given moment.
I think our best bet is to actively encourage people to use modules
through cmd/go. For example, using "go list -m" to see the list of
currently used modules and their versions, and "go get" to update
dependencies.
One example of a greatly under-utilised command is "go list -m all".
Far too many users are filling this need by reading go.mod directly.
(Worth noting that looking at go.mod diffs in PRs/commits can still be
useful to get a sense of what's changed, even without understanding MVS
and go.mod tidying. I don't think we need to fight against that.)
As much as I like canonical documentation pages like /ref/mod, I think
this kind of usability outreach best matches a post on blog.golang.org.
Think like https://blog.golang.org/go1.13-errors, which brought many
gophers up to date with new ways to use errors, but for go.mod and
go.sum files. That blog post is still useful as an introduction today.
I think it would be fine for the content to be written once and mostly
left static. The advice to generally not try to consume or edit go.mod
and go.sum directly should be timeless for the foreseeable future.
Perhaps the way we use "go get" or "go mod" will change over time, and
for that reason the blog post should remain brief and link to /ref/mod
for details and examples.
I'm sure there are more plans for content on modules. This is just my
current impression and a hopefully useful take on a solution :)
Jay Conrod
Aug 4, 2021, 2:47:59 PM8/4/21
to Daniel Martí, Steve Traut, Bryan Mills, Michael Matloob, golang-dev
+Steve Traut +Bryan Mills +Michael Matloob
I would be really great to have an article like this. I feel like we could even write one just for 'go get'.
My main question is whether this should be a blog post that's only published once, a "facts and myths" document that's updated over time, or a section in an FAQ like https://golang.org/doc/faq. My instinct is that we should go with one of the latter two options so that folks can find it more easily later. But a blog post is a good way to get a lot of visibility, too.
--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-dev/20210803231738.n5qsm5ro2ttos5uf%40carbon.localhost.
Daniel Martí
Aug 5, 2021, 8:45:26 AM8/5/21
to Jay Conrod, Steve Traut, Bryan Mills, Michael Matloob, golang-dev
> I would be really great to have an article like this. I feel like we could
> even write one just for 'go get'.
That's also a great idea. "go get" and "go install" are both going
> even write one just for 'go get'.
through a bit of a redesign with modules, and some outreach would help
speed up the process of getting users up to speed.
> My main question is whether this should be a blog post that's only
> published once, a "facts and myths" document that's updated over time, or a
> section in an FAQ like https://golang.org/doc/faq. My instinct is that we
> should go with one of the latter two options so that folks can find it more
> easily later. But a blog post is a good way to get a lot of visibility, too.
Docs are great, and I like how complete /ref/mod has become.
Long term, I agree that any canonical information should be in a well
maintained document like that.
In the short term, the vast majority of users won't even notice if you
add a section to a large documentation page about go.mod and go.sum
misconceptions. The pages are already very large, so I think we can only
expect a user to skim the entire document once, and go back to it
sporadically to look up answers for the questions they already have.
And I don't think some users realise that they might need to change
their workflows a bit, to make modules work better. They might get
confusing error messages, or get frustrated when things don't work, but
ultimately they might not know the root reason or where to look up a
solution.
That's why a blog post, which will reach many more users, could be a way
to accelerate the recycling of misunderstandings and deprecated workflows.
The blog post itself could be light on details and instead point to
sections of maintained docs, to prevent going out of date too fast.
As an example: a useful "news" blog post, like the one about 1.13
errors, will most likely hit the top of Reddit and Hacker News, and be
shared on places like Twitter for a few weeks. I don't think I've ever
seen sections of maintained docs get that much attention :)
Stephen Illingworth
Aug 5, 2021, 8:54:20 AM8/5/21
to golang-dev
Speaking as a likely consumer of such information, I think this is a worthwhile exercise. I've been successfully using modules since the inception by editing go.mod by hand, so it's interesting to hear that this isn't the ideal way of working. And I'm sure I'm not alone in thinking this.
You say that 'go mod tidy' can upset hand editing but I'm not even sure why I would ever want to run 'go mod tidy' in the first place, that's how far adrift my understanding is. And yet, as I say, I feel I'm using the modules system successfully.
I look forward to reading a up to date tutorial/blog on module workflow.
David Golden
Aug 5, 2021, 9:50:50 AM8/5/21
to golang-dev
On Thu, Aug 5, 2021 at 8:45 AM 'Daniel Martí' via golang-dev <golan...@googlegroups.com> wrote:
That's why a blog post, which will reach many more users, could be a way
to accelerate the recycling of misunderstandings and deprecated workflows.
The blog post itself could be light on details and instead point to
sections of maintained docs, to prevent going out of date too fast.
I like this idea very much. I'm willing to volunteer as a reader/editor. I'm not a modules expert, but I know enough that I'm probably representative of the audience you're trying to reach. (Outside Go, I've spent over a decade as a CPAN administrator and maintainer of the Perl module toolchain, so I have a deep appreciation of the issues that modules are attempting to solve.)
David
Daniel Martí
Aug 10, 2021, 8:02:53 AM8/10/21
to 'Daniel Martí' via golang-dev
On Wed, Aug 04, 2021 at 00:17:38 +0100, 'Daniel Martí' via golang-dev wrote:
> And that's tricky: to do so, you at least need to fully understand SIV.
For those reading this thread later, I made a typo here: I meant MVS,
> And that's tricky: to do so, you at least need to fully understand SIV.
not SIV. MVS, along with the "go mod tidy" rules being changed in Go
1.17, is part of why it's hard for a human to fully understand the
dependency graph by reading a go.mod file.
Anderson Queiroz
Aug 10, 2021, 12:45:02 PM8/10/21
to Daniel Martí, 'Daniel Martí' via golang-dev
Hi all,
This is a great idea! It'd be really helpful to have such material. I've been helping my company to migrate to Go in the last 2 years, and it's a tipic that causes a lot of confusion.--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-dev/20210810120240.2geqqdh6qxnkkypf%40carbon.localhost.
--
Anderson Queiroz | Tech Lead Engineering
Blacklane GmbH | Feurigstr. 59 | 10827 Berlin | Germany
Managing Directors: Dr. Jens Wohltorf, Frank Steuer, Dr. Sascha Meskendahl
Register court: Amtsgericht Berlin-Charlottenburg, Germany
Registration No.: HRB 136941 B, VAT No.: DE281637869
Managing Directors: Dr. Jens Wohltorf, Frank Steuer, Dr. Sascha Meskendahl
Register court: Amtsgericht Berlin-Charlottenburg, Germany
Registration No.: HRB 136941 B, VAT No.: DE281637869
Daniel Martí
Sep 22, 2021, 6:42:04 AM9/22/21
to golan...@googlegroups.com
Hi all,
This thread is over a month old, and at least in my case,
I had already forgotten to follow up about it :)
It seems like we're all in agreement the blog post should happen,
so to be able to properly track that,
I've opened https://github.com/golang/go/issues/48539.
This thread is over a month old, and at least in my case,
I had already forgotten to follow up about it :)
It seems like we're all in agreement the blog post should happen,
so to be able to properly track that,
I've opened https://github.com/golang/go/issues/48539.
Reply all
Reply to author
Forward
0 new messages