modules and package cyclic dips

[Scott Cotton]

Hi all,

I am wondering about cyclic dependencies w.r.t. modules.  I haven't been able to find anything related, but also haven't verified all the possible related discussion venues, which would take a while.

Let me start out with an observation:  Suppose

module A has packages

   A

   A/x

module B has packages 

 B

 B/y

It is possible for A to import B, and B/y to import A/x.  This would not be a cycle in terms of package imports.

But to my understanding, it would be a cycle in terms of module dependencies.  Normally, I'd think that might be a bad idea.  However, I have seen and used this pattern in order to facilitate testing, where it felt legitimate because at least one of the imports above was only for testing.

So, some questions come to mind:

Should cyclic module dependencies be allowed?

Should a module, following a cycle, be able to depend on an earlier version of itself? (I saw this once...)

Any ideas out there?

Best,

Scott

[Paul Jolly]

Hi Scott,

> Should cyclic module dependencies be allowed?

Yes, indeed in some situations they are totally necessary.

I wrote up an experience report on this very topic:
https://gist.github.com/myitcv/79c3f12372e13b0cbbdf0411c8c46fd5

> Should a module, following a cycle, be able to depend on an earlier version of itself? (I saw this once...)

I can't see how this would work; indeed I can't even unravel it in my
head! Do you have a concrete example to help explain?

Thanks,


Paul

[Scott Cotton]

Hi Paul,

On 9 September 2018 at 13:44, Paul Jolly <pa...@myitcv.io> wrote:

Hi Scott,

> Should cyclic module dependencies be allowed?

Yes, indeed in some situations they are totally necessary.

I wrote up an experience report on this very topic:
https://gist.github.com/myitcv/79c3f12372e13b0cbbdf0411c8c46fd5

Interesting.  I'm not sure what cyclic module dependencies means.  I do know some package managers (not go) boast of having a "solid transitive dependency model".  I hope that any cycles in modules dependencies are either avoided or treated in a very clear simple way by go's modules.

 

> Should a module, following a cycle, be able to depend on an earlier version of itself? (I saw this once...)

I can't see how this would work; indeed I can't even unravel it in my
head! Do you have a concrete example to help explain?

Unfortunately, my concrete example is lost in sands of time, so I can only give a rough idea.  I had cyclic module dependencies, somewhat unintended, but it crept in via some test case.  I was playing with 111 or late 111 release candidate with it and asked it to rebuild go.mod at an untagged HEAD (I think) that was a few commits ahead of say v0.1.2.  Then go.mod had that my module required v0.1.1 of itself in go.mod "indirectly".  All I could figure out was that the module dependency cycle A -> B -> A had B depending on an older version of A via a test case.

Scott

 

 

Thanks,


Paul

--

Scott Cotton

President, IRI France SAS

http://www.iri-labs.com

[Sameer Ajmani]

Are you seeing errors when there are cyclic module dependencies? As I recall, cyclic dependencies between modules (not packages) must be allowed:
https://research.swtch.com/vgo-mvs
"Note that F 1.1 requires G 1.1, but G 1.1 also requires F 1.1. Declaring this kind of cycle can be important when singleton functionality moves from one module to another. Our algorithms must not assume the module requirement graph is acyclic."

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Paul Jolly]

> Interesting. I'm not sure what cyclic module dependencies means. I do know some package managers (not go) boast of having a "solid transitive dependency model". I hope that any cycles in modules dependencies are either avoided or treated in a very clear simple way by go's modules.

In my experience report I explain how the situation comes about in the
case of GopherJS. The "reverse" dependency (i.e. the one that makes
the cycle) is, I suspect, typically (always?) going to be something to
do with testing.

> Unfortunately, my concrete example is lost in sands of time, so I can only give a rough idea. I had cyclic module dependencies, somewhat unintended, but it crept in via some test case. I was playing with 111 or late 111 release candidate with it and asked it to rebuild go.mod at an untagged HEAD (I think) that was a few commits ahead of say v0.1.2. Then go.mod had that my module required v0.1.1 of itself in go.mod "indirectly". All I could figure out was that the module dependency cycle A -> B -> A had B depending on an older version of A via a test case.

I suppose you could say that transitively this module depends on the
same major version of itself, yes.

The same is true in my experience report. And it's by definition that
you end up with a reference to and "older" version because of the
existence dependence of the modules.

But because of semantic import versioning (i.e. same major version in
this situation) and compatibility within that major version,
everything works out, even when running go test all from either of the
modules involved in the cycle.

I found this easiest to reason about by looking at the process
involved with _introducing_ the submodule in the GopherJS repo that
creates the cycle:

https://gist.github.com/myitcv/79c3f12372e13b0cbbdf0411c8c46fd5#the-process


Paul

[Scott Cotton]

Hi Sameer,

I don't know what is considered an error and not an error with cyclic module dependencies.

Honestly, it makes my head hurt and simply requires too much thought that I'd rather spend on the code.

For example, I don't want to think what will happen to some SCC in a module dependency graph after

a decade of development, in particular if a module can depend on an earlier version of itself.

Scott

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Scott Cotton]

Hi Paul,

Yes I see that in terms of compatibility it "all works out", but it seems 

underspecified what should happen. Also, although your experience reports are

clearly presented and make sense, when I step back my own impression 

is: that's not simple.  I'd rather be spending time coding than figuring out 

or worrying about chains of cyclic dependencies going back in time indefinitely.

Scott

[Paul Jolly]

Hi Scott,

> Yes I see that in terms of compatibility it "all works out", but it seems
> underspecified what should happen.

Which bit do you think is underspecified?

To my mind the behaviour is very clearly defined, notwithstanding the
next point.

> Also, although your experience reports are
> clearly presented and make sense, when I step back my own impression
> is: that's not simple.

That's certainly true. But...

> I'd rather be spending time coding than figuring out
> or worrying about chains of cyclic dependencies going back in time indefinitely.

As would I, but I think in some situations these cycles are
unavoidable, assuming you can't collapse the two modules down into one
(and in the GopherJS we can't).

Thanks,


Paul

[Scott Cotton]

Hi Paul,

On 9 September 2018 at 16:26, Paul Jolly <pa...@myitcv.io> wrote:

Hi Scott,

> Yes I see that in terms of compatibility it "all works out", but it seems
> underspecified what should happen.

Which bit do you think is underspecified?

Perhaps how to ensure a security patch is actually applied for module that depends on earlier versions of itself indefinitely might be one consideration worth specifying.  Otherwise, I'm not sure to what extent the docs on swtch.com represent the authoritative intent behind the docs, as I can't find anywhere on golang.org that refers to cyclic module dependencies and how to deal with them, and it is unclear to what extent or whether Russ's work on swtch.com still constitute "the" current docs.

 

To my mind the behaviour is very clearly defined, notwithstanding the
next point.

> Also, although your experience reports are
> clearly presented and make sense, when I step back my own impression
> is: that's not simple.

That's certainly true. But...

> I'd rather be spending time coding than figuring out
> or worrying about chains of cyclic dependencies going back in time indefinitely.

As would I, but I think in some situations these cycles are
unavoidable, assuming you can't collapse the two modules down into one
(and in the GopherJS we can't).

You could use 3 (or more?) modules acyclicly, couldn't you?

Scott

Thanks,


Paul

[Sameer Ajmani]

With respect to errors, I'm asking how things failed when you had a cyclic module dependency. My expectation is that this should just work. If your module 0.12 has a dependency on itself with min version 0.11, then 0.12 satisfies that dependency (as long as it's following the import compatibility rule, which isn't necessarily expected for pre-1.0 modules).

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Scott Cotton]

Hi Sameer,

When I had a module self-dependency, I considered the fact that it worked a bug.  I had not followed 

closely enough til this discussion to think of it as expected.

As someone who has a tendency to often ask themself: "worse case how can this be a problem?"

the list is indeed long and severe for modules which depend on themselves backwards in time.  

For cyclic dependencies which are somehow synchronised so that there is no backwards in time

propagation, my impression would be "that's complicated", but I can't see offhand how it would be

as problematic as backward in time self dependencies.

Scott  

 

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Scott Cotton

President, IRI France SAS

http://www.iri-labs.com

[Sameer Ajmani]

If a module depends on an earlier version itself, and successive versions are backwards compatible, why is it not OK for the current version to satisfy that dependency?

Scott  

 

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Scott Cotton

President, IRI France SAS

http://www.iri-labs.com

[Scott Cotton]

Hi Sameer,

Thanks for asking, here are some thoughts,

With time, this could create man many many copies of (different versions) of a module.

this would slow down fetch, storage, compilation, etc potentially a lot, and it would only

get worse and worse over time.

if a security patch is applied to the most recent version in a version-compatible space of

a module that depends on an earlier version of itself, then the security hole may still exist.

Moreover, if the SCC of module dependencies is outside the control of the authors of the 

module being patched, it seems there is might be no way they could propagate the patch 

without editing history, which violates the very notion of versioning to begin with.

The notion of software moving forward by versioning is in part an increase in reliability, 

not just security patches, so I would be frightened to use cyclic modules even in code

for which I were certain there would be no security patches (like fixed memory, known cpu bounds 

math algorithms for example)

Other than that, it is to me very confusing and counter-intuitive that a version of some software

depend on a previous version of itself.  Maybe I'm missing something in the modules 

specification or vision, and maybe, (even hopefully) I am wrong about these concerns.  

I could list more related ideas, but given that I might be wrong I'll leave it at that.

Thanks,

Scott

Scott  

 

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Scott Cotton

President, IRI France SAS

http://www.iri-labs.com

--

Scott Cotton

President, IRI France SAS

http://www.iri-labs.com

[Sameer Ajmani]

I think there's a disconnection between how you and I understand this system works. In a given build, there is only a single version of a module; there cannot be multiple copies, let alone many copies.So if v1.11 of module A depends on v1.3 of module B, which in turn depends on v1.12 of module A, then the build will choose A v1.12. The is only one version of A in the build.

Scott  

 

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Scott Cotton

President, IRI France SAS

http://www.iri-labs.com

--

Scott Cotton

President, IRI France SAS

http://www.iri-labs.com

[Scott Cotton]

Well, ok.

Did you mean if A v1.12 of a module depends on v1.3 of B which depends on v1.11 of A?

So go modules don't actually necessarily depend on the listed dependencies?  

That is quite counteruintintive to me.

It is much simpler in any case to use acyclic dependencies.

Scott

Scott  

 

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Scott Cotton

President, IRI France SAS

http://www.iri-labs.com

--

Scott Cotton

President, IRI France SAS

http://www.iri-labs.com

--

Scott Cotton

President, IRI France SAS

http://www.iri-labs.com

--

Scott Cotton

http://www.iri-labs.com

[Scott Cotton]

Maybe if I say more it will help the communication/alleviate the disconnection:

First, my overall point is that I have decided to use only acyclic dependencies, as cyclic ones are too complicated for me.

I don't feel easily convincible otherwise; that comes from a combination of using go modules where some cycles crept in

and a strong bias for a dead simple dependency methodology. 

Second, there is perhaps some confusion about what "depends" means in this thread.

(*) My intuition of what A depends on B means is, in plain English, that A references B in order to work.  So one would have

to have at least one copy of B in order for A to work, for example.

In go.mod, there is a require statement which lists requirements of a module.  I have been interpreting requirements as 

dependencies.  Requirements, follow a syntax which usually take the form of a (package, semver) tuple, nicely documented

with lots of syntactic variations that, while nice and useful, I honestly hope to avoid having to think about due to my bias above.

To me, if A vX.Y.Z (your current module you are working on) requires  B v.Q.R.S in go.mod, then that means A vX.Y.Z depends on B v.Q.R.S. In a few cases we've had A=B and Q.R.S < X.Y.Z.  

There are perhaps technical definitions of dependency in how the system works that differ from how I tend to interpret what a dependency is, as above.  If this is the case, it adds to cognitive noise for me because it takes mental effort to understand that I 

prefer to place elsewhere.

Now there is perhaps also some question of "// indirect", as that seems to be involved in the dendency cycles discussed so far. 

There is some documentation, such as on the wiki, that states that these are transitive dependencies not specified in other dependencies.  But it is still presented as a "dependency". 

Given my intuition of dependency (*) above, cycles simply don't make sense.  Not at all, in any finite memory universe :)  I am 

aware that one can refine the dependency order so it all makes sense and works in terms of semver compatibility, as in Paul's experience report.  Maybe somewhere in that process and our dialog about it, the idea of compatibility got mixed up with dependency in order to pull off the MVS magic that made it "all work out".  

Fortunately, for me, I have simple solution: don't use cycles between module dependencies (eg go.mod requirements).  I'm afraid however that this leaves me in a relatively uninformed state to inform a discussion about cycles in "how the system works" in much more detail.

Best,

Scott

[Paul Jolly]

> First, my overall point is that I have decided to use only acyclic dependencies, as cyclic ones are too complicated for me.
> I don't feel easily convincible otherwise; that comes from a combination of using go modules where some cycles crept in
> and a strong bias for a dead simple dependency methodology.

That's totally fine, with the slight caveat being that in some cases
they're unavoidable (e.g. the GopherJS case).

I've drafted a guide that presents what I hope is a simpler example of
a cyclic dependency:

https://github.com/myitcv/go-modules-by-example/tree/013_cyclic/013_cyclic

One point to note is that this example will become simpler still once
https://github.com/golang/go/issues/26241 is resolved.

Would appreciate any feedback, but please don't feel obliged.

I fear it may take a few iterations to get the wording/emphasis right...

Thanks,


Paul

[Sameer Ajmani]

Go modules can use the listed version or later minor versions. So if B depends on v1.11 of A, then it can also work with v1.12 of A. Therefore a valid set of versions for the build is A v1.12 and B v1.3.

Scott  

 

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Scott Cotton

President, IRI France SAS

http://www.iri-labs.com

--

Scott Cotton

President, IRI France SAS

http://www.iri-labs.com

--

Scott Cotton

President, IRI France SAS

http://www.iri-labs.com

[Scott Cotton]

Thanks for the clarification.

Maybe having a way for "go mod" to output what is used would be less confusing?  I would have found so at least.

Cycles crossing major versions may be more problematic. 

similarly in the case of  coordinating a collection of modules, dependency cycles may lead unintentionally to different used versions

depending on the build context.  While the potential use of multiple versions is a useful feature of modules, I am not convinced 

that in this context it is desirable.  Also, if cycles can span arbitrarily large sets of uncoordinated modules, then it seems to me that

is not the intended use case for cycles and that might be undesirable or even lead to the inability of a module maintainer in the cycle

to effectively propagate an update.

If I come across anything more concrete, I'll file an issue.  Just food for thought at this point.

Best

Scott 

[thepud...@gmail.com]

   > "Maybe having a way for "go mod" to output what is used would be less confusing?"

Hi all,

To see a list of the selected module versions, one way is to use use 'go list -m all'.  This shows the actual versions used based on all of the various requirements in a build.  Sample output:

   $ go list -m all

   example.com/my/module

   github.com/aws/aws-sdk-go v1.15.35

   github.com/go-ini/ini v1.25.4

Another other useful tool is for inspecting requirements is 'go mod graph' (which prints the module requirement graph), and if you are curious why a particular module is showing up in your go.mod, you can run 'go mod why -m <module>' to answer that question. 

You can read some more about these here:

   https://tip.golang.org/cmd/go/#hdr-The_main_module_and_the_build_list

   https://tip.golang.org/cmd/go/#hdr-Explain_why_packages_or_modules_are_needed

   https://tip.golang.org/cmd/go/#hdr-Print_module_requirement_graph

The first link above also provides a short description of the build list, which is related to several of the questions raised in this thread:

   ----------------

   "The set of modules providing packages to builds is called the "build list". The build list initially contains only the main module. Then the go command adds to the list the exact module versions required by modules already on the list, recursively, until there is nothing left to add to the list. If multiple versions of a particular module are added to the list, then at the end only the latest version (according to semantic version ordering) is kept for use in the build."

   ----------------

--thepudds

[Scott Cotton]

Thanks!  Very useful.

+1 for aliasing "go mod list" to "go list -m"

--
You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/h2r8ktgOOZI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-nuts+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Scott Cotton

http://www.iri-labs.com

[thepud...@gmail.com]

Hi all,

One additional set of brief comments.

This was a fairly nuanced and long thread, and I'm pretty sure I did not follow all of it, but I think a question related to the conversation here is -- what does 'require foo v1.1.1' really mean?

Perhaps one way to think about that usage of the verb 'require' is that it really needs to be read in the context of semantic versioning (https://semver.org).

If something states it requires v1.1.1 of foo, in the context of semantic versioning that really means that particular requirement for v1.1.1 would be satisfied by foo >= v1.1.1 and < v2, given semver defines versions in that interval to be valid and backwards compatible substitutions for v1.1.1 (and anything v2 or higher is considered incompatible with any v1 version).

And a variation of that first question is -- what does it mean for a build if you have both a 'require foo v1.1.1' and a 'require foo v1.2.2'?

One initial guess might be that the build ends up with two copies of foo (v1.1.1 and v1.2.2) given there are two 'require' statements with those two different versions. That is a reasonable guess, but not actually what happens.

If you instead read that verb 'require' in the context of semantic versioning, it means that if the build instead picks a _single_ copy of foo at version v1.2.2, that actually would satisfy _both_ of the requirements for v1.1.1 and v1.2.2 (given v1.2.2 is defined by semver to be a backwards compatible substitution for v1.1.1).  And of course, this is what the go build system actually would do in this scenario.

Said another way, when you use modules, the go command _mandates_ that modules use semantic versioning and expects that the versions accurately describe compatibility: it assumes that v1.2.2 is a backwards compatible replacement for v1.1.1, and hence v1.2.2 is a valid answer if you have both 'require foo v1.1.1' and a 'require foo v1.2.2'.

In any event, this is all still fairly new, and everyone is still learning about the overall system. (The broader community is of course learning about the new system, but even the people who implemented this system are in a learning phase, including because they are learning about how the community will use the new system). 

--thepudds

To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Scott Cotton

http://www.iri-labs.com

[Sameer Ajmani]

Thanks. The relationship between semantic versioning and Go packages is described in detail here: https://research.swtch.com/vgo-import

[Scott Cotton]

Thanks Sameer,

I have a question about semver and modules, perhaps it was already answered somewhere, but if so I don't know where.

The question is: semver.org allows for pre-release notation, the document below does not mention pre-release.

To some extent Go uses pre-releases.

I, for one, find the idea useful and would like to use pre-release info with modules.

So can anyone comment, inform, or provide a pointer to some documentation about pre-releases as they relate to modules?

Best,

Scott

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/h2r8ktgOOZI/unsubscribe.

To unsubscribe from this group and all its topics, send an email to golang-nuts+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[thepud...@gmail.com]

Hi Scott,

Modules do provide support for semver pre-release notation. You can read more about it in these sections of the documentation, for example:

  https://tip.golang.org/cmd/go/#hdr-Module_queries

  https://tip.golang.org/cmd/go/#hdr-Pseudo_versions

  https://tip.golang.org/cmd/go/#hdr-Module_aware_go_get

--thepudds

[Scott Cotton]

Thanks again @thepudds for the helpful info.  

with module aware go get -u at tip on golang.org, it says

"""

The -u flag instructs get to update dependencies to use newer minor or patch releases when available. Continuing the previous example, 'go get -u A' will use the latest A with B v1.3.1 (not B v1.2.3).

"""

For example, if I have a requirements for M vQ.R.S and the latest available minor or patch release is 

M vQ.R.{S+1}-alpha.1, with nothing else after vQ.R.S available.

Then does go get -u in module aware mode update to the alpha?  If so, should it?  (I would think not, 

but reading the docs I would think it would update to the alpha)

Scott

 

To unsubscribe from this group and all its topics, send an email to golang-nuts+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Scott Cotton]

Hi all,

I just ran a test and answered my own question:

It does not consider a "prerelease" a "release",  so get get -u doesn't update to prereleases.

In case it helps anyone, this distinction of terminology was unclear to me in the move to 

support prereleases.  Apparently, a prerelease is not a kind of release in the docs on tip.golang.org.

Best,

Scott 

[thepud...@gmail.com]

Hi Scott,

Regarding your question about pre-release tags and 'go get -u', my understanding is that when the documentation says:

    'The -u flag instructs get to update dependencies to use newer minor or patch releases when available.'

  

...the use of the phrase 'newer minor or patch release' is exclusive of prelease tags (e.g., 'v1.2.3-alpha1').

My understanding here is based on the documentation, what I've read elsewhere, and my experience so far.

There are some similar sentiments expressed elsewhere in the documentation, such as in the "Module-aware go get" section:

    'For each named package or package pattern, get must decide which version of the corresponding module to use. By default, get chooses the latest tagged release version, such as v0.4.5 or v1.2.3. If there are no tagged release versions, get chooses the latest tagged prerelease version, such as v0.0.1-pre1. If there are no tagged versions at all, get chooses the latest known commit.'

Or in the "Module queries" section:

    'All queries prefer release versions to pre-release versions. For example, "<v1.2.3" will prefer to return "v1.2.2" instead of "v1.2.3-pre1", even though "v1.2.3-pre1" is nearer to the comparison target.'

All that said, as I mentioned before, I think we are all still learning...

--thepudds

[thepud...@gmail.com]

Hi Scott, 

Great, it looks like you ran your own test too. 

(We had both posted at approximately the same time a few minutes ago).

--thepudds

[Sameer Ajmani]

Hi Scott, thanks for boiling this down to the specific behavior you're looking for. Is there a specific feature request you'd like to make for this? If so, please file it in golang.org/issue. Thanks!

S