Is it considered to separate runtime and cmd/compile irrelevant standard libraries out from the core language repo and hosted in a different module, eg. std?

[changkun]

Dear golang-nuts,

As https://github.com/golang/go/issues/27151, https://github.com/golang/go/issues/6853 and many relevant issues discussed, Go download is huge.

The Go download contains everything in the main repo. Since Go modules are now a success, is it considered to separate all runtime irrelevant (meaning, no need runtime support to implement, e.g. net poller need runtime support because there is a scheduler) libraries out of the main repo?

For instance, the following packages can be separated to be stored in a different repository, called std module:

std/archive

std/bufio

std/bytes

std/compress

std/container

std/context

std/crypto

std/database

std/encoding

std/errors

std/expvar

std/flag

std/go

std/hash

std/html

std/image

std/index

std/io

std/log

std/math

std/mime

std/path

std/plugin

std/regexp

std/sort

std/strconv

std/strings

std/syscall

std/testdata

std/testing

std/text

std/unicode

Say we have a separate repository called golang/std, import these packages can have the same import path, instead of "std/io", go modules can overwrite the import path exactly like what it did for "github.com/user/mod/v2", set the imported std modules in go.mod file, import as "io" directly.

Thanks for reading.

Changkun

[Tamás Gulácsi]

It can be done, but what for?

Just skimmed the list, but your list contains only required packages - maybe mime, regexp, database, expvar could be excluded, but the others are needed for "go build" & co.

[changkun]

but the others are needed for "go build" & co.

This can be solved by first issuing an std module first, then using go modules to import these packages for go build.

The idea is to separate compiler and runtime, then have a minimum go core distribution. The Go development process then can speed up and separate from std and core language. Just like tools in golang/tools repo.

Moreover, right now, the issue tracker is now centralized in golang/go repo, which is verbose for the repo subscriber, a separate repo can also making a decentralized issue tracer possible.

 

[Volker Dobler]

On Monday, 27 January 2020 12:27:35 UTC+1, changkun wrote:

Dear golang-nuts,

As https://github.com/golang/go/issues/27151, https://github.com/golang/go/issues/6853 and many relevant issues discussed, Go download is huge.

Neither of these issues benefits from splitting the stdlib from the

compiler and/or the "runtime" (whatever you mean by that).

Separating the compiler from its stdlib seems strange at least

and does not help, neither to increase development speed nor

to increase convenience.

V. 

[Axel Wagner]

The original message mentions size. The given list is 25MB/337MB uncompressed or 7MB/115MB compressed. So in terms of saved space, we are talking ~6%. It's not nothing, but it's also not a lot. Especially as you'll most likely need to download them right after anyway.

You also mentions splitting up the issue tracker. But note, that even though the various x-repos are already separate repositories, they still use the same issue-tracker. Clearly, having a unified issue tracker is perceived as a *benefit* to the Go team, not a detriment.

It might indeed be worth versioning the stdlib as modules at some point. I know that was discussed as part of the question of what "Go 2" means. Having the ability to bump major versions of the stdlib would enable to overhaul APIs at some point in the future. But note that the Go 1 stability promise stays in place, so again, neither in terms of size, nor in terms of splitting management, this would provide any benefits. Programs using Go v1 stdlib packages will likely continue to exist into the far future, so even *if* there's at some point a v2 of the stdlib, v1 will still need to be shipped and supported (though likely the v1 API would wrap v2 packages, so the actual size increase would be moderate). But all of this is still far in the future, AFAICT.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/e1f86127-bdcb-411b-b50a-991845fc3e90%40googlegroups.com.

[Ian Lance Taylor]

On Mon, Jan 27, 2020 at 7:02 AM 'Axel Wagner' via golang-nuts
<golan...@googlegroups.com> wrote:
>
> The original message mentions size. The given list is 25MB/337MB uncompressed or 7MB/115MB compressed. So in terms of saved space, we are talking ~6%. It's not nothing, but it's also not a lot. Especially as you'll most likely need to download them right after anyway.
>
> You also mentions splitting up the issue tracker. But note, that even though the various x-repos are already separate repositories, they still use the same issue-tracker. Clearly, having a unified issue tracker is perceived as a *benefit* to the Go team, not a detriment.
>
> It might indeed be worth versioning the stdlib as modules at some point. I know that was discussed as part of the question of what "Go 2" means. Having the ability to bump major versions of the stdlib would enable to overhaul APIs at some point in the future. But note that the Go 1 stability promise stays in place, so again, neither in terms of size, nor in terms of splitting management, this would provide any benefits. Programs using Go v1 stdlib packages will likely continue to exist into the far future, so even *if* there's at some point a v2 of the stdlib, v1 will still need to be shipped and supported (though likely the v1 API would wrap v2 packages, so the actual size increase would be moderate). But all of this is still far in the future, AFAICT.

I see two different things to consider. One is the possibility of
splitting the release into separate downloadable files. That by
itself seems simple enough, and we could do it if it seems useful.
But as Axel says it doesn't seem all that useful. It seems like it
would make downloading a release more complex for 99% of people for
the benefit of 1% of people. That's not an impossible tradeoff but it
needs to really be a big help for the 1%. I don't see the benefit
myself (but admittedly I have a fast Internet connection anyhow).

The more interesting thing to consider is that if we split the release
like that, people will naturally start mixing and matching different
versions, either on purpose or by accident. That could be useful in
some ways, but it is also much harder to support. Considering how
much trouble we have making a single unified release, it's not obvious
to me that the Go team has the bandwidth to handle separate release
cycles that need to work together.

Ian

> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAEkBMfHmJi0vOkG39CuTXrqAY0uO-x9aL12desRXE_hsmBUoVw%40mail.gmail.com.

[Robert Engels]

Doesn’t this piggyback on making the runtime a shared library so it can be updated without application recompile?

> On Jan 27, 2020, at 2:24 PM, Ian Lance Taylor <ia...@golang.org> wrote:
>
> On Mon, Jan 27, 2020 at 7:02 AM 'Axel Wagner' via golang-nuts

> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAOyqgcVuXaBeJUwcbw7cLbpMR_GHxRgtggB%2Bi9MWRAQFzrxYeg%40mail.gmail.com.

[Ian Lance Taylor]

On Mon, Jan 27, 2020 at 1:27 PM Robert Engels <ren...@ix.netcom.com> wrote:
>
> Doesn’t this piggyback on making the runtime a shared library so it can be updated without application recompile?

I guess I'm not sure what you mean. That already works today, for
people who choose to use it. It seems independent of how we manage Go
releases.

Ian

[robert engels]

What I am referring to is a Go shared runtime library, that my Go executable would link against at runtime, so that if I compile my Go program against 1.13.0 it will run against any 1.13.x stdlib Go runtime installed by the client.

This way for security patches, only new runtimes need to be shipped, and executables do not need to be recompiled. It also results in far smaller Go binaries (especially with greater Go adoption).

If stdlib runtime changes result in an incompatible API, the minor version needs to be changed, e.g. 1.13.x becomes 1.14.0

Based on this link in the FAQ Why is my trivial program such a large binary? I do not think something like this exists.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAOyqgcU66aAXq6d%3DwAmNLHX3G2V3FsFUsGiMRt1dg%3DpuQQiETA%40mail.gmail.com.

[Ian Lance Taylor]

On Mon, Jan 27, 2020 at 4:22 PM robert engels <ren...@ix.netcom.com> wrote:
>
> What I am referring to is a Go shared runtime library, that my Go executable would link against at runtime, so that if I compile my Go program against 1.13.0 it will run against any 1.13.x stdlib Go runtime installed by the client.
>
> This way for security patches, only new runtimes need to be shipped, and executables do not need to be recompiled. It also results in far smaller Go binaries (especially with greater Go adoption).
>
> If stdlib runtime changes result in an incompatible API, the minor version needs to be changed, e.g. 1.13.x becomes 1.14.0
>
> Based on this link in the FAQ Why is my trivial program such a large binary? I do not think something like this exists.

It turns out that it does exist, at least on some systems.

go install -buildmode=shared std
go build -linkshared x.go

Ian

[changkun]

I think splitting std libraries and language core distribution may speed up and making Go development process more scalable.

Literatures indicates this pattern is the critical aspect to development scalability (M.Beck, One the Hourglass model, ACM communications, https://cacm.acm.org/magazines/2019/7/237714-on-the-hourglass-model/fulltext)

In this case, 

- the main repo team focuses on the core language distribution (meaning a compiler that implements language spec, a runtime package that supports all kinds of runtime mechanism support), provides the minimum "std-only" APIs to support all standard library use cases. Also, a dedicated issue tracker, remove all kinds of sub repo, std issues, better managing language evolution history.

- the std repo team (or "hand over to Go community" but may never be possible), as one of the Go user groups, provides use case and experiences reports to the main repo, similar to the regular regression reports from kubernetes project, versioned std evolves without bothering the main repo team. Go users can tracking std issues without bothered by whatever cmd/* has proposed to change.

As Axel pointed out, this kind of splitting shapes what future Go is, I partly agree with the argument of "all of this is still in far future". However, I see this can be quickly done either within one future early cycle or incrementally kick out these packages from the main repo. There are some observed messes and hacks in a unified release:

- The main repo's crypto package requires x/crypto, x/sys, etc.

- A shared internal package, which not even satisfy the Go language spec

- go:linkname to avoid cycle import

- ...

Rethinking the minimum runtime API spec that supports std libraries may help shape what future Go could be.

A consideration might be the breaking of Go 1 promise. A quick idea to resolve this is to use the Go module as I briefly mentioned before.

Say std libraries are distributed in "golang.org/std/pkgname", importing "pkgname" or "golang.org/std/pkgname" both points to the same standard library. 

Nothing breaks here. "golang.org/x/pkgname2" is a practice already exist.

This kind of decentralization indeed requires a lot of work to the Go team (unclear yet) but seems more decent for the long good, experiences and lessons are learned from many other languages and open source projects, e.g. C++ committee, kubernetes SIGs.

 

>> To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.

>> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/e1f86127-bdcb-411b-b50a-991845fc3e90%40googlegroups.com.
>
> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.

[Ian Lance Taylor]

On Tue, Jan 28, 2020 at 12:27 AM changkun <euryu...@gmail.com> wrote:
>
> - the main repo team focuses on the core language distribution (meaning a compiler that implements language spec, a runtime package that supports all kinds of runtime mechanism support), provides the minimum "std-only" APIs to support all standard library use cases. Also, a dedicated issue tracker, remove all kinds of sub repo, std issues, better managing language evolution history.
> - the std repo team (or "hand over to Go community" but may never be possible), as one of the Go user groups, provides use case and experiences reports to the main repo, similar to the regular regression reports from kubernetes project, versioned std evolves without bothering the main repo team. Go users can tracking std issues without bothered by whatever cmd/* has proposed to change.

You just took one team and turned it into two teams. Where are those
extra people going to come from?

Again, the single team can barely manage to do a Go release today.
We're late every time, and we find it just as frustrating as everyone
else. Increasing the amount of work may be more scalable, but making
something more scalable only helps if you have idle people who want to
help but have nothing to do.

Ian

[changkun]

Dear Ian,

Don't worry, I didn't mean to give suggestions about how the Go team should work, apparently your past decade experience is more convenient than any whimsicality. 

Besides, I believe the Go team's decision is much sophisticated and better, I was randomly posted this and see if I can find any undetailed planned proposal leaks, sorry about any inappropriate wording.

 Changkun