proposal: public module authentication with the Go notary

[Russ Cox]

Hi all,

I wanted to let people here know about the proposal design we just published:
golang.org/design/25530-notary, for golang.org/issue/25530. (We also mentioned this general idea in our blog post from back in December, https://blog.golang.org/modules2019.)

As usual, comments are welcome on the issue or,

if you prefer not to use the issue tracker, here in this thread.

Thanks.

Russ

[Sam Whited]

TL;DR — Right now when fetching a module Google isn't involved, and I'd
like it to stay that way. Please let me configure and use different
notaries without also using a proxy.

> There is no plan to allow use of alternate notaries, which would add
> complexity and potentially reduce the overall security of the system,
> allowing different users to be attacked by compromising different
> notaries.

If you're somewhere that Google doesn't service (eg. due to U.S. export
laws) does this mean you're entirely out of luck and can't use the new
security features? It seems unfortunate that a developer in Iran would
have to fall back to the TOFU model just because Google decided to bless
themselves and not allow anyone else to run a notary or jump through
extra hoops to configure or run a proxy.

> We originally considered having multiple notaries signing individual
> go.sum entries and requiring the go command to collect signatures from
> a quorum of notaries before accepting an entry. That design depended
> on the uptime of multiple services and could still be compromised
> undetectably by compromising enough notaries. That is, that design
> would blindly trust a quorum of notaries.

This seems strictly superior to blindly trusting a single notary. Why
does Google think it will be more secure than Google and Mozilla, or
Google and Microsoft, or Google and <your company>?

As far as uptime is concerned, you're always limited to the uptime of
your least available notary, I don't necessarily think Google's uptime
will be any better than any other large company that might choose to run
a notary, so that seems fine: I have the option of trusting only
notaries with high availability, meaning that it's likely no worse than
if I just trusted Google.

> The design presented here uses the transparent log eliminates blind
> trust in a quorum of notaries and instead uses a “trust but verify”
> model with a single notary.

We could also "trust but verify" a quorum of notaries, so this seems
like a false dichotomy.

On a more personal note, having a non-Google controlled notary seems
like an absolute requirement to me and I would prefer to avoid using a
Google run service entirely if possible. Google doesn't need to know
anything about what modules my company is using. I do appreciate the
privacy section, which addresses this, but tying the notary use to using
a proxy or running your own proxy is likely out of reach for many
individuals (including me).

—Sam

> --
> You received this message because you are subscribed to the Google
> Groups "golang-dev" group. To unsubscribe from this group and stop
> receiving emails from it, send an email to golang-
> dev+uns...@googlegroups.com. For more options, visit
> https://groups.google.com/d/optout.

--
Sam Whited

[Manlio Perillo]

Thanks for the proposal.

I noted that the notary cache is stored in $GOPATH/pkg, and this seems to conflict with the proposal to remove $GOPATH/pkg.

Manlio Perillo 

[mikioh.publi...@gmail.com]

hi,

i have a request and a question:

- please share the detail on the notary service; the proposal mentions certificate transparency as background information but i'm still not sure what's the entities, role model and functionality of the notary service from the workflow point of view. when referring to rfc6962bis, i'm guessing that we are go.sum file publishers, the notary service is a go.sum submitter and server, the cmd/go is a go.sum client (besides, probably no exposed monitoring and auditing features for the stored go.sum lines.)

- can a package without a go.sum file prevent the notary service from submitting go.sum lines?

[Filippo Valsorda]

On Wed, Mar 6, 2019 at 8:38 AM <mikioh.publi...@gmail.com> wrote:

- please share the detail on the notary service; the proposal mentions certificate transparency as background information but i'm still not sure what's the entities, role model and functionality of the notary service from the workflow point of view. when referring to rfc6962bis, i'm guessing that we are go.sum file publishers, the notary service is a go.sum submitter and server, the cmd/go is a go.sum client (besides, probably no exposed monitoring and auditing features for the stored go.sum lines.)

The references are RFC 6962 (not RFC 6962-bis, which is substantially different), and https://research.swtch.com/tlog.

The contents of the leafs/records are go.sum lines for the specific module version.

 

- can a package without a go.sum file prevent the notary service from submitting go.sum lines?

No, note that what's notarized is a synthetic go.sum file containing the two lines for the specific module version, not the go.sum inside the .zip, which instead contains hashes for its dependencies.

[Russ Cox]

On Mon, Mar 4, 2019 at 1:06 PM Sam Whited <s...@samwhited.com> wrote:

TL;DR — Right now when fetching a module Google isn't involved, and I'd
like it to stay that way. Please let me configure and use different
notaries without also using a proxy.

As it says in the proposal, the notary configuration is in $GOROOT/lib/notary/notary.cfg. Editing that file will change what notary is expected.

Russ

[Russ Cox]

On Tue, Mar 5, 2019 at 9:16 AM Manlio Perillo <manlio....@gmail.com> wrote:

I noted that the notary cache is stored in $GOPATH/pkg, and this seems to conflict with the proposal to remove $GOPATH/pkg.

The proposal to remove $GOPATH/pkg (golang.org/issue/4719) was filed when that meant the compiled .a files.

Now we store a bit more there. When we remove the compiled .a files, maybe we'll leave

the other stuff there, or maybe we will move it. Time will tell.

I posted a longer reply on the issue tracker:

https://github.com/golang/go/issues/4719#issuecomment-470268217

Russ

[Sam Whited]

On Wed, Mar 6, 2019, at 20:23, Russ Cox wrote:
> As it says in the proposal, the notary configuration is in
> $GOROOT/lib/notary/notary.cfg. Editing that file will change what
> notary is expected.

But before I can convince every new employee to configure the proper
notary they will have managed to accidentally send all my dependencies
to Google without realizing it.

—Sam

[Matthew Dempsky]

Perhaps it would help if the proposal outlined a privacy policy that Google commits to adhere to for operating the Go notary, and which can only be revised through the Go proposal process. For example, Google's Public DNS service provides a privacy policy: https://developers.google.com/speed/public-dns/privacy

There's still a reasonable concern that Go proposals are ultimately decided upon largely---if not entirely---by Google employees (after taking into consideration external feedback), but evolving towards a multi-stakeholder approach (e.g., the Java Community Process) seems like an orthogonal issue to address.

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.

[Russ Cox]

On Wed, Mar 6, 2019 at 7:14 PM 'Matthew Dempsky' via golang-dev <golan...@googlegroups.com> wrote:

Perhaps it would help if the proposal outlined a privacy policy that Google commits to adhere to for operating the Go notary, and which can only be revised through the Go proposal process. For example, Google's Public DNS service provides a privacy policy: https://developers.google.com/speed/public-dns/privacy

I'd forgotten about this page, and it's a great model. Thanks for the reminder.

Russ

[Jakub Cajka]

I have a similar fears and worries here. First thing that popped on my mind(personally as a external observer) this is a move of Google to fully productize(data mine) a Go users community(for very limited payback), even if it is not a main driver it will make it irresistibly easy to do so(with google hosted notary).

IMHO this feature should be opt-in and the actual default instance of the notary should be run by trustworthy and independent 3rd party(ideally NPO) with clear and transparent privacy policy and on community accessible infrastructure(so a community members can actually participate in the maintenance and verify policies surrounding it, i.e. not like a current Go infra is run, only Google employees can contribute/touch it).

As I will carefully evaluate all the details of the final implementation, but from this first look I'm currently leaning to "patch out" or de-configure by default this feature in Fedora when/if it lands in upstream GC, to preserve the privacy of the users.

JC

PS: This is just my personal Fedora's GC maintainer take on this issue.

>
> On Mon, Mar 4, 2019, at 17:32, Russ Cox wrote:
> > Hi all,
> >
> > I wanted to let people here know about the proposal design we just
> > published: golang.org/design/25530-notary, for golang.org/issue/25530.
> > (We also mentioned this general idea in our blog post from back in
> > December, https://blog.golang.org/modules2019.)
> >
> > As usual, comments are welcome on the issue or, if you prefer not to
> > use the issue tracker, here in this thread.
> >
> > Thanks. Russ
> >
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "golang-dev" group. To unsubscribe from this group and stop
> > receiving emails from it, send an email to golang-
> > dev+uns...@googlegroups.com. For more options, visit
> > https://groups.google.com/d/optout.
>
> --
> Sam Whited
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to golang-dev+...@googlegroups.com.

[Russ Cox]

On Thu, Mar 7, 2019 at 5:52 AM Jakub Cajka <jca...@redhat.com> wrote:

As I will carefully evaluate all the details of the final implementation, but from this first look I'm currently leaning to "patch out" or de-configure by default this feature in Fedora when/if it lands in upstream GC, to preserve the privacy of the users.

Thanks for this feedback. I would only ask that you remember that publishing the design proposal is the first step in the public discussion, not the last. Please remember to check back in for the final result before you decide to apply local patches in Fedora.

Russ

[Jakub Cajka]

----- Original Message -----
> From: "Russ Cox" <r...@golang.org>
> To: "Jakub Cajka" <jca...@redhat.com>
> Cc: "Sam Whited" <s...@samwhited.com>, "rprichard via golang-dev" <golan...@googlegroups.com>,
> gol...@lists.fedoraproject.org
> Sent: Thursday, March 7, 2019 9:02:50 PM
> Subject: Re: [golang-dev] proposal: public module authentication with the Go notary
>

> On Thu, Mar 7, 2019 at 5:52 AM Jakub Cajka <jca...@redhat.com> wrote:
>
> > As I will carefully evaluate all the details of the final implementation,
> > but from this first look I'm currently leaning to "patch out" or
> > de-configure by default this feature in Fedora when/if it lands in upstream
> > GC, to preserve the privacy of the users.
> >
>
> Thanks for this feedback. I would only ask that you remember that

> publishing the design proposal is the *first* step in the public

> discussion, not the last. Please remember to check back in for the final
> result before you decide to apply local patches in Fedora.
>
> Russ
>

I will be observing the future iterations. I'm looking forward to them :).

JC

[Nicolas Mailhot]

Hi Jakub,

The notary part of Go modules, like the rest of the module
implementation, suffers from a lack of understanding of integration and
QA workflows, and a simplistic dev-centric worldview.

As designed, it will have to be patched out by every single entity
trying to perform large scale Go code integration and QA, starting with
the usual suspects, Linux distributions. Where that will leave upstream
Go, since its main target platform is Linux, I have no idea (I suspect
there will be some angst @Google about it).

I write this from the POW of the person that tries to update the Fedora
Linux Go tooling so we can ship Go with module mode on in august.
Whatever we do at that time will be mirrored by Fedora downstreams like
RHEL and Centos and all the other Linux variants that get their
inspiration Fedora-side.

It's not an armchair analysis, I already wrote way too much module-
oriented custom code because the upstream tooling is deficient, and I've
read my share of upstream issue reports.


CONTINUOUS TRANSPARENT INTERNET LOOKUPS ARE NOT ACCEPTABLE POST−DEV

When you integrate large volumes of code, targeting multiple hardware
architectures, it’s not acceptable to have the result of the Arm QA run
differ from the result of the x86_64 run just because it was scheduled
some minutes later on the QA farm, the internet state changed in the
meanwhile, and the Go tools decided to “help” you by looking at the
internet behind your back and updated transparently the state being
QAed. That's nothing new of Go specific, and that’s why our build system
disables network access in build containers after populating them with
the code to be built and tested, and already did so before someone
decided to invent Go.

But, that QA reproducibility constrain was not understood by Go
developers, and pretty much all the Go tools that were ported to module
mode, will attempt Internet accesses at the slightest occasion, and
change the project state based on those access results, with no ability
to control or disable it, and nasty failure modes if the internet access
fails.

The Go issue tracker is filling up with reports of people expressing
their incredulity (or more) after hitting a variation of this problem on
their QA systems.


QA−ED CODE IS NOT PRISTINE DEV CODE

No code is ever perfect.

When you integrate large volumes of code, targeting multiple hardware
architectures, you *will* hit problems missed by the original upstream
dev team. And you *will* have to patch them up, and you *will* have to
build the result without waiting for upstream to look at it because, in
production, the show must go on, some of those will eventually have
zero-day security implications, and the upstreaming lag is not
acceptable (assuming upstream is available and friendly, which is not
necessarily the case).

So no serious shop is ever going to build its information system from
pristine upstream dev code or pristine upstream Go modules. Pristine
upstream code building is a nice ideal, but in the real world, it’s
utopia.

If you don’t believe me, take your favorite app, and try to build a
system containing it from genuine unadultered upstream code (that means,
without relying on Fedora, Debian or whatever, since we all know they
are full of “nasty” patched code).

Again, that technical real-world-is-not-perfect was not understood by Go
developers. Sure they gave us the replace directive in Go module. But
that is opt-in. So it only works on a small scale, when fixing third
party code is an exception, and you only have a handful of projects that
use this third-party code.

On a large scale everyone will just preventively rename everything by
default just to retain the ability to perform fixes easily. You'll end
up with blanket replacement of every "upstream" module name with
"qaed/upstream" forks. And probably easier to rewrite all the imports to
point to qaed/upstream by default in the source code.

That will result in a huge mess, much larger than the current GOPATH
vendoring mess.

All because the design does not take into account the last mile QA
patching that occurs before putting code in production.

Blindly enforcing renaming rules because you don't understand or accept
the existence of QA middlemen produced things like Iceweasel. And I
don't think Mozilla or Debian were ever happy about it.


PUBLISHING QA-ED CODE CAN NOT DEPEND ON A THIRD PARTY

At least, not if you want the result to work in a free software
ecosystem. Freedom 4 “The freedom to distribute copies of your modified
versions to others” is not “The freedom to distribute copies of your
modified versions to others, but only if you tell this third party
first”.

And Google is free to choose not to target a free software ecosystem,
but that means all the entities that do target a free software ecosystem
(like Linux distributions) or have happily built their infrastructure on
free software products (like every single major cloud operator out
there), will have fork Go, or reorient their software investments
somewhere else, or some combination of those. And I don't think Google
wants that, or will be happy about it.

But, that's the situation the current notary design will create.
Because in trying to enforce a dev utopia, it is going to break the
workflows of a large proportion of the current Go ecosystem.


Any working system will need a way to declare trust in a local authority
(the shop QA team) or a set of authorities (the shop QA team, and
trusted third parties), and not have this trust rely on continuous
access to a server, regardless of who controls it (so detached digital
signatures, not direct server hash lookups).

And I’ll stop here before I write things I will regret later.

Regards,

--
Nicolas Mailhot

[Russ Cox]

On Fri, Mar 8, 2019 at 8:09 AM 'Nicolas Mailhot' via golang-dev <golan...@googlegroups.com> wrote:

The notary part of Go modules, like the rest of the module
implementation, suffers from a lack of understanding of integration and
QA workflows, and a simplistic dev-centric worldview.

This is not an auspicious beginning.

This mail came across as trying more to be antagonistic than constructive.

Even so, I really would like to understand your concerns

and either help address misunderstandings or adjust the

design appropriately.

CONTINUOUS TRANSPARENT INTERNET LOOKUPS ARE NOT ACCEPTABLE POST−DEV

[snip]

But, that QA reproducibility constrain was not understood by Go
developers, and pretty much all the Go tools that were ported to module
mode, will attempt Internet accesses at the slightest occasion, and
change the project state based on those access results, with no ability
to control or disable it, and nasty failure modes if the internet access
fails.

This really couldn't be farther from the truth.

You can turn off module changes using -mod=readonly in CI/CD systems.

We added it explicitly for that use case. Quoting golang.org/issue/26361 (July 2018):

CI systems need a way to test what's in the go.mod and fail if 

something is missing instead of looking to satisfy dependencies 

automatically. We meant for -getmode=local to mean this, but it 

really means something a little bit different. We should add a 

-getmode that does mean "you can't change go.mod". Maybe 

-getmode=noauto.

The eventual spelling was -mod=readonly, and it shipped in Go 1.11

with the initial module preview.

More generally, we understand the importance of reproducibility.

For example, see this post and this talk.

The Go issue tracker is filling up with reports of people expressing
their incredulity (or more) after hitting a variation of this problem on
their QA systems.

If you could point to specific examples, that would help me understand

your concerns a bit better. I did try to find some but failed:

$ issue cmd/go QA

26746 proposal: doc/install: define minimum supported VCS versions

14812 runtime: GC causes latency spikes

$ issue is:closed cmd/go QA

24301 cmd/go: add package version support to Go toolchain

22491 proposal: cmd/link: support BUILD_PATH_PREFIX_MAP for reproducible binaries when built under varying path

27160 x/build: set up AIX builder

12660 x/mobile: support Xcode 7

$ 

I've read this a couple times and found it a little hard to follow,

but I think what you are saying is that the go.sum and notary checks

are going to cause serious problems because Fedora and other

distributions want to create modified copies of module versions and

use them for building the software they ship. I don't see why that

would be the case.

I would have expected that if Fedora modified a library, they

would give it a different version number, so that for example modifying

v1.2.2 would produce v1.2.3-fedora.1 (it would be nice if SemVer

had a kind of 'post-release' syntax; maybe some day).

Go.sum and notary checks would only trigger at all if Fedora

were to take v1.2.2, modify it, and then try to pass it off as the

original v1.2.2, but that's indistinguishable from a man-in-the-middle

attack. Adding a suffix to the version string to indicate the presence

of Fedora-specific patches does not seem, at least to me, to be

anywhere near the level of renaming Firefox to IceWeasel, and it

avoids confusion about "which" v1.2.2 a build is using.

Furthermore, once you have created those versions and want to

build software using them, all it takes is to create a new module

with just a go.mod file (no source code) listing those versions as

requirements and then run builds referring to the unmodified original

top-level targets. Those targets' dependencies will be bumped forward

to the Fedora-patched versions listed in the go.mod file, without

any need to modify any of the client modules at all.

Or maybe I misunderstood what you were trying to say in this section.

PUBLISHING QA-ED CODE CAN NOT DEPEND ON A THIRD PARTY

At least, not if you want the result to work in a free software
ecosystem. Freedom 4 “The freedom to distribute copies of your modified
versions to others” is not “The freedom to distribute copies of your
modified versions to others, but only if you tell this third party
first”.

And Google is free to choose not to target a free software ecosystem,
but that means all the entities that do target a free software ecosystem
(like Linux distributions) or have happily built their infrastructure on
free software products (like every single major cloud operator out
there), will have fork Go, or reorient their software investments
somewhere else, or some combination of those. And I don't think Google
wants that, or will be happy about it.

But, that's the situation the current notary design will create.
Because in trying to enforce a dev utopia, it is going to break the
workflows of a large proportion of the current Go ecosystem.

I'm still not sure exactly what the dev utopia is.

Can you help me understand better?

Is the utopia the fact there would be a consistent meaning for

mymo...@v1.2.2 across all possible references and that any

modified source code would have to present a modified version number?

Is the inability to silently modify code without changing the version

number what's going to break the workflows of a large proportion of

the Go ecosystem? I'd like to understand that better if so.

 

Any working system will need a way to declare trust in a local authority
(the shop QA team) or a set of authorities (the shop QA team, and
trusted third parties), and not have this trust rely on continuous
access to a server, regardless of who controls it (so detached digital
signatures, not direct server hash lookups).

Your use of the word "continuous", both in this quoted text and in the

heading earlier, makes me think you might not realize that the notary

access is only needed when adding a new line to the go.sum file.

The steady state of a module that a module's go.mod file lists specific

versions of all its direct dependencies, and its go.sum file lists the 

cryptographic hashes of all its direct and indirect dependencies.

The only time the system falls out of the steady state is when a code

change adds an import of a new module; the next build of that changed

code reestablishes the steady state. And the 'go mod tidy' command's

job is exactly to establish that state for all possible builds in the module.

Once that state is established - once go.sum has a hash for each

dependency needed in a build - there is zero notary access during

any builds of that module. And as I mentioned above, a CI/CD system

or any other system can use -mod=readonly (either on the command line

or in the $GOFLAGS environment variable) to disable any attempt to

reestablish the steady state (that is, disable any attempt to modify go.mod

or go.sum) except by an explicit 'go get' or 'go mod tidy' command.

So the notary access is occasional, predictable, and controllable.

It is not continuous.

Put a different way, the notary is how the go command populates go.sum

by default. If a QA system needs to populate it a different way, that's fine:

one option is 'go get -insecure', another option is GONOVERIFY=*, and

a third option is to just write the desired go.sum directly:

it's a simple, line-oriented text file. Even if a QA team insists on modifying

mymo...@v1.2.2 without giving it a new version, if they also update the

go.sum files of modules using mymo...@v1.2.2, the builds will succeed,

all without any notary access.

Stepping back from the specific details, again I am more than happy

to understand everyone's concerns with the notary design and work to

address them. We've been having a productive conversation on the

GitHub issue, and I hope we can have a productive conversation here too.

Best,

Russ

[Nicolas Mailhot]

Le vendredi 08 mars 2019 à 10:28 -0500, Russ Cox a écrit :
> On Fri, Mar 8, 2019 at 8:09 AM 'Nicolas Mailhot' via golang-dev <
> golan...@googlegroups.com> wrote:
> > The notary part of Go modules, like the rest of the module
> > implementation, suffers from a lack of understanding of integration
> > and QA workflows, and a simplistic dev-centric worldview.
>
> This is not an auspicious beginning.
> This mail came across as trying more to be antagonistic than
> constructive

I'm sorry, my level of English is not sufficient to convey information
you want to ignore, and dress it up so you feel good about it. I've
tried to stick to plain facts. If you object to plain facts I can't do
anything about it.

> Even so, I really would like to understand your concerns
> and either help address misunderstandings or adjust the
> design appropriately.
>
> > CONTINUOUS TRANSPARENT INTERNET LOOKUPS ARE NOT ACCEPTABLE POST−DEV
> >
> > [snip]
> >
> > But, that QA reproducibility constrain was not understood by Go
> > developers, and pretty much all the Go tools that were ported to
> > module
> > mode, will attempt Internet accesses at the slightest occasion, and
> > change the project state based on those access results, with no
> > ability
> > to control or disable it, and nasty failure modes if the internet
> > access
> > fails.
>
> This really couldn't be farther from the truth.
>
> You can turn off module changes using -mod=readonly in CI/CD systems.

It would be nice if it where true. Unfortunately even a simple command
like tell me what you know about the local code tree
go list -json -mod=readonly ./...

will abort if the CI/CD system cuts network access to make sure builds
do not depend on Internet state.

That's what the nasty failure modes are about. If the go command is not
sure about anything, it will “solve” things by trying to download new
bits from the Internet. If the Internet is not available, it will abort
violently, not degrade gracefully and work with what it has.

The no-modification no-internet CI/CD constrain was not taken into
account into the original design. It was bolted on later, and the
bolting is imperfect. Just take a vacation in some paradisiac place with
no internet access, and see how far you can do go coding in module mode.
*That* will replicate our CI/CD constrains (paradisiac place as a bonus,
we don't have those in our build farms – see I'm trying not to be
antagonistic).

> > QA−ED CODE IS NOT PRISTINE DEV CODE
> >

> I've read this a couple times and found it a little hard to follow,
> but I think what you are saying is that the go.sum and notary checks
> are going to cause serious problems because Fedora and other
> distributions want to create modified copies of module versions and
> use them for building the software they ship. I don't see why that
> would be the case.
>
> I would have expected that if Fedora modified a library, they
> would give it a different version number, so that for example
> modifying v1.2.2 would produce v1.2.3-fedora.1

And that's not the case neither for Fedora, nor RHEL, nor Debian, nor
pretty much any large scale integrator, because when you integrate
masses of third-party code you will eventually hit bugs in pretty much
every component, so having to run patched code at every layer is the
norm not the exception.

It would be terribly inconvenient to have to rename or renumber or
replace everything, and then have to convince all the other components
to use the renamed or renumbered versions of the components they depend
on. It would be even more inconvenient to do it just in time and
continuously flip flop between upstream and local names and numbers.

That would actually add friction to merging back and returning to
pristine upstream code, because merging back would now cost a
rename/renumber, instead of being a net win. Patching is forced on you
(you hit a bug in upstream code). Upstreaming is a deliberate virtous
choice (you've already fixed your problem locally). If you make
upstreaming more expensive, people just stop doing it.

There are legal ways to force distributions to do the wasteful
renaming/renumbering dance. They will take it as an hostile imposition.
Where do you think Iceweasel came from? No love lost here.

> Go.sum and notary checks would only trigger at all if Fedora
> were to take v1.2.2, modify it, and then try to pass it off as the
> original v1.2.2

That's the standard Linux distribution workflow.

> but that's indistinguishable from a man-in-the-middle attack.

The whole purpose of a distribution, is to be a giant middleman, freely
chosen by the end user, between the code released upstream, and this end
user¹. All middlemen are not hostile. You need to understand that or
your system will not work.

In the real world you have friendly middlemen. Sometimes layers of them
(it is quite common for local organisations to add another level of
changes over distro changes). Sometimes those middlemen make changes.
Sometimes they just check things for nastiness.

A correct trust test is not “it exists exactly this way on the
Internet”. What kind of assurance is that? If I dump malware on a public
URL, it’s trusted as long as no AV touches it mid-flight?

A correct trust test is “has the state been signed by an entity the user
trusts”.

So, digital signatures. A way to configure the public keys of trusted
third parties. And, if no signature by a trusted third party is found
locally, at last resort, and only if the user asked for it, look on the
internet if a notary signature exists. Of course, that's a piss-poor
level or trust, but better that than nothing.

> Furthermore, once you have created those versions and want to
> build software using them, all it takes is to create a new module
> with just a go.mod file (no source code) listing those versions as
> requirements and then run builds referring to the unmodified original
> top-level targets. Those targets' dependencies will be bumped forward
> to the Fedora-patched versions listed in the go.mod file, without
> any need to modify any of the client modules at all.

# dnf repoquery --disablerepo=* --enablerepo=rawhide \
--whatprovides 'golang(*)' |wc -l
766

(here Debian people are laughing at me because we are lagging behind
them on the Go integration front)

We have several hundreds more Go components in the integration queue
waiting for review.

Some of those will end up as several Go modules.

And you want us to renumber all this, and patch all the other module
files that use those with the new numbers, and redo it all every time
something changes, all year round, just because the go command can’t
accept that the code state Fedora builds from, may differ from the one
observed on the Internet?

Really?

REALLY ?

Do you really REALLY think any sane integrator will play this dance long
instead of patching out notary checks from its go command, and be done
with it?

> Is the utopia the fact there would be a consistent meaning for
> mymo...@v1.2.2 across all possible references and that any
> modified source code would have to present a modified version number?

The utopia is thinking that everything is released in a perfect state by
upstream, so changes downstream need not happen, and therefore any
downstream change is necessarily an attempt to inject malware by Mr
Nasty.

> Is the inability to silently modify code without changing the version
> number what's going to break the workflows of a large proportion of
> the Go ecosystem? I'd like to understand that better if so.

Distributions distinguish between the upstream version number and their
own build id. So you have

mymo...@v1.2.2 release 1
mymo...@v1.2.2 release 2

and so on. Each release id is a separate build that can involve
different patches (some release ids are more complex than a single
number).

Only a single release can exist within the distribution at a given time,
so makefiles, go mod files and so on need not differentiate between
release X and release Y, they will only see one of those at any time and
they better work with it because they won't be given or allowed any
other.

System artefacts (libraries, binaries, go modules) are built from plain
upstream source code, as downloaded by distribution processes directly
from uptream VCS or website, after preparation (removal of problem
parts, patching, etc) in pristine containers, isolated from the
Internet, and populated only with a minimal distribution installation
and the content of the distribution components necessary for their
build.

So to build github.com/my/thing version x.y.z that declares in its
module file

module github.com/my/thing

require (
github.com/some/dependency v1.2.3
github.com/another/dependency/v4 v4.0.0
)

A. We will populate a clean container with

1. a minimal system
2. the go compiler
3. the most recent system component that provides the go module
github.com/some/dependency ≥ 1.2.3 and < 2
and all its dependencies
(go module as produced by our own built of this other component, not
as downloaded by go get from the internet)
4. the most recent system component that provides the go module
github.com/another/dependency/v4 ≥ 4.0.0 and < 5
and all its dependencies (ditto)
5. the source code for github.com/my/thing x.y.z as downloaded and
checked from the internet by a human (not by go get), and then sealed

B. We will cut internet access of this container

C. We will prepare the github.com/my/thing x.y.z source (remove problem
parts, patch bugs, remove vendored code, remove at least local replaces
in go.mod)

I'm fairly certain we will nuke go.sum because we want to build from our
reference state against our reference versions, not reproduce whatever
upstream tested on ubuntu or windows.

D. We will point the go compiler to the local module files (via GOPROXY)

E. We will ask a go command to transform our github.com/my/thing x.y.z
source state in a module files that can be used by other Go code (zip
ziphash info mod files) once deployed in our GOPROXY directory. It would
be nice if go mod pack existed upstream otherwise we will write and use
our own utility.

That will involve sanitizing the upstream mod file (remove indirect
requires, remove local replaces, not sure about non-local replaces yes,
by gut feeling is that we should remove them too and force the fixing of
imports in source preparation, but I'm not sure yet)

For various technical reasons it's not possible to deploy directly in
GOPROXY at this stage, so at this point you have the prepared
github.com/my/thing x.y.z source code in unpacked state in a directory,
the module it needs in GOPROXY, and a packed version of
github.com/my/thing x.y.z in a staging directory.

D. we will ask the go compiler to build the various binaries that need
to be produced from github.com/my/thing version x.y.z

E. we will pack the result files (binaries and go modules) in system
components, so they can be deployed at need. The component containing
github.com/my/thing x.y.z will record a need for
github.com/some/dependency ≥ 1.2.3 and < 2 and
github.com/another/dependency/v4 ≥ 4.0.0 and < 5

So the internet access won't exist when you think it exists, the files
arrive on disk without go get, and the state they are available on is
not the upstream state.

Having worked in a proprietary integration shop before, and working with
proprietary integrators today, the workflows are no so different, except
proprietary shops tend to be a lot laxer, allowing internet access when
they should not, and either processing code as downloaded from the
internet, with no checks, or forking it to death, without trying to re-
attach to upstream.

Best regards,

¹ The function of this middleman is beat upstream code into shape so the
user does not have to do it itself. Beating into shape does involve
large-scale changing of upstream code.

Users choose the distribution system because their alternatives are:
1. to hope every single upstream they use never makes a mistake
requiring a last-mile fix (fat chance on that)
2. wasting their time doing the last-mile fixing themselves, instead of
delegating this function to distributors.

It's all free software. An unhappy distribution user can take the source
code he needs and get it integrated somewhere else. No strings attached.

--
Nicolas Mailhot

[Ian Lance Taylor]

On Fri, Mar 8, 2019 at 4:52 PM 'Nicolas Mailhot' via golang-dev

<golan...@googlegroups.com> wrote:
>
> Le vendredi 08 mars 2019 à 10:28 -0500, Russ Cox a écrit :
> > On Fri, Mar 8, 2019 at 8:09 AM 'Nicolas Mailhot' via golang-dev <
> > golan...@googlegroups.com> wrote:
> > > The notary part of Go modules, like the rest of the module
> > > implementation, suffers from a lack of understanding of integration
> > > and QA workflows, and a simplistic dev-centric worldview.
> >
> > This is not an auspicious beginning.
> > This mail came across as trying more to be antagonistic than
> > constructive
>
> I'm sorry, my level of English is not sufficient to convey information
> you want to ignore, and dress it up so you feel good about it. I've
> tried to stick to plain facts. If you object to plain facts I can't do
> anything about it.

You say that this is information that we want to ignore. Why leads
you to say such a thing? What makes you think that we want to ignore
it? We have never claimed to know everything. Adding modules to the
go tool is a work in progress.

Problems can be identified and solved but a necessary step is that
everyone treat each other with respect. When you accuse others of
wanting to ignore you, that makes your message seem like an attack,
and indeed makes people more likely to ignore it. Your message would
be a lot easier to read if you did just stick to plain facts.

Thanks.

Ian

[Matthew Dempsky]

On Fri, Mar 8, 2019 at 4:52 PM 'Nicolas Mailhot' via golang-dev <golan...@googlegroups.com> wrote:

It would be nice if it where true. Unfortunately even a simple command
like tell me what you know about the local code tree
  go list -json -mod=readonly ./...

will abort if the CI/CD system cuts network access to make sure builds
do not depend on Internet state.

The behavior you described sounds like an issue to me. However, I wasn't able to easily reproduce it locally: I tried running "strace -f go list -json -mod=readonly ./... |& grep connect" under a few scenarios (e.g., missing go.sum, or incomplete go.mod file), and none of them seemed to result in network access.

Can you provide more detailed reproduction steps?

The no-modification no-internet CI/CD constrain was not taken into
account into the original design. It was bolted on later, and the
bolting is imperfect.

I don't think this is true. Google's internal build system has used this same "no-modification no-internet" constraint for around a decade:

http://google-engtools.blogspot.com/2011/09/build-in-cloud-distributing-build-steps.html

https://mike-bland.com/2012/10/01/tools.html#blaze-forge-srcfs-objfs

Supporting this usage pattern has been and remains very important to Go.

> I would have expected that if Fedora modified a library, they
> would give it a different version number, so that for example
> modifying v1.2.2 would produce v1.2.3-fedora.1

And that's not the case neither for Fedora, nor RHEL, nor Debian, nor
pretty much any large scale integrator,

When I run "gcc --version" on my work Debian machine, I see:

    gcc (Debian 7.3.0-5) 7.3.0

If Debian is okay naming their version of GCC "Debian 7.3.0-5", I'd think they're okay releasing a patched Go package as v1.2.3-debian.1.

You even seem to suggest this later when talking about "mymo...@v1.2.2 release 1" and "mymo...@v1.2.2 release 2." Russ is just talking about a different way of encoding those version strings.

because when you integrate
masses of third-party code you will eventually hit bugs in pretty much
every component, so having to run patched code at every layer is the
norm not the exception.

The need to make changes to open source packages is familiar to Google. E.g., Google's internal build system incorporates a lot of open source packages, and many need to be modified to accommodate Google's internal development idiosyncracies. Even the Go toolchain and standard library themselves are patched internally.

I think it would help if you could highlight the specific technical hurdles you're running into trying to bundle Go packages into RPMs. I expect the mechanisms are in place to do what you need, but I can believe the tooling could use improvements to handle Linux-distribution-scale integration efforts better, and I think the Go project is interested in supporting those efforts.

And you want us to renumber all this, and patch all the other module
files that use those with the new numbers, and redo it all every time
something changes, all year round, just because the go command can’t
accept that the code state Fedora builds from, may differ from the one
observed on the Internet?

Really?

REALLY ?

Do you really REALLY think any sane integrator will play this dance long
instead of patching out notary checks from its go command, and be done
with it?

I would expect the amount of manual integration work needed would scale with the amount of local changes required, not with the amount of dependencies. E.g., if you have to modify a core Go module to work better on Fedora, I would think you make that one change, and your build system and tooling would handle automatically rebuilding dependencies as appropriate. I wouldn't expect you to have to manually renumber/rename every downstream dependency.

If you're finding that's not the case, please share the issues you're running into so they can be discussed concretely and addressed.

The utopia is thinking that everything is released in a perfect state by
upstream, so changes downstream need not happen,

As pointed out above, Google and Go's build systems are not built for this utopia, but for the real world where downstream changes are needed.

Finally, you included a detailed description of your package build system (which sounds very similar to Google's internal build system), but didn't seem to highlight how the Go module system or proposed Go notary system cause problems.

E.g., you mention that in step A.3 that you download and install the Fedora version of any package dependencies. I would think these packages can contain any extra metadata for step C to be able to rename/renumber the current package (if even necessary) without Internet access.

[Benny Siegert]

Last time this discussion came up in person, someone from the Go team
suggested replacing all the go.mod files with new ones that are
entirely based on package metadata -- i.e. declared dependencies,
their installed location etc.

This seems like a reasonable solution, and it would be a pity if the
notary made this impossible.

> --
> You received this message because you are subscribed to the Google Groups "golang-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
Benny

[Nicolas Mailhot]

Le vendredi 08 mars 2019 à 18:01 -0800, Matthew Dempsky a écrit :
> On Fri, Mar 8, 2019 at 4:52 PM 'Nicolas Mailhot' via golang-dev <
> golan...@googlegroups.com> wrote:
> > It would be nice if it where true. Unfortunately even a simple
> > command
> > like tell me what you know about the local code tree
> > go list -json -mod=readonly ./...
> >
> > will abort if the CI/CD system cuts network access to make sure
> > builds
> > do not depend on Internet state.
>
> The behavior you described sounds like an issue to me. However, I
> wasn't able to easily reproduce it locally: I tried running "strace -f
> go list -json -mod=readonly ./... |& grep connect" under a few
> scenarios (e.g., missing go.sum, or incomplete go.mod file), and none
> of them seemed to result in network access.
>
> Can you provide more detailed reproduction steps?

That's because upstream go workflows (from a distribution POW) cheat by
downloading blindly things from the internet before they have been
checked, filling the go cache silently with them, and grandfathering
directly downloaded third party code in a build as original source code.

We do not allow that. We make the promise to our users that we only make
available things, that we created from direct upstream source code, and
third party things, that already passed distribution QA checks in their
own separate build.

And that, only if this those party things were identified as necessarily
for the build¹ and installed (for go modules) in the GOPROXY directory
used for this specific build via system components.

At the very very first steps of a build recipe when we ask the copy of
the codebase being built "what are you, what are you composed of, what
to you need for building" there is literally only the codebase being
inspected and the go compiler available and GOPROXY is empty. Because we
need the answer to this question, to translate it into system component
ids, so the build container can be populated with those system
components, making the module files available inside GOPROXY.

But in module mode, go will assume everything is available all the time,
and if it is not available it can be downloaded directly from the
internet, so instead of answering the question from direct first-level
elements (the codebase and nothing else) it will try to work
transitively and abort because second level code is not available yet.

> > > I would have expected that if Fedora modified a library, they
> > > would give it a different version number, so that for example
> > > modifying v1.2.2 would produce v1.2.3-fedora.1
> >
> > And that's not the case neither for Fedora, nor RHEL, nor Debian,
> > nor
> > pretty much any large scale integrator,
>
> When I run "gcc --version" on my work Debian machine, I see:
>
> gcc (Debian 7.3.0-5) 7.3.0

And when you run
$ ls -l /usr/lib

you won't see any libfoo.so.x.y.z-debian or libfoo.so.x.y.z-5

Because the only libfoo.so.x.y.z allowed on system at any given time is
the one built from (debian-patched) foo x.y.z source code, no matter if
it's release 1 2 3 or 5 of the Debian build, no matter the level of
Debian patching and massaging each build carries.

So giving the underlying build command the "you're building 7.3.0 build
release 5 debian" info so it appears in version info of the binary is OK
and encouraged (we'll even pass it as ldflags info to make sure it is
recorded in reach binary).

But having the underlying build command try to discriminate between
different build releases, or the build releases or the third party
artefacts brought in the build root (like would happen if the build
release was exposed to the golang semver resolution engine) is not OK.

The underlying build command has not business doing any processing on
release info, it's system component metadata, the system component
engine will give it the release it needs to work with at any given time,
if the build command tries to second guess the system component engine
things start breaking fast.

> If Debian is okay naming their version of GCC "Debian 7.3.0-5", I'd
> think they're okay releasing a patched Go package as v1.2.3-debian.1.
>
> You even seem to suggest this later when talking about "
> mymo...@v1.2.2 release 1" and "mymo...@v1.2.2 release 2." Russ is
> just talking about a different way of encoding those version strings.
>
> > because when you integrate
> > masses of third-party code you will eventually hit bugs in pretty
> > much
> > every component, so having to run patched code at every layer is the
> > norm not the exception.
>
> The need to make changes to open source packages is familiar to
> Google. E.g., Google's internal build system incorporates a lot of
> open source packages, and many need to be modified to accommodate
> Google's internal development idiosyncracies. Even the Go toolchain
> and standard library themselves are patched internally.
>
> I think it would help if you could highlight the specific technical
> hurdles you're running into trying to bundle Go packages into RPMs. I
> expect the mechanisms are in place to do what you need, but I can
> believe the tooling could use improvements to handle Linux-
> distribution-scale integration efforts better, and I think the Go
> project is interested in supporting those efforts.

Ok, thanks a lot, that would make things a lot simpler.

I'll list here the various commands we need implemented to plug go
modules in our system. Implemented means by upstream go or ourselves,
though we obviously would prefer it to be upstream go side, because that
ensures they are not broken in the future by upstream go tooling
changes, that other distros use the same commands as us, that we can
limit divergence and share the QA and fixing burden.

I believe that the commands needed apt-side for Debian and Ubuntu, are
pretty much the same, because apt/deb and dnf/yum/rpm have been on a
convergent evolution track for a long time, and at this point their
high-level architecture is the same, only implementation details differ
(we customarily scrouge Debian patches and fixes and they scrouge ours,
and we metoo each other's problem reports in upstream issue trackers).

Like I already wrote, our build process starts with a bare raw unpacked
copy of upstream sources in a specific directory. This copy will have
already been modified to remove things we do not like and patch out
problems². Obviously the patching out is an iterative process that's why
each release of a build may carry slightly different patches.

This copy typically does not include any VCS info³. Since go expects to
find the module version information there, and does not record it in the
project go.mod like other languages, life already sucks for us because
we need to carry a variable with this info and pass it manually to later
commands.


[available resources:
— minimal system,
— go compiler,
— empty system GOPROXY,
— prepared unpacked project sources in a specific directory]

A. We need a command to identify all the go module trees present in the
source directory. A variation of
$ find . -type f -name go.mod
would probably work, but that lacks all the sanity checks the upstream
go may want to apply now or in the future, to make sure a "go.mod" file
is a legitimate go module descriptor.


[available resources:
— minimal system,
— go compiler,
— empty system GOPROXY,
— prepared unpacked project sources in a specific directory]

B. We need another command (or the same) that lists:
1. the go module name of each of those trees,
2. their consolidated first-level requirements (not the list of their
individual first-level requirements)

Probably not both at once, or if both in a structured format we can
reprocess. The most convenient command is
<command> <directory> --provides → one module name per line
<command> <directory> --requires → one constrain
(module + associated version constrain) per line

The --requires command needs to filter out the modules names already
present in the source tree, because in a multi-module X project that
contains X/foo and X/bar modules, we want to build X/foo against the
local version of X/bar, not some other one

(and we will have some auditing work on the --provides output to make
sure every provides line actually belongs to the project being built and
is not a copy of someone else's code)

I lack the necessary perspective today to say how replaces should be
handled.

Local replaces clearly need ignoring, if some code needs the X third-
party module, we want it to use our curated X module, not some project-
local directory containing a fork of X.

Non-local replaces? That's less clear to me.

If a non local replace of B by C in module A means that every downstream
user of module A will then use C instead of B when processing A code,
that's fine with us, the fact B exists is an internal go compiler
information, we only need to know C is needed in the A build container
and in any container that uses the A module.

But, if a non local replace of B by C in module A only affects direct A
builds, and indirect A builds will still use B, we'll just cry and yell
and curse in a quiet corner, and then either remove the replacing in the
A module file, or make it transitive by patching our version of A code
imports statements to use C everywhere instead of A.

The result of the listing command will then be translated in rpm system
identifiers, and used to populate the build GOPROXY.

Translation basically means:
1. applying some form of namespacing:
“I need the go module named X” becomes “I need golang-module(X)”
(sadly the golang() namespace is already taken by GOPATH sources, and
can not be shared because go keeps GOPATH and module mode separate)
2. translating the golang version constrain syntax in rpm version
constrain syntax. That's pretty easy for semver:
“I need X...@v1.2.3” becomes
“I need golang-module(X) ≥ 1.2.3 and < 2”

For non-semver constrains like a specific hash-identified snapshot
that's less nice, we do not allow hashes as valid version IDs rpm
side (because hash ordering can not be deduced from the hashes
themselves) so anything that absolutely require a specific snapshot
of X will be translated into “I need golang-module(X)(commit=hash)”
and golang-module(X)(commit=hash) is a separate ID from
golang-module(X)

Because dependency cycles exist in the real-world, and because we are
extra-careful to build only from things we already vetted, there will be
cases where we won't be able to satisfy upstream project requirements

If A needs B that needs C that needs A we will have build one of A B C
in reduced more at one point to break the cycle (we call that
boostrapping). No idea if it's better to inform the listing command that
one of the module needs it reports will be ignored, or if it's better to
ignore it silently.


[available resources:
— minimal system,
— go compiler,
— system GOPROXY populated with distro-produced go modules corresponding
to identified project requirements,
— prepared unpacked project sources in a specific director]

C. we need a go mod pack command that creates packed module files from
the unpacked project sources into a staging GOPROXY directory.

We can pass it the project version as argument (since it can not read it
in the upstream go.mod files).

Since VCS info is not available, the time component of the info file can
not be populated with it. And, we definitely do not want the command to
record the current time (because then the file content would depend on
when it was built, and we have processes that check that arch-agnostic
files files produced by different architecture builders are bit-for-bit
identical, and we have security auditors that replay builds later in
their own system and are alarmed when the replay produces different
results). And, frankly, the time info in VCSes is messy and unreliable,
and mtimes can not be used (the source preparation process can apply
patches, and that will change mtimes, even if you apply the very same
patch in two different runs).

Thus, we'd prefer this time info not to exist in our own info files, or
be set to zero, or, if go really needs it, to pass it as parameter to go
mod pack, (but that's more manual work our side to decide what to pass).

This command needs to operate in bulk on all the module trees previously
identified (so if it found 3 module trees, create 3 packed modules in
the staging GOPROXY), or a subset (explicitly provided module list).

This command needs to output the list of created module files (the zip
ziphash info mod files). The list file is a special case, we would
consider its file enveloppe as belonging to each of the versions of a
module, and the file content belonging to no one in particular (ghost
files in rpm tech). So it can be listed or not either way we will apply
special processing to it. Probably, better to list it for cleanliness.

The "files created" list can be either outputted to stdout (in which
case it outputs nothing else to stdout) or to a list file specified as
argument. No particular preference, we can handle both.

This list is used to ventilate the produced files in specific system
components: upstream project A may contain sources for A/B/C A/B/D A/E
modules, and we can choose to put all of those in a single system
component, or separate in two A/B/C+A/B/D and A/E, or go full granular
with three system components. The level of splitting depends on the
consequences in the system component dependency graph.

This go mod pack will need to be more discriminating than just “bulk
pack every file under the go.mod root directory, .gitignore included”
like I see go is doing right now.

If a project includes font files (for example the go font), we'll want
to expose those in /usr/share/fonts so they are not restricted to the go
compiler. If a project includes documentation files we will deploy them
in /usr/share/doc not in the module zip file. And so on (protobuf files
come to the mind)

So, typically, we'd want go mod pack to pack *only* the module tree Go
source code (things that can be used by the go compiler) ideally only
the source code that can be used on our platform (GOOS=linux), and vet
the packing of any other file.

So a basic go mod pack invocation only packs go source files and
testdata.

You have an info or dry run mode that says "default packing ignored/will
ignore all those files in the origin tree"

You can pass lists of regexes to the go mod pack command to either
include other non source code files, or to exclude already selected
files from the packing (--include regex and --exclude regex flags).

The regexes are obviously very module specific, depend on the module
resource needs and the distribution packaging policies. The generic go
tooling need not worry about those policies, just apply the produced
regexes.

Ideally, we could pass the go mod pack a --without module[@version]
argument, that causes it to not pack any the code that requires
module[@version]. That's necessary while bootstrapping and to cull very
expensive unit or integration tests that require hundreds of third-party
modules to run. Otherwise we can approximate it via exclude (but that
will require more Fedora manual work to identify the required excludes).

Please understand, we are not doing all this filtering and culling in
opposition to upstreams Go projects. We have no special wish to deviate
from them. Our ideal world is usptream code releases that can be used
directly as-is without any patching or filtering.

But, a lot of upstream projects lack discipline.

They will include files they have no legal right to distribute (and we
will remove those before step A).

They will include files that can be distributed, but not modified (and
entities like Fedora and Debian will think very hard if they really want
to relay those, because they are contrary to our free software policies,
and our users like to know that everything we ship can be modified
without legal problems).

They will reference third-party modules with incompatible (or completely
missing) licensing.

They will ship integration code that makes no sense outside of their own
information system.

They will ship for years broken and bitrotten project/test/example
code.

For them, continuing to include this code within their project is free.
their go get will happily download hundreds of unveted and unnecessary
Go modules from the internet. No one is likely to sue them for small
legal mistakes.

But shipping this code is *not* free for us. A distribution is big
enough it can be sued. Any failing unit test costs human time to check
it is an harmless failure. Any module dep pulled by this unnecessary
code is yet another software project that needs to be checked and
audited and integrated and maintained at the system component level
before it is available in the CI system.

So, we need to aggressively cull anything not necessary, to keep the our
integration costs and risks down.

And we really really would appreciate if project unit tests (+ testdata)
and project production code were split in separate zip files, with
separate build requirements, so the unit test dependency costs were
optional (we'd typically pay the project A unit test costs when
integrating project A, not when integrating something that uses project
A).

For all those reasons, the go modules files produced at this stage will
definitely *not* match the hashes produced by the notary, even if we do
not change a sigle line of go code.



[available resources:
– minimal system,
– go compiler,
– system GOPROXY populated with distro-produced go modules corresponding
to identified project requirements
– prepared unpacked project sources in a specific directory,
— staging GOPROXY containing candidate go modules corresponding to the
built project]

D. we need a command to build the project binaries from the files in the
system GOPROXY and the unpacked project sources,
OR from the system GOPROXY and the staging GOPROXY contents

(ideally, the second option, to make sure the candidate module files in
the staging directory are complete)



After this step we've hopefully finished producing files from the
prepared unpacked project sources. At this stage rpm deploys every
produced candidate file in a location that mirrors the target deployment
paths, under a specific prefix

So:
– a file in /usr/bin is a binary installed from existing system
components
– a file in /prefix/usr/bin is a candidate binary produced from the
ongoing build
— a file in GOPROXY is a Go module installed from existing system
components
— a file in /prefix/GOPROXY is a candidate Go module file produced from
the ongoing build


[available resources:
– minimal system,
– go compiler,
– system GOPROXY populated with distro-produced go modules corresponding
to identified project requirements
– prepared unpacked project sources in a specific directory,
— candidate tree under /prefix containing all the files that will end up
in new system modules. Some of those will replace existing files in /.
]

E. we need a command to run the unit tests of every produced go proxy
module (in the staging GOPROXY). Any failure (returning error code)
aborts the build process.

We'd really like this command to only take into account the candidate
tree, and the existing / tree (for files that do not belong to the
target system components).


[available resources:
– minimal system,
– go compiler,
– system GOPROXY populated with distro-produced go modules corresponding
to identified project requirements
– prepared unpacked project sources in a specific directory,
— candidate tree under /prefix containing all the files that will end up
in new system modules. Some of those will replace existing files in /.
]

F. At this point rpm starts ventilating the files contained in the
staging tree in new system components. Therefore, it needs to compute
the metadata of each of those system components

So, we have a new run of "what are you and what do you need" command,
this time on every mod/zip/zipphash/info fileset in /prefix/GOPROXY,
instead of the content of the prepared unpacked project sources. So,
probably some variation of the command in B.

Practically, those inspection runs can not be triggered by complex
objects like filesets, only individual files, so we'll probably trigger
them on the mod files, and assume that is the mod file is present, the
rest is likely to be here too.

rpm keeps track or where each file will end up, and attributes the
result of the query to the corresponding system component (it does not
tell the queried fileset where it will end up)


So we now have brand-new shiny system components that contain clean
audited GOPROXY module filesets, and declare they provide
golang-module(foo) = x.y.z, and that they need
golang-module(bar) ≥ a.b.c and < a+1

Installation of one of those system components will make the module
fileset exist in GOPROXY

But, that is not sufficient for go, because of the list files.

So, we need a final command, that rpm can invoque when it adds or
removes files in GOPROXY, to compute the list version indexes.

It could be approximated by a simple stupid shell script, but the go mod
cache code upstream performs more sanity checks than that, and we'd like
those sanity checks to be replicated in the "update list files" command.

And, that's pretty much all. We create clean new module files, we put
them in system components, installation of those components causes the
module files to exist in GOPROXY, and we'd like for the compiler to make
use of those module files without bothering us with notary checks (that
will fail pretty much all the time,due to how the whole integration
process is structured). The system components are digitally signed but
the build process has no access to the signing key (that is done on a
separate system with an HSM for security reasons) so the build process
can not generate detached signature files.

> I would expect the amount of manual integration work needed would
> scale with the amount of local changes required, not with the amount
> of dependencies. E.g., if you have to modify a core Go module to work
> better on Fedora, I would think you make that one change, and your
> build system and tooling would handle automatically rebuilding
> dependencies as appropriate.

Yes that's how it works, but only because the underlying build commands
are not supposed to discriminate between fedora build ids (releases)

We build
mymo...@v1.2.2 with some level of fixing of patching (release 1). That
creates a mod/zip/zipphash/info fileset

We build all the things that need mymo...@v1.2.2

Some time later a new issue causes us to adjust the fixing done for
mymo...@v1.2.2

So we build mymo...@v1.2.2 release 2 with the new fixes. For a lot of
languages that would be enough, and produce a new dll, transparently
used by all downstream users.

With Go static building that creates a new
mod/zip/zipphash/info fileset for mymo...@v1.2.2 (with new hashes)

and we have to perform the additional step of rebuild everything that
uses mymo...@v1.2.2, recursively.

We don't want the compiler to complain about the hash change or to
compare it to some internet notary.

We can certainly pass the release id info to the pack command in C so it
is recorded somewhere in the produced mod/zip/zipphash/info fileset and
each release fileset is unique.

That is, as long as it is only recorded for information purposes, and
the go command does not try to treat the result as different from
mymo...@v1.2.2, to discriminate between releases, or request a specific
release.

Best regards,

¹ Because first, sometimes the on-disk representation of two components
will clash. That's not supposed to happen with Go modules but our system
is not Go specific, so it was not specced around Go module
particularities, and who knows at this point if the module no-clash
design will be robust WRT the weird things upstreams like to invent in
the real word.

And second, we want to be sure that what we know about component
relationships is the truth, so security teams can rely on the system
component dependency graph when analysing incidents. The best way to
make sure it is the truth is to disallow anything not identified in the
CI container.

² (vendored copies of third party code, things we can not distribute
legaly like for example arial.ttf when upstream needs a "free" font and
does not understand copyright law, etc).

³ For historical reasons, because out CI infrastructure design antedates
modern VCSes, and because we absolutely do not want the build commands
to start pulling things from VCS history that do not match the version
we're attempting to build

--
Nicolas Mailhot

[Ian Denhardt]

Quoting 'Matthew Dempsky' via golang-dev (2019-03-08 21:01:32)

> Can you provide more detailed reproduction steps?

Here's a reproducer:

> $ go mod init example.org/my-project
> go: creating new go.mod: module example.org/my-project
> $ cat >> main.go << "EOF"
> package main
> import _ "example.com/other-module"
> func main() {}
> EOF
> $ go list -mod=readonly ./...
> build example.org/my-project: cannot load example.com/other-module:
> import lookup disabled by -mod=readonly

I think we'd both mis-understood Nicolas's complaint the first time --
it's not that it still tries to access the network with -mod=readonly,
but that even for basic commands for which a read-only mod file/no
network access really *shouldn't* be a problem, it will still just bomb
out.

I'm not familiar with the code, but this sounds like early in the common
program logic there's a check as to whether any of the module data needs
updating, then abort if it can't be done -- before even looking at what
the user has asked us to do.

I agree this is kinda sad; it seems like it would be better if the go
tool still tried to fulfill the user's request to the extent possible
when -mod=readonly is set.

-Ian

[Manlio Perillo]

On Friday, March 8, 2019 at 4:28:27 PM UTC+1, rsc wrote:

On Fri, Mar 8, 2019 at 8:09 AM 'Nicolas Mailhot' via golang-dev <golan...@googlegroups.com> wrote:

The notary part of Go modules, like the rest of the module
implementation, suffers from a lack of understanding of integration and
QA workflows, and a simplistic dev-centric worldview.

 

> [...]

 

I would have expected that if Fedora modified a library, they

would give it a different version number, so that for example modifying

v1.2.2 would produce v1.2.3-fedora.1 (it would be nice if SemVer

had a kind of 'post-release' syntax; maybe some day).

What about build metadata?

https://semver.org/#spec-item-10

"Thus two versions that differ only in the build metadata, have the same precedence.".

So it can be assumed that they identify the same version, but with a different go.sum hash.

 

> [...]

Manlio Perillo 

[Volker Dobler]

On a first, corse reading this sounds like you want a traditional

"GOPATH-build": You prepare all your sources by patching each

and every package used until you are satisfied and then you

compile and pack. This is basically the GOPATH approach: Have

everything laying around in one and one version only in the

filesystem and use what is there.

If one of the main assumptions of your build process is that

you cannot rely on go.mod anyway (because what the upstream

author codified in the go.mod must be validated and changed

anyway) then maybe a GOPATH build is pretty much what you

are asking for.

V.

[Russ Cox]

On Sat, Mar 9, 2019 at 2:55 PM Ian Denhardt <i...@zenhack.net> wrote:

Here's a reproducer:

> $ go mod init example.org/my-project
> go: creating new go.mod: module example.org/my-project
> $ cat >> main.go << "EOF"
> package main
> import _ "example.com/other-module"
> func main() {}
> EOF
> $ go list -mod=readonly ./...
> build example.org/my-project: cannot load example.com/other-module:
> import lookup disabled by -mod=readonly

I think we'd both mis-understood Nicolas's complaint the first time --
it's not that it still tries to access the network with -mod=readonly,
but that even for basic commands for which a read-only mod file/no
network access really *shouldn't* be a problem, it will still just bomb
out.

The go command is "bombing out" here because the code in question

is trying to import "example.com/other-module", yet no dependency listed

in go.mod provides that package (there are no dependencies in go.mod at all!).

[One of the things go list computes is the dependency graph

(visible if you use go list -json or go list -f '{{.Deps}}').]

It's true that if you have a CI/CD system, then for a successful build

you need to push complete code to it, and that includes a go.mod with

listed dependencies sufficient to cover the relevant imports,

as I described in my original reply to Nicolas. If go.mod were up to date

(for example, if 'go mod tidy' found no work to do) and the local

Go module cache already contained the listed dependencies,

then the 'go list -mod=readonly' command you showed would complete

just fine. You can't push code with an incomplete dependency list to the

CI/CD system and expect it to build, any more than you can push

code with missing function definitions and expect it to build.

Best,

Russ

[Russ Cox]

On Sun, Mar 10, 2019 at 10:47 PM Manlio Perillo <manlio....@gmail.com> wrote:

What about build metadata?

https://semver.org/#spec-item-10

"Thus two versions that differ only in the build metadata, have the same precedence.".

So it can be assumed that they identify the same version, but with a different go.sum hash.

 

Having a different go.sum means they differ in actual file content.

That's not the same version, nor is it "differ only in the build metadata"

(it literally has different source code!).

The Go module system strips the build metadata entirely to reach

a canonical version name, and it only allows one go.sum hash for

each such version.

Best,

Russ

[Nicolas Mailhot]

And that's exactly what won't work for us, because (unfortunately)
upstream state is often imperfect, using upstream code in production
involves some preparing (including patching).

Regards,

--
Nicolas Mailhot

[Nicolas Mailhot]

Le 2019-03-11 09:47, Volker Dobler a écrit :
> On a first, corse reading this sounds like you want a traditional
> "GOPATH-build": You prepare all your sources by patching each
> and every package used until you are satisfied and then you
> compile and pack. This is basically the GOPATH approach: Have
> everything laying around in one and one version only in the
> filesystem and use what is there.

It's not GOPATH (we have lots of problems with GOPATH) because the
result is shared among all the projects that need the module within
Fedora (distributions can scale because maintenance and patching of
individual components is delegated).

We are quite happy Go finaly started to work on code deployment aspects,
and waited for something like go modules for a long time (actual shared
libraries would be even better, but still).

However, we can't migrate to Go modules if they assume a workflow which
is incompatible with our own constrains.

Regards,

--
Nicolas Mailhot

[Nicolas Mailhot]

Le 2019-03-11 15:43, Russ Cox a écrit :
> On Sat, Mar 9, 2019 at 2:55 PM Ian Denhardt <i...@zenhack.net> wrote:
>
>> Here's a reproducer:
>>

>>> $ go mod init example.org/my-project [1]
>>> go: creating new go.mod: module example.org/my-project [1]

>>> $ cat >> main.go << "EOF"
>>> package main

>>> import _ "example.com/other-module [2]"

>>> func main() {}
>>> EOF
>>> $ go list -mod=readonly ./...

>>> build example.org/my-project [1]: cannot load
>> example.com/other-module [2]:

>>> import lookup disabled by -mod=readonly
>>
>> I think we'd both mis-understood Nicolas's complaint the first time
>> --
>> it's not that it still tries to access the network with
>> -mod=readonly,
>> but that even for basic commands for which a read-only mod file/no
>> network access really *shouldn't* be a problem, it will still just
>> bomb
>> out.
>
> The go command is "bombing out" here because the code in question

> is trying to import "example.com/other-module [2]", yet no dependency

> listed
> in go.mod provides that package (there are no dependencies in go.mod
> at all!).
> [One of the things go list computes is the dependency graph
> (visible if you use go list -json or go list -f '{{.Deps}}').]
>
> It's true that if you have a CI/CD system, then for a successful build
> you need to push complete code to it, and that includes a go.mod with
> listed dependencies sufficient to cover the relevant imports,
> as I described in my original reply to Nicolas. If go.mod were up to
> date
> (for example, if 'go mod tidy' found no work to do) and the local
> Go module cache already contained the listed dependencies,
> then the 'go list -mod=readonly' command you showed would complete
> just fine. You can't push code with an incomplete dependency list to
> the
> CI/CD system and expect it to build, any more than you can push
> code with missing function definitions and expect it to build.

And as I have explained in the detailed description Matthew requested,
our construction of the CI/CD environment is incremental, so the
assumption in the go tool code that "everything is there and it it is
not it can be downloaded directly" does not work for us

Regards,

--
Nicolas Mailhot

[Florian Weimer]

* Russ Cox:

> On Sun, Mar 10, 2019 at 10:47 PM Manlio Perillo <manlio....@gmail.com> wrote:
>
> What about build metadata?
> https://semver.org/#spec-item-10
>
> "Thus two versions that differ only in the build metadata, have the same precedence.".
> So it can be assumed that they identify the same version, but with a different go.sum
> hash.
>
>
> Having a different go.sum means they differ in actual file content.
> That's not the same version, nor is it "differ only in the build
> metadata" (it literally has different source code!).

But the source code contains build metadata, for example target
selection for architecture alternatives. Distributions are probably the
only ones who build most Go packages on ppc64le and s390x, so trivial
changes (e.g., to use generic code on additional architectures) are to
be expected.

Thanks,
Florian

[Manlio Perillo]

What about disabling go.sum check?

I wrote a (incomplete) patch that instruct cmd/go to ignore go.sum when the module download mode is "trust":

https://gist.github.com/perillo/59f3be87dd2b64d7d734c74d1f344cd5

Probably it will not be integrated in the official Go repository, but it can be used by people with special requirements in order to prevent each package manager to patch the go tool in different ways.

Thanks

Manlio Perillo 

[Ian Denhardt]

Quoting Russ Cox (2019-03-11 10:43:28)

> [One of the things go list computes is the dependency graph
> (visible if you use go list -json or go list -f '{{.Deps}}').]

Ah, that makes more sense.

[Matthew Dempsky]

On Mon, Mar 11, 2019 at 8:01 AM Nicolas Mailhot <nicolas...@laposte.net> wrote:

And as I have explained in the detailed description Matthew requested,

To be clear, I was asking for details to reproduce the technical issues you're running into. I may have missed them, but I don't believe you've provided these.

our construction of the CI/CD environment is incremental, so the
assumption in the go tool code that "everything is there and it it is
not it can be downloaded directly" does not work for us

If your build system provides the Go source/packages, then Go won't try to download them directly itself. This is how Go works within Google's build system (which, again, has the same no-network-access limitation as the build system you're describing, and yet supports building Go programs/packages).

It sounds like you're having trouble with providing the Go source/packages in the format expected by cmd/go. If you would provide reproduction steps of what you're doing and the problems you're running into, we can help advise what changes you need to make, and maybe identify tooling improvements to make that easier.

[Florian Weimer]

* via golang-dev:

> If your build system provides the Go source/packages, then Go won't
> try to download them directly itself. This is how Go works within
> Google's build system (which, again, has the same no-network-access
> limitation as the build system you're describing, and yet supports
> building Go programs/packages).

How do you plan to bypass the notary requirement? Do you have a
non-Internet connection to it?

Or do you pre-populate the expected data before starting the builds,
similar to OCSP stapling?

Thanks,
Florian

[Matthew Dempsky]

On Mon, Mar 11, 2019 at 11:09 AM Florian Weimer <fwe...@redhat.com> wrote:

How do you plan to bypass the notary requirement?

I don't work on the Go-inside-Google integration work, so someone else may have to weigh in if it's appropriate to share those plans. My point with mentioning Google's hermetic build system is more to emphasize that the Go developers are very familiar with that build model and care about keeping it working. The claims that the Go module system is being built without consideration of those requirements seem very mistaken.

That said, the notary is only involved when adding new lines to the go.sum file to handle adding or updating dependencies. There's no requirement to contact the Go notary when the go.sum file is already complete.

[Ian Denhardt]

Quoting Matthew Dempsky (2019-03-11 13:54:07)

In a parallel thread, a Nix developer was asking about basically the
same use case, and was pointed at `go build -mod=vendor`. It seems like
this does exactly what is wanted here -- just use the code we have
locally. Nicolas, does that address your use case?

-Ian

[Volker Dobler]

On Monday, 11 March 2019 15:56:37 UTC+1, nicolas.mailhot wrote:

Le 2019-03-11 09:47, Volker Dobler a écrit :
> On a first, corse reading this sounds like you want a traditional
> "GOPATH-build": You prepare all your sources by patching each
> and every package used until you are satisfied and then you
> compile and pack. This is basically the GOPATH approach: Have
> everything laying around in one and one version only in the
> filesystem and use what is there.

It's not GOPATH (we have lots of problems with GOPATH) because the
result is shared among all the projects that need the module within
Fedora (distributions can scale because maintenance and patching of
individual components is delegated).

I am not sure I understand the problem you had with GOPATH

based builds.

 

We are quite happy Go finaly started to work on code deployment aspects,
and waited for something like go modules for a long time (actual shared
libraries would be even better, but still).

However, we can't migrate to Go modules if they assume a workflow which
is incompatible with our own constrains. 

I'm still not convinced that your constraints are not self-contradictory.

On one hand you cannot use the dependency information provided by

go.mod and go.sum because each package needs manual patching

and on the other hand you dismiss that build mode that allows exactly

this as being too problematic. Are you sure there actually _is_ a build

mode that would suite your constraints?

V. 

[David Riley]

It's a good suggestion, but that, unfortunately, won't have the desired effect. Where SemVer "pre-release" metadata has the effect of, say, 1.2.3 taking precendence over 1.2.3-beta.42 or 1.2.3-rc.10-seriously-this-time-its-right, its build metadata has explicitly no precedence, so the effect of two equal versions with differing build metadata is indeterminate and you might as well treat it as a random choice.

You could *in theory* include distro-local revisions in that metadata, but standard tooling (such as Go's modules) won't respect that kind of thing when determining which version to use, so it's not particularly helpful here because there's nothing to compel the build system to use it rather than the upstream version or vice versa unless you call out the specific full tag (which, generally, you don't want to do).

Per a previous poster, it would have been quite helpful if the SemVer author(s) had included post-release or subsuming variant codings (such as architecture-specific or distro-specific builds of a version) rather than requiring reversion to the somewhat clunky, limited and semantically-misleading x.y.z++-prerelease versioning currently available. This is a versioning paradigm that's been present in Linux package managers since as long as I can remember (though my memory really only stretches back to a bit before the glibc transition era). Ah, well, maybe in SemVer 3.0.


- Dave

[roger peppe]

I believe it would be possible to remove all go.sum files and use an independent notary and Go proxy.

That would fit the workflow, but no-one would know what actual code they're running. Which is the same as things generally are today, it seems, except in our Go module Utopia.

At least the binary in question could introspect itself with debug.BuildInfo to tell the user that it had been altered, so if you care about being MitMed, you could check that against the central notary info before daring to run the binary. If you trust the binary to faithfully report its own version info of course.

--

[Nicolas Mailhot]

Which we be pretty much the case all the time for us, since we build
against out own curated set of modules, not the ones upstream found on
the Internet.

In fact, I'm pretty sure we will start each build by removing the
upstream go.sum file altogether.

Regards,

--
Nicolas Mailhot

[Nicolas Mailhot]

That's definitely *not* what we want (our processes call for removing
vendor and its equivalents in other languages before going to the build
step). We do *not* like vendor. We do *not* like GOPATH. Modules are
good. As long as we can create and use *our* modules, not someone else's
idea of how a module content should look like. Our modules are basically
the same thing as upstream modules (API compatible) with lots of warts
removed.

As I already noted, there is a deep lack of understanding Go upstream
side of how we work (for all computer system languages, not just for
Go). What they imagine is not what we need or want.

Regards,

--
Nicolas Mailhot

[Nicolas Mailhot]

We do not need nor want any compel at the go build command layer. We
want release (distro revision) selection at the go build layer to be
implicit, because the CI/CD system already makes sure only one of those
will exist at any time in the GOPROXY directory. The whole "opt-in
explicitly module by module on a specific distro revision" approach
would be a major nuisance to implement.

The only compel that would be useful for us is "use the best module
available in this GOPROXY tree, do not take into account any other
module source", assuming go build could ever use more than one GOPROXY
source at once (which would be a natural evolution, as soon as one stops
assuming there in one source of module truth on the internet and nothing
else, and the proxy just proxies this source).

Regards,

--
Nicolas Mailhot

[Nicolas Mailhot]

Le 2019-03-11 22:25, roger peppe a écrit :
> I believe it would be possible to remove all go.sum files and use an
> independent notary and Go proxy.
>
> That would fit the workflow, but no-one would know what actual code
> they're running.

That's not a problem distro side, because everything is already tracked
and checksummed and signed (and tracked generically so the same tracking
applies for go modules and the out-of-module resources the go module may
use or interact with)

Regards,

--
Nicolas Mailhot

[Jakub Cajka]

I think that our issue in Fedora is that we would like to continue using something that resembles current "GOPATH/vendor" approach where the tooling is not trying to kind of out smart us. As we have in distribution environment full control about the build environment and dependencies and we want/will have to retain it. I believe that having some "dumb mode" for operating all the go commands/tools(without the hard need for a modules (verification), even with reduced features) would be much appreciated(so you can use it as any other compiler/interpreter by just throwing code at it).

IMHO we will tap the metadata that the Go modules carry to simplify our processes and automations, but for example having to run our own notary(even just in build environment) just to be able to use go build/go list/go vet... might be really costly and we will be most probably looking for any alternative.

It has been mentioned several times, in this bunch of treads, that Google's internal build infrastructure has same requirements and is facing similar challenges. It would be great to hear how do they plan to "workaround" up coming module changes.

JC

[Tristan Colgate]

Don't google use Blaze/Bazel for builds internally? I don't think go
get and modules are hugely relevant in that context (see
https://github.com/bazelbuild/rules_go#does-this-work-with-go-modules).

--
Tristan Colgate-McFarlane
----
"You can get all your daily vitamins from 52 pints of guiness, and a
glass of milk"

[Ian Denhardt]

Quoting Nicolas Mailhot (2019-03-12 04:22:45)

> > In a parallel thread, a Nix developer was asking about basically the
> > same use case, and was pointed at `go build -mod=vendor`. It seems like
> > this does exactly what is wanted here -- just use the code we have
> > locally. Nicolas, does that address your use case?
>
> That's definitely *not* what we want (our processes call for removing
> vendor and its equivalents in other languages before going to the
> build step).

Not sure we're on the same page re: what was suggested. The idea is
you'd point the vendor directory at something that contains the distro's
versions of the dependencies, and use -mod=vendor to bypass all of the
smarts around fetching modules and such. I was not proposing using an
existing vendor directory provided by upstream or anything like that.

This is more or less what Jakub described in a sibling comment, as far
as I can tell.

> As I already noted, there is a deep lack of understanding Go upstream
> side of how we work (for all computer system languages, not just for
> Go). What they imagine is not what we need or want.

This definitely jives with my own memories from the last time I was a
maintainer on a distro -- language package managers and module systems
generally tend to be much more oriented towards developers using the
modules than to distro maintainers packaging them. It's to the point
where as an end-user of a distro I have a boat load of executables in
places like ~/.local/bin, because I use enough oddball tools that aren't
popular enough to be maintained by the distros themselves, and packaging
them myself just isn't worth the trouble. I do wish these tools played
more nicely together.

[Nicolas Mailhot]

Le 2019-03-13 03:24, Ian Denhardt a écrit :
> Quoting Nicolas Mailhot (2019-03-12 04:22:45)
>
>> > In a parallel thread, a Nix developer was asking about basically the
>> > same use case, and was pointed at `go build -mod=vendor`. It seems like
>> > this does exactly what is wanted here -- just use the code we have
>> > locally. Nicolas, does that address your use case?
>>
>> That's definitely *not* what we want (our processes call for removing
>> vendor and its equivalents in other languages before going to the
>> build step).
>
> Not sure we're on the same page re: what was suggested. The idea is
> you'd point the vendor directory at something that contains the
> distro's
> versions of the dependencies, and use -mod=vendor to bypass all of the
> smarts around fetching modules and such.

But, again, I don't *want* a vendor directory, distro or otherwise.

vendor (and GOPATH) have always been a major PITA to assemble and
maintain (and I speak as the person, that wrote at least half of the
code that assembles and maintains them Fedora-side).

Ideally, I'd like proper shared libs, because rebuilding every dependent
on changes is an obstacle to robust security (you forget one rebuild and
poof, you're owned).

Baring that, I'd settle for a directory of modules, which was what
GOPROXY was supposed to deliver, before Go upstream embarked in its mad
crusade to squeeze out anything between dev and production.

--
Nicolas Mailhot

[thepud...@gmail.com]

Hi Nicolas,

This is a very long thread, so sorry if I have not digested everything here, but regarding your recent comment here:

  "I'd settle for a directory of modules, which was what 

   GOPROXY was supposed to deliver, before Go upstream embarked in its mad 

   crusade to squeeze out anything between dev and production."

Focusing on that option for a moment, I am guessing when you say "squeeze" there you are referencing the subject of this thread, namely the proposal for a notary?

If so, as far as I understand, that is something you would be able to opt out of? E.g., the proposal suggests the notary can be disabled entirely with GONOVERIFY=*. How much does it help your use cases if you can opt out of the notary?

One related challenge sounds like how to package up code to transform it into a the form of a module that can live in a directory of modules accessed by GOPROXY=file:///path/to/your/local/copies.  As I think you've seen, the conversation here is still on-going:

    #28835 -- "proposal: cmd/go: subcommands to add and remove modules from the module cache"

    https://github.com/golang/go/issues/28835

However, even in advance of that, I have seen different people put together one-off shell scripts or similar that are capable of putting a properly formed entry into the module cache such that it can be used by GOPROXY=file://your/path. For example:

    https://github.com/komuw/goproxy/blob/master/goproxy.sh#L71

I have not personally used that example and hence cannot personally vouch for it, but from a very quick look it at least seems to be in the ball park. I understand that is different from having first-class support in the 'go' tool for something like #28835, but it might be an avenue to consider especially if you need more flexibility than the current 'go' tool provides today or after #28835.

I think you mentioned the time field as being problematic, but that you wondered if you could zero it out. That would be worth a try I would think, perhaps with actual zeros, or perhaps using Go's zero value for time.Time (which the 'go' command uses in some cases such as v0.0.0-00010101000000-000000000000, e.g., in https://go-review.googlesource.com/c/go/+/152739 ). To be clear, though, that is speculation on my part as to what might work for you and what the 'go' command might accept.

As others have said, the 'go.sum' is a line-oriented text file, and in theory you could edit or construct your own if that is a path you wanted to pursue. In case it helps, here is an importable Go package that contains the logic to compute the SHA256 checksums for a module:

    https://github.com/rogpeppe/go-internal/blob/master/dirhash/hash.go#L25

There are a couple of related functions in that same package that allow you to pass in the directory to hash. I was able to use that package seemingly successfully on another project. That package is an extraction of some functionality from the Go tooling. In other words, that is internal code that is currently being managed and exposed by some people from the broader community, and hence that particular code is not covered under any type of Go1 Compatibility promise... but as far as I am aware at this point the logic to create the hash is considered stable by the core Go team for the 'h1' hash calculation.

A related issue is how to construct a hash for 'go.sum' when a module does not itself have a 'go.mod'. I think what the 'go' tool does is synthesize a 'go.mod' that just contains the 'module' line and nothing else, which is illustrated with this playground example:

    https://play.golang.org/p/8ebHdCxUKmI

(I tested that exactly once, though, so please do not be too disappointed if my quick attempt is not correct).

In any event, it sounds like you are very deep on all of this already, so there is a healthy chance I have not said anything useful to you here, but wanted to at least share some thoughts and related pointers.

Regards,

thepudds

[Anthony Martin]

'Nicolas Mailhot' via golang-dev <golan...@googlegroups.com> once said:
> And that's exactly what won't work for us, because (unfortunately) upstream
> state is often imperfect, using upstream code in production involves some
> preparing (including patching).

Just to be clear, you want to claim to your users that you're using a
specific version of an upstream module and you want to be able to modify
it's code without giving it a different version or name?

Anthony

[Jakub Cajka]

----- Original Message -----
> From: "Anthony Martin" <al...@pbrane.org>
> To: "Nicolas Mailhot" <nicolas...@laposte.net>
> Cc: "Russ Cox" <r...@golang.org>, "golang-dev" <golan...@googlegroups.com>
> Sent: Wednesday, March 13, 2019 9:11:38 PM
> Subject: Re: [golang-dev] proposal: public module authentication with the Go notary
>

Pretty much. That is what most of the integrators(Linux distributions) do, although we are trying to avoid that at all cost, sometimes we have to use downstream patches, for various reasons(meeting deadlines, not upstream-able,...).

[Florian Weimer]

* Jakub Cajka:

> ----- Original Message -----
>> From: "Anthony Martin" <al...@pbrane.org>
>> To: "Nicolas Mailhot" <nicolas...@laposte.net>
>> Cc: "Russ Cox" <r...@golang.org>, "golang-dev" <golan...@googlegroups.com>
>> Sent: Wednesday, March 13, 2019 9:11:38 PM
>> Subject: Re: [golang-dev] proposal: public module authentication with the Go notary
>>
>> 'Nicolas Mailhot' via golang-dev <golan...@googlegroups.com> once said:
>> > And that's exactly what won't work for us, because (unfortunately) upstream
>> > state is often imperfect, using upstream code in production involves some
>> > preparing (including patching).
>>
>> Just to be clear, you want to claim to your users that you're using a
>> specific version of an upstream module and you want to be able to modify
>> it's code without giving it a different version or name?
>>
>> Anthony
>>
>
> Pretty much. That is what most of the integrators(Linux distributions)
> do, although we are trying to avoid that at all cost, sometimes we
> have to use downstream patches, for various reasons(meeting deadlines,
> not upstream-able,...).

But do users who run the go command currently see this?

I thought that it's strictly for internal use, for building Go
applications for the distribution itself. So the users here would be
Fedora developers. It's kind of like golang.org/x, but using Fedora/RPM
technology.

I assume that Go programmers using the Fedora Go packages would still
obtain whatever upstream Go would download using “go get”, even if the
sources are also packaged for Fedora's own (internal) use.

Thanks,
Florian

[Brad Fitzpatrick]

On Wed, Mar 13, 2019 at 1:30 PM Jakub Cajka <jca...@redhat.com> wrote:

----- Original Message -----
> From: "Anthony Martin" <al...@pbrane.org>
> To: "Nicolas Mailhot" <nicolas...@laposte.net>
> Cc: "Russ Cox" <r...@golang.org>, "golang-dev" <golan...@googlegroups.com>
> Sent: Wednesday, March 13, 2019 9:11:38 PM
> Subject: Re: [golang-dev] proposal: public module authentication with the Go notary
>
> 'Nicolas Mailhot' via golang-dev <golan...@googlegroups.com> once said:
> > And that's exactly what won't work for us, because (unfortunately) upstream
> > state is often imperfect, using upstream code in production involves some
> > preparing (including patching).
>
> Just to be clear, you want to claim to your users that you're using a
> specific version of an upstream module and you want to be able to modify
> it's code without giving it a different version or name?
>
>   Anthony
>

Pretty much. That is what most of the integrators(Linux distributions) do, although we are trying to avoid that at all cost, sometimes we have to use downstream patches, for various reasons(meeting deadlines, not upstream-able,...).

There are two reasons why a distro generally packages up a library for a language (let's say "golang-github-foo-bar-devel")

1) because they need that library to ship some other high-level binary (e.g. caddy or docker or kubernetes that a user might want to install as a binary), and Docker needs golang-github-foo-bar-devel to build.

2) because they want to offer golang-github-foo-bar-devel to developers to build their own programs.

If you want to modify source code in the distro (for whatever reasons) for case (1), fine. We won't (and can't) stop you. But we'd still like you to modify your version strings to add a -distro_v1 or -distro_v2 sort of suffix to them so https://godoc.org/rsc.io/goversion and such tools accurately report what's in binaries.

If you want to do (2), that seems much more painful for all parties (distros, Go upstream, our collective users) and much less useful for users. Eventually users are going to hit a dependency that's not in their distro anyway. And even if they all were in the distro, they should be able to publish a go.mod+go.sum file in their github repos that other people from distros are able to use. If my local distro is lying to me about the state of the world and writing bogus go.sum files that people from other distros can't verify, that seems bad.

I think distros should spend their time focusing on packaging Go for their own use for (1) and not about trying to help developers for case (2).

[Robert Engels]

It seems the distro folks should vendor and distribute binaries. If I’m using a distro to develop Go programs I’m testing and verifying against what they are expected to building against not a proprietary version. That is the whole point of using modules and specifying the versions. At least that’s how I see it...

[Manlio Perillo]

On Wednesday, March 13, 2019 at 9:44:52 PM UTC+1, Brad Fitzpatrick wrote:

> [...]

 

There are two reasons why a distro generally packages up a library for a language (let's say "golang-github-foo-bar-devel")

1) because they need that library to ship some other high-level binary (e.g. caddy or docker or kubernetes that a user might want to install as a binary), and Docker needs golang-github-foo-bar-devel to build.

2) because they want to offer golang-github-foo-bar-devel to developers to build their own programs.

If you want to modify source code in the distro (for whatever reasons) for case (1), fine. We won't (and can't) stop you. But we'd still like you to modify your version strings to add a -distro_v1 or -distro_v2 sort of suffix to them so https://godoc.org/rsc.io/goversion and such tools accurately report what's in binaries.

If you want to do (2), that seems much more painful for all parties (distros, Go upstream, our collective users) and much less useful for users. Eventually users are going to hit a dependency that's not in their distro anyway. And even if they all were in the distro, they should be able to publish a go.mod+go.sum file in their github repos that other people from distros are able to use. If my local distro is lying to me about the state of the world and writing bogus go.sum files that people from other distros can't verify, that seems bad.

I think distros should spend their time focusing on packaging Go for their own use for (1) and not about trying to help developers for case (2).

I agree.

Modern languages, like Python, Rust and Go all have good package managers.  There is no need for a distribution to package them.

Packaging is useful for big library/tool written in C/C++, like GCC, libstdc, llvm and so on, that are a pain to build, even for developers.

I'm a bit uncertain about packaging databases like PostgreSQL or web server like Nginx.  At least PostgreSQL and Nginx are easy to build, and a web developer will need to have multiple versions installed.

Manlio Perillo

[Jakub Cajka]

Pretty much we are doing that(more trying to, in Fedora). Building complex Go projects with vetted and sane dependencies that are not ideally vendored/bundled and forgotten as many upstream do that, even including downstream patches for the vendored/bundled code...

Even if we wanted to we really can't do what are you implying in the (2) as it requires crazy amount of manpower to do for nearly no gain compared to working in upstreams(which I believ most of the Fedora contributors do).

[Jakub Cajka]

----- Original Message -----
> From: "Robert Engels" <ren...@ix.netcom.com>
> To: "Brad Fitzpatrick" <brad...@golang.org>
> Cc: "Jakub Cajka" <jca...@redhat.com>, "Anthony Martin" <al...@pbrane.org>, "Nicolas Mailhot"
> <nicolas...@laposte.net>, "Russ Cox" <r...@golang.org>, "golang-dev" <golan...@googlegroups.com>
> Sent: Wednesday, March 13, 2019 11:34:04 PM
> Subject: Re: [golang-dev] proposal: public module authentication with the Go notary
>

> It seems the distro folks should vendor and distribute binaries. If I’m using
> a distro to develop Go programs I’m testing and verifying against what they
> are expected to building against not a proprietary version. That is the
> whole point of using modules and specifying the versions. At least that’s
> how I see it...
>

We want to maintain just one copy of the dependency for whole distribution so we try not to vendor and yes we do distribute binaries along with all bit that has been used in the build of the said binary so you can reproduce the build.
I'm probably misunderstanding you, but to be clear, we don't have any proprietary code in Fedora. Everything is public with FOSS licenses[1].

JC

[1] https://fedoraproject.org/wiki/Licensing:Main

[Jakub Cajka]

----- Original Message -----
> From: "Manlio Perillo" <manlio....@gmail.com>
> To: "golang-dev" <golan...@googlegroups.com>
> Sent: Wednesday, March 13, 2019 11:39:44 PM
> Subject: Re: [golang-dev] proposal: public module authentication with the Go notary
>

I will disagree with you here. Mostly as most "modern" languages encourage static linking(even when it is not necessary) and bundling of the "forked"/vendored dependencies. As current world stands there is actually IMHO less incentive to package big "old" C/C++/... projects, as you state, they are relatively easily build-able in sane(know, not forked dependencies) configuration than most of the "modern" languages projects.

[Nicolas Mailhot]

1. We want to claim to building/linking/selecting *software* that our
massaged code is equivalent to the unchanged upstream version in our
distribution context (we don't vouch it will work on a windows platform,
for example). So, identical name and version to upstream in the
building/linking/selecting tools.

2. It's fine and encouraged to tag it in such a way humans see it is our
massaged version of the upstream code. We're not trying to trick humans
or inventory software. Specific tags that do not participate in building
/ linking /selecting decisions are fine.

Moreover:

3. When building official Fedora software, we want to use exclusively
our own massaged code. It will be deployed in a specific system
directory.

4. As I wrote in my (long) description massaging code involved the
creation of staging artefact directories used in addition to the system
artefact directory.

5. Our human users would often like to mix this massaged code with
upstream code, when they need a project or version we haven't proofed
yet. Depending on language tooling facilities, this mixing is possible
or not. So it's an appreciated bonus, but not mission critical. That may
seem contradictory to 1. since we use the same artefact names and
versions as upstream. Software can distinguish between system code and
upstream code, because system code is deployed in specific directories,
as per the FHS and *nix traditions.

Regards,

--
Nicolas Mailhot

[Nicolas Mailhot]

Not necessarily. This direct upstream fetching is often used in addition
and as a complement to distribution-proofed code, not as a complete
substitute.

The closer you get to production deployments the more you appreciate
that someone checked and fixed some of the modules you need
distribution-side so you don't have to worry about checking them
yourself.

In full hardened mode you completely stop using direct upstream
fetching. If you need something not available in the distribution, you
start by contributing it to the distribution (or run your own
mini-distribution complement), to benefit from distribution auditing and
review processes and tools.

--
Nicolas Mailhot

[Nicolas Mailhot]

Le 2019-03-13 23:39, Manlio Perillo a écrit :

> Modern languages, like Python, Rust and Go all have good package
> managers. There is no need for a distribution to package them.

Unfortunately, none of the language-specific package managers cope well
with anything that mixes their language source code with something else.
So the language-specific package managers tend to fail as soon as a
project reaches the complexity, that requires such mixing. You need
something language-agnostic to coordinate all the project parts.

This is blindingly obvious, to anyone that had to work with Go code,
that used protobuf for example. Forking everything inside the language
to avoid dealing with the conventions and needs of other languages has
huge limits.

--
Nicolas Mailhot

[Volker Dobler]

On Thursday, 14 March 2019 09:42:11 UTC+1, Jakub Cajka wrote:

We want to maintain just one copy of the dependency for whole distribution so we try not to vendor and yes we do distribute binaries along with all bit that has been used in the build of the said binary so you can reproduce the build. 

That really sounds like a big GOPATH build:

Each package is there in the GOPATH in just one, "massaged" copy.

All binaries are built against these massaged packages in GOPATH,

no binary uses vendored stuff. If a enduser wants to reproduce the

build of a binary he can install this big massaged GOPATH directory.

The argument against GOPATH based build was largely something

around "We tried, it was hard, we don't like it. And we have to rebuild

all binaries on any change to make sure the change is incorporated

in all binaries."

(The second argument is a good one but given that the go tool does

most of the heavy lifting and no recompile stuff which is unaffected

by the change this seems just a tiny inconvenience.)

To me the first argument is a direct consequence of massaging all

packages into one state: If you want/need to do this then you have

to do this and this work does not go away magically.

I still do not see why you cannot use a GOPATH-style build. Or maybe

a 5-GOPATH-style build with 5 clusters of binaries sharing a GOPATH.

V.

 

[Nicolas Mailhot]

Le 2019-03-13 20:33, thepud...@gmail.com a écrit :
> Hi Nicolas,

Hi thepudds,

Thank you for the constructive comments.

> This is a very long thread, so sorry if I have not digested everything
> here, but regarding your recent comment here:
>
> "I'd settle for a directory of modules, which was what
> GOPROXY was supposed to deliver, before Go upstream embarked in its
> mad
> crusade to squeeze out anything between dev and production."
>
> Focusing on that option for a moment, I am guessing when you say
> "squeeze" there you are referencing the subject of this thread, namely
> the proposal for a notary?

Unfortunately, it does not manifest just in the notary.

It manifests in lots of things that were added to go for modules. In
module mode go tooling tries to “fix” your code and inject artefacts
directly downloaded from the internet at every turn.

The notary was just the last straw.

> If so, as far as I understand, that is something you would be able to
> opt out of? E.g., the proposal suggests the notary can be disabled
> entirely with GONOVERIFY=*. How much does it help your use cases if
> you can opt out of the notary?

It would certainly help.

Disabling verification for specific module directories would be better.

Having the ability to set this configuration in an /etc file to avoid
carrying a large set of environment variables all the time would be even
better.

> One related challenge sounds like how to package up code to transform
> it into a the form of a module that can live in a directory of modules
> accessed by GOPROXY=file:///path/to/your/local/copies. As I think
> you've seen, the conversation here is still on-going:
> #28835 -- "proposal: cmd/go: subcommands to add and remove modules
> from the module cache"
> https://github.com/golang/go/issues/28835

Yes I'm aware of that conversation. It would be nice if it went forward.
Unfortunately, this is just a part (a huge part true, but only a part)
of the go module infrastructure that we need.

We also have the issue of regenerating list indexes once something (the
system component manager) added or removed the produced modules files
from the system module directory/directories.

And, because Go upstream chose to allow cross references in upstream Go
modules, we have the huge issue of bootstrapping, ie producing a set of
modules with a direct acyclic graph of dependencies, to handle the
initial seeding.

That requires computing reduced-function modules variants without the
problematic dependency links.

This problem is exacerbated by the Go upstream decisions to:
— make tests an integral non-optional part of modules (unit tests are a
huge source of cycles)
— separate GOPATH and module code sources (we can not reuse the
bootstrapping done over years of Fedora GOPATH packaging)

> However, even in advance of that, I have seen different people put
> together one-off shell scripts or similar that are capable of putting
> a properly formed entry into the module cache such that it can be used
> by GOPROXY=file://your/path. For example:
> https://github.com/komuw/goproxy/blob/master/goproxy.sh#L71

And I have my own (unpublished incomplete and unfinished) code to try
doing the same thing. Not elegant code I'm especially proud of, just
quick and dirty “get it to work before august” code.

The script you linked will only work for the most simple projects.
Handling all the cases that the internal code of go knows about takes a
lot more code. That, even before adding the complexity induced by the
vet-code-before-it-is-available distribution requirement.

A lot of the things I wrote in this thread were informed by what I
learnt writing this code.

I lost the number of times when I had to change my coding approach
because something I did triggered the “go to the internet and download
upstream stuff” go module logic.

My code is slowly evolving from “use upstream go commands with minimal
changes” to “rewrite from scratch”. The go upstream module behavior just
does not work in our environment.

> I think you mentioned the time field as being problematic, but that
> you wondered if you could zero it out. That would be worth a try I
> would think, perhaps with actual zeros, or perhaps using Go's zero
> value for time.Time (which the 'go' command uses in some cases such as
> v0.0.0-00010101000000-000000000000, e.g., in
> https://go-review.googlesource.com/c/go/+/152739 ). To be clear,
> though, that is speculation on my part as to what might work for you
> and what the 'go' command might accept.

That is just one example, why writing the tooling is just the first
step. You need to apply the tooling to a large set of real-world code to
understand all the side-effects of your design decisions. Expecting to
switch to module mode the day after the tooling is (finally) written is
unrealistic.

As things stand, I'm pretty pessimistic that we will be able to make the
switch to module mode for Go 1.13, or even the next version. Too many of
the assumptions, that drove Go module design decisions, do not apply in
our environment. It's going to take time to find out how to bridge the
gap gracefully.

> As others have said, the 'go.sum' is a line-oriented text file, and in
> theory you could edit or construct your own if that is a path you
> wanted to pursue. In case it helps, here is an importable Go package
> that contains the logic to compute the SHA256 checksums for a module:
>
> https://github.com/rogpeppe/go-internal/blob/master/dirhash/hash.go#L25

Thank you. I already use this package. It is a lifesaver, even though it
is not endorsed, and is missing the parts needed to regenerate list
files.

> A related issue is how to construct a hash for 'go.sum' when a module
> does not itself have a 'go.mod'.

I don't think this will be a problem.

We need to name artefacts to pack them, we always made the distinction
between artefact name and download url, so our naming reference will be
the go.mod Module section.

I can't imagine why we would ever pack anything that is lacking this
module descriptor.

> In any event, it sounds like you are very deep on all of this already,
> so there is a healthy chance I have not said anything useful to you
> here, but wanted to at least share some thoughts and related pointers.

The pointers are appreciated nevertheless, you never know when you have
missed something.

Best regards,

--
Nicolas Mailhot

[Nicolas Mailhot]

Le 2019-03-14 11:11, Volker Dobler a écrit :

> I still do not see why you cannot use a GOPATH-style build. Or maybe
> a 5-GOPATH-style build with 5 clusters of binaries sharing a GOPATH.

Because when you package and build software at the distribution scale,
you do not make the world available to every single build, you construct
a minimal contained environment for every build. This build environment
is managed, it's not the full distribution haystack¹.

That means a huge unstructured mass of source code like vendor or GOPATH
does not work, you need to manage the artefacts composing the build
environment, and that requires artefacts with clear boundaries and
versions¹.

Modules are good because they finally give us those boundaries and
versioning.

¹ Besides the full haystack is rarely instalable, because in the real
world, you have conflicting artefacts. As would be blindingly obvious to
anyone that tried to construct a single GOPATH that worked for all the
Go software distributions build.

--
Nicolas Mailhot

[Volker Dobler]

I see. Thanks for the explanation.

V.

[Manlio Perillo]

But what PostgreSQL and Nginx do is to state their dependencies in the documentation.

It it responsibility of the system admin to make sure the dependencies are installed and compatible.

Isn't this the same as it was done in Go with GOPATH?

The problem is that an OS distribution maintainer *can* afford the burden to ensure that only one version of the dependency is installed on the system, and that all packages depending on them build correctly.

A normal developer can't.  And what about the case where the developer needs to maintain a web application used on different servers, with different OS and with different versions?

Another difference is that "system" libraries are usually written in C, there are "few" of them and usually are mature with a stable API.

This does not apply with  modern languages, where you have hundred of packages, usually written by inexperienced programmers (but still used indirectly by important packages).

Go has chosen to favor the normal developers. 

But still, I agree that it should also support OS distribution maintainers,

and probably normal developers should start to think like them (https://research.swtch.com/deps).

Manlio Perillo 

[Manlio Perillo]

Right, I forgot about this issue.

There are special cases however, as an example SQLite that make it easy to embed in another project.

I find it strange that in all these years nobody decided to build a "system" packages registry, where package names are independent from OS package names.

A separate registry can be used to map standard package names to OS specific names.

A Go module can then use a "foreign depends" directive, to state the system packages required, and that can be installed by a separate command.

 

Manlio Perillo

[Robert Engels]

I don’t think that is true. I want to test, especially integration test, against what “any” user would be compiling against, not just Fedora vX users. Since Go is local compiled, having packages that differ than I would get is a development nightmare. Maybe this works for the largest library/apps but it can’t work for the smallest.

Maybe a locally compiled Fedora binary would want this, but why would they be recompiling the binary - they would update the binary package.

[Bryan C. Mills]

It sounds like `replace` directives are a better fit for your use-case than injecting things at the proxy level.

In particular, a `replace` directive explicitly establishes equivalence between the replaced version and its replacement.

You could layer that on top of the `go` command as a script that examines the output of `go list -m all` and adds filesystem-local `replace` directives for all of the reported modules. Then you can set `GOPROXY=off` and build with the resulting `go.mod` file and everything should work exactly as you expect within that module.

Users who explicitly want your replacement modules could run the same script in their own codebase. That would allow them to use your curated copies, but would make the choice to do so very explicit — so they would not be confused when the resulting source code does not match what they observe upstream.

[Russ Cox]

I will be away from work starting tomorrow until April 1, so I will not be able to follow up here until then.

Before I go I wanted to just say thank you to Nicolas for the detailed explanation of Fedora's needs.

They've helped me think about it quite a bit. When I get back I want to continue the conversation about

what we can do to support what distributors need while not taking away what users need as well.

Thanks.

Russ

[Nicolas Mailhot]

Le 2019-03-14 14:32, Robert Engels a écrit :
> I don’t think that is true. I want to test, especially integration
> test, against what “any” user would be compiling against, not just
> Fedora vX users.

Like I wrote it's "not necessarily".

That depends whether:
– your target is to produce a production binary for a specific system
(and you want to pick up all the last-mile security fixes of that
system) or
— your target is to produce system-agnostic code (and you don't care
about the last mile fixes, because another team will work on them after
you produce your code, even if you don't realize this).

The distribution objective is always to reduce last-mile fixing to the
minimum, but that depends on the quality of the upstream code, and on
the lag upstream takes to fix the problems detected distribution-side.

Like others noted in the thread, both Fedora and Google carry production
patches to the golang codebase itself, and lots of upstream projects do
not attain the quality of the golang codebase.

--
Nicolas Mailhot

[Nicolas Mailhot]

Le 2019-03-14 15:10, Bryan C. Mills a écrit :

Hi Bryan

> It sounds like `replace` directives are a better fit for your use-case
> than injecting things at the proxy level.
> In particular, a `replace` directive explicitly establishes
> equivalence between the replaced version and its replacement.

As noted elsewhere in the thread, while replace is a fine mechanism to
handle exceptions, it's really not adapted for mass use. In our build
environment use of our own modules is not the exception and replacing
every single line of upstream module descriptors would be both
resource-intensive and brittle.

So we'd be a lot more likely to patch out the notary (or other) checks
that make this necessary in the go command shipped to Fedora users, that
to play this dance.

--
Nicolas Mailhot

[Nicolas Mailhot]

Le 2019-03-14 15:45, Russ Cox a écrit :

Russ,

> I will be away from work starting tomorrow until April 1, so I will
> not be able to follow up here until then.
> Before I go I wanted to just say thank you to Nicolas for the detailed
> explanation of Fedora's needs.
> They've helped me think about it quite a bit. When I get back I want
> to continue the conversation about
> what we can do to support what distributors need while not taking away
> what users need as well.

Thanks a lot for your attention. I realize it's hard to project oneself
in another way of doing things, with different vocabulary, conventions
and priorities. So, I didn't expect instant miracles.

Just coming to a shared understanding of what works or not for
distributions would be a huge step forward.

Have a nice vacation,

--
Nicolas Mailhot

[Robert Engels]

I am not sure that Go license terms allow you to distribute a Go command that is incompatible with mainline. This was hashed out with Java long ago...

[Nicolas Mailhot]

Le 2019-03-14 18:17, Robert Engels a écrit :
> I am not sure that Go license terms allow you to distribute a Go
> command that is incompatible with mainline. This was hashed out with
> Java long ago...

I don't think Go has this legal requirement, but if it has, and upstream
Go chooses a direction that does not work for us, it will just be
openjdk/iceweasel time.

Java is a pretty good example of why such situations are bad both for
upstream and downstream. Last I've checked upstream Java was still
trying to heal all the ramifications of past legal arguings.

--
Nicolas Mailhot

[Brad Fitzpatrick]

On Thu, Mar 14, 2019 at 10:17 AM Robert Engels <ren...@ix.netcom.com> wrote:

I am not sure that Go license terms allow you to distribute a Go command that is incompatible with mainline. This was hashed out with Java long ago...

You're not sure or you haven't looked? It's a pretty standard, permissive license: https://golang.org/LICENSE

Or was there something else stated elsewhere that led you to make this statement?

[Bryan C. Mills]

On Thu, Mar 14, 2019 at 12:28 PM Nicolas Mailhot <nicolas...@laposte.net> wrote:

Le 2019-03-14 15:10, Bryan C. Mills a écrit :

Hi Bryan

> It sounds like `replace` directives are a better fit for your use-case
> than injecting things at the proxy level.
> In particular, a `replace` directive explicitly establishes
> equivalence between the replaced version and its replacement.

As noted elsewhere in the thread, while replace is a fine mechanism to
handle exceptions, it's really not adapted for mass use. In our build
environment use of our own modules is not the exception and replacing
every single line of upstream module descriptors would be both
resource-intensive and brittle.

You don't have to replace “every single line”. A `replace` directive in the main module applies to all of its dependencies, so all you need to do — when building a specific module — is to query the transitive dependencies of that module (`go list -m all`) and replace them within that module.

The tooling for that should be nearly trivial: you could probably do it with `sed`, and if not, a wrapper binary in Go would probably be no more than a hundred lines.

[Robert Engels]

I’ve followed this for a long time, and was just assuming as it is pretty standard but my reading of the license would seem to allow it - and that’s a problem IMO. 

When you create something called Go and DISTRIBUTE it, and it’s not compatible with Go, you can’t call it Go, nor replace a standard Go - think of the chaos you could create... this was the crux of the Sun v Ms Java debate. 

If it’s used internally then you can do whatever you want. 

[Brad Fitzpatrick]

On Thu, Mar 14, 2019 at 10:57 AM Robert Engels <ren...@ix.netcom.com> wrote:

I’ve followed this for a long time, and was just assuming as it is pretty standard but my reading of the license would seem to allow it - and that’s a problem IMO. 

I fail to parse this sentence. You've followed what? You assumed the license is standard (it is), and you read the license and the license allows it (redistribution) and that's a problem? How? What's the problem?

 

When you create something called Go and DISTRIBUTE it, and it’s not compatible with Go, you can’t call it Go, nor replace a standard Go

Where does the license say that?

[Manlio Perillo]

On Thursday, March 14, 2019 at 7:06:39 PM UTC+1, Brad Fitzpatrick wrote:

On Thu, Mar 14, 2019 at 10:57 AM Robert Engels <ren...@ix.netcom.com> wrote:

I’ve followed this for a long time, and was just assuming as it is pretty standard but my reading of the license would seem to allow it - and that’s a problem IMO. 

I fail to parse this sentence. You've followed what? You assumed the license is standard (it is), and you read the license and the license allows it (redistribution) and that's a problem? How? What's the problem?

 

When you create something called Go and DISTRIBUTE it, and it’s not compatible with Go, you can’t call it Go, nor replace a standard Go

Where does the license say that?

 

Maybe a misinterpretation of the clause:

  Neither the name of Google Inc. nor the names of its

  contributors may be used to endorse or promote products derived from

  this software without specific prior written permission

> [...]

Manlio Perillo 

[Robert Engels]

Sorry, I wasn’t clear. I wasn’t saying that the license prohibits it, just the opposite, it allows it, and that’s a problem... you can review the sun v ms to see the arguments against allowing this, and the courts agreed (is was specifically prohibited by the license, but why suns arguments that it should be upheld). 

The fact that the Go license allows for it will probably cause interoperability and compatibility problems down the road. Looks like it is already starting (a distributed Go that won’t implement the distribution verification...)

--

[Brad Fitzpatrick]

On Thu, Mar 14, 2019 at 2:16 PM Robert Engels <ren...@ix.netcom.com> wrote:

Sorry, I wasn’t clear. I wasn’t saying that the license prohibits it, just the opposite, it allows it, and that’s a problem... you can review the sun v ms to see the arguments against allowing this, and the courts agreed (is was specifically prohibited by the license, but why suns arguments that it should be upheld). 

The fact that the Go license allows for it will probably cause interoperability and compatibility problems down the road. Looks like it is already starting (a distributed Go that won’t implement the distribution verification...)

I'm not terribly worried. I think we'll all figure it out before it comes to that, finding something that meets the needs of upstream, distros, and users.

 

[Jakub Cajka]

----- Original Message -----
> From: "Manlio Perillo" <manlio....@gmail.com>
> To: "golang-dev" <golan...@googlegroups.com>

> Sent: Thursday, March 14, 2019 12:58:50 PM
> Subject: Re: [golang-dev] proposal: public module authentication with the Go notary
>
>
>

Maybe something like nixpkgs/nixos? https://github.com/NixOS/nixpkgs I have just discovered that. It is bit different approach to what rpm/dep and Linux distros did/are doing. Even it seems to blend some parts reassembling the proposed notary. But I guess we are straying bit too much off topic for golang-dev :).

JC

[Nicolas Mailhot]

Le jeudi 14 mars 2019 à 04:58 -0700, Manlio Perillo a écrit :
>
> I find it strange that in all these years nobody decided to build a
> "system" packages registry, where package names are independent from
> OS package names.
> A separate registry can be used to map standard package names to OS
> specific names.
>
> A Go module can then use a "foreign depends" directive, to state the
> system packages required, and that can be installed by a separate
> command.

I can tell you why this didn't happen and isn't likely to happen soon:

A. Except for the most simple cases, a computer project is not a single-
tech single-component atom, it's a mix of things that need to be managed
individually.

B. reality is messy. A lot of those things have no language-specific
manager (stuff which is not computer code, generic enough not to have a
single steward authority like C/C++, things like protobuf). So there are
no "standard package names" to map to "os package names" for a lot of
things contained in computer projects. Yet they still need some form of
management. Integrators have to invent buckets to put things in. They
don't invent the same buckets or the same bucket layouts.

C. upstreams are nowhere near as clean as they feel they are, they will
mix things which do not belong to them in their own "pure-tech"
components (not belong = "other project" files, or files that are really
foreign tech), because manual copy is a lot easier than setting up a
clean automated supply chain. So even when "standard package names"
exist the "standard packages" are not clean atoms, they need untangling
it's not a one to one or even a one to many relationship, it's a lot
more complex than that

D. you have cross-tech constrains (deps between atoms of different
tech). Expecting to call a set of language-specific package managers,
with no coordination between their various dependency graphs, is
unlikely to produce working results for a large set of real-world
software. You need to map components into a single transaction engine,
not execute engines in parallel. Thanksfuly there are not so many ways
to manage versioning constrains in a software project, so it's usually
possible to express language-specific versioning constrains in a generic
way, but sometimes someone has been “creative” and the versioning spills
in the os package names.

All this assuming upstream projects that work perfectly as is, and do
not need specific distributor surgery to work at all in a managed
context.

And yes once all this is said and done you could imagine mapping a high-
level project id to a low-level list of os components (that will
themselves pull in a set of deps). But, what is a high-level project id?
Project A (people) release one zip in may and two in august. Does that
means the project split? Something else? It's a huge rabbit hole.

--
Nicolas Mailhot

[David Golden]

Hi, Nicholas.  This has been a very long thread to follow, so apologies in advance if I've taken an oversimplified view of things.

I'm trying to understand that crux of the issues and I feel there are really two, slightly-related ones:

(a) distribution patched modules will have signatures that don't match the published signatures in upstream repo go.sum files or the public notary

(b) go metadata examination tools (e.g. `go list -json -mod=readonly ./...`) fail without internet access; i.e. they fail if they can't fully analyze the dependency tree even if only local information is requested

I think the solution to (a) could be that distributions need to GOPROXY a notary as well as modules -- possibly starting with a copy of the public notary so that any lookups can be resolved locally without network access.  As the build process adds modules to the local GOPROXY, they also update the go.sum lines in the GOPROXY notary.  Part of the process of cleaning up upstream code and putting it into the local GOPROXY would be to remove an existing go.sum file and regenerate it using the GOPROXY notary for patched dependencies.

I think the solution to (b) could be having "go list" (and other tools) operate in a lazy or degraded mode either when a network isn't available or on request by a flag (`-local-only`).  If someone wants to use the tool to just get a list of local information, they don't need the tool to resolve the full, deep dependency information tree "just in case".  The '-find' flag is an example of a mode that does this: `go list -mod=readonly -find ./...` doesn't error in the repro case.  But I could imagine a package wanting to use `go list` to produce a list of shallow dependencies for further processing and `-find` doesn't populate the Imports field.  Shallow examination of local files should be possible without network access and `-mod=readonly` only covers part of the need.

Are there other problems I've overlooked?  Do my proposed solutions seem feasible and do you think they would address the issues?

Regards,

David Golden

[Nicolas Mailhot]

Le vendredi 15 mars 2019 à 11:13 -0400, David Golden a écrit :

> Hi, Nicholas.  This has been a very long thread to follow, so
> apologies in advance if I've taken an oversimplified view of things.

 

Hi David

 

Thank you for making the effort to understand the thread

 

> (a) distribution patched modules will have signatures that don't
> match the published signatures in upstream repo go.sum files or the
> public notary

>

> (b) go metadata examination tools (e.g. `go list -json -mod=readonly
> ./...`) fail without internet access; i.e. they fail if they can't
> fully analyze the dependency tree even if only local information is
> requested

 

That's two of the main issues. The third big issue is that we need a way to break dependency cycles for the initial build of things that participate in a cycle. Breaking cycles often involves heavy surgery to one of the links in the cycle (much more than the usual light massaging and patching).

 

Also, I have no idea if the problems that manifest in go metadata examination tools are also presenty in tools used later in the build process. I haven't managed to progress so far.

 

> I think the solution to (b) could be having "go list" (and other

> tools) operate in a lazy or degraded mode either when a network isn't

> available or on request by a flag (`-local-only`).  If someone wants

> to use the tool to just get a list of local information, they don't

> need the tool to resolve the full, deep dependency information tree

> "just in case". 

 

I basically agree, but that implies tooling fixes

 

> I think the solution to (a) could be that distributions need to
> GOPROXY a notary as well as modules -- possibly starting with a copy
> of the public notary so that any lookups can be resolved locally
> without network access.

 

I don't think you realize how hard that would be to implement, even if we wanted to (and we don't).

 

For security and reproducibility reasons, a build is cut from the network. That means notarizing anything can not happen within the go-aware build environment, it must be done by the build manager.

(if it happened within the build environment that would be no better than the ziphash security-wise).

 

Because rpm is an old mature ecosystem, there are several of those on the market. We use at least four of them within Fedora, at various stages of the package onboarding process (direct uncontained local builds, lightweigh system on private packager computers, lax and less secure for packaging experiments, very strict for production builds). 

 

That's only the "official" Fedora build managers, an entity wishing to participate in the Fedora/Centos/RHEL ecosystem can use other solutions locally for the packages it adds over or contributes to Fedora/Centos/RHEL (the contributed packages are rebuilt in our own systems before acceptance). I'm pretty sure some corps use non-free-software solutions like artifactory to produce their rpm packages, for example.

 

So, implementing notarization would require modifying several build managers, managed by several entities, before we can be use it.

 

Right now none of those managers needs to be aware of the technical content produced by builds, it's just a bunch of rpm files with standard rpm metadata for them.

 

So each of the build managers owners would need to be convinced to implement a language specific plugin system just for go (at least the 4 systems we use, Suse would need to modify its owns, etc etc). Unless you get them to agree on a common plugin API (which takes design time, and consensus, not to mention the build managers can be written in different languages) the resulting plugin systems will be incompatible with each other.

 

Assuming this is done (and that won't happen, because every build manager upstream first second and last reaction will be "you are out of your mind, go away") you still need to write the plugins, and get them vetted, and get them deployed on build infrastructures.

 

Speaking of which that also requires running a notarizing infrastructure. 

 

This is hard and expensive to do securily and at scale. "Just delegate it to Google" (of Fedora, or Red Hat) won't cut it because our userbase is deeply attached to the freedom to do its own thing without asking Google, or Fedora, or Red Hat, permission.

 

So another requisite to deploy such a configuration with our userbase (and one that will be raised by build system owners, if you manage to convince them to listen at all) would be to ensure rolling your own notary is easy. That means an easy to build and deploy DIY notary implementation, that scales from garage software lab needs to massive high availability build farms (like the ones implemented by Fedora). DIY notary means independent from Google, or Red Hat, or Fedora, no network strings attached.

 

Independent probably implies layered, so the modules produced by Fedora are attested by the Fedora Notary, and the modules produced by someone else over Fedora use a separate system

 

Then (did I already say builds are cut from the network), you need another process to inject the notary attestations in the build environment. That means either another go-specific build manager plugin, or repackaging as system components a dump of the attestations of each notary, every time something new is attested.

 

Lastly, assuming that by some miracle you get all those things done, there's no way the whole notary roundtrip will be free. It will cost time and computer resources. For this reason alone it will be unacceptable for entities like Fedora, where "mass rebuild" is massive and takes weeks over scores of builders in different hardware architectures. "Just a little bit longer" means "system release slips by a week or more" when multiplied by the amount of building we do, so that won't happen.

 

And, BTW, none of this actually serves any purpose security wise for our userbase, because every single file we ship to users is already hashed and signed in a generic, non language-specific way.

 

> What is the component owning this file?

 

$ rpm -q --whatprovides /usr/lib/golang/api/go1.1.txt

golang-1.12-1.fc31.x86_64

 

> What key signed this component

 

$ rpm -qi golang-1.12-1.fc31.x86_64

Name        : golang

Version     : 1.12

Release     : 1.fc31

Architecture: x86_64

Install Date: Fri 08 Mar 2019 11:54:36 AM CET

Group       : Unspecified

Size        : 7252427

License     : BSD and Public Domain

Signature   : RSA/SHA256, Wed 27 Feb 2019 10:17:30 PM CET, Key ID 50cb390b3c3359c4

Source RPM  : golang-1.12-1.fc31.src.rpm

Build Date  : Wed 27 Feb 2019 09:39:22 PM CET

Build Host  : buildhw-10.phx2.fedoraproject.org

Relocations : (not relocatable)

Packager    : Fedora Project

Vendor      : Fedora Project

URL         : http://golang.org/

Bug URL     : https://bugz.fedoraproject.org/golang

Summary     : The Go Programming Language

Description :

The Go Programming Language.

 

It's not blind notary signing, it was signed by a specific entity for a specific purpose. You can authorize several keys on a system, and signing entities can use different keys for different purposes (Fedora typically changes its key for every major release, so replay attacks from old or experimental components are blocked).

 

> Does the on-disk state match the recorded and signed hashes?

 

$ rpm -V golang-1.12-1.fc31.x86_64

<list of eventual discrepancies>

[David Golden]

On Fri, Mar 15, 2019 at 1:33 PM Nicolas Mailhot <nicolas...@laposte.net> wrote:

> I think the solution to (b) could be having "go list" (and other

> tools) operate in a lazy or degraded mode either when a network isn't

> available or on request by a flag (`-local-only`).  If someone wants

> to use the tool to just get a list of local information, they don't

> need the tool to resolve the full, deep dependency information tree

> "just in case". 

 

I basically agree, but that implies tooling fixes

Exactly. I was trying to scope the request for changes to the tools.

 

 

> I think the solution to (a) could be that distributions need to
> GOPROXY a notary as well as modules -- possibly starting with a copy
> of the public notary so that any lookups can be resolved locally
> without network access.

 

I don't think you realize how hard that would be to implement, even if we wanted to (and we don't).

We may be misunderstanding each other. My understanding of GOPROXY is that it can be "file://path/to/local/stuff" and that already your idea for how to build Go modules is to patch them and inject them into a local GOPROXY directory -- that's how you can provide a locally-patched version of the upstream transparently.  I believe that the notary proposal also merely requires that certain endpoints exist, e.g. "lookup/M@V" and "/record/R".  So it seemed that just as you envision packaging up the modified source into the GOPROXY directory, you would only need to compute the necessary checksum information and update the GOPROXY directory.

Certainly, that would require tooling that doesn't exist today to run what is effectively a "command line notary".

If that doesn't make sense to you, then could you please elaborate more on what you mean in your example of a "system GOPROXY populated with distro-produced go modules corresponding

to identified project requirements" and so forth?  If one can populate a GOPROXY with distro-produced go modules, I don't see why one can't also populate a GOPROXY with distro-produced checksum information.

It's possible that the transparent logs design doesn't translate well to local operation, or that the go build tools require more than the per-module checksum information, but if the model I've described of publishing source and checksum to GOPROXY could work for your build processes, then we begin to have requirements for amending the notary proposal to support that sort of offline operation.

Regards,
David

[Ian Denhardt]

Quoting Jakub Cajka (2019-03-15 04:05:58)

> > I find it strange that in all these years nobody decided to build a
> > "system" packages registry, where package names are independent from OS
> > package names.
> > A separate registry can be used to map standard package names to OS
> > specific names.
> >
> > A Go module can then use a "foreign depends" directive, to state the system
> > packages required, and that can be installed by a separate command.
> >
> >
> > Manlio Perillo
>
> Maybe something like nixpkgs/nixos? https://github.com/NixOS/nixpkgs I
> have just discovered that. It is bit different approach to what
> rpm/dep and Linux distros did/are doing. Even it seems to blend some
> parts reassembling the proposed notary. But I guess we are straying
> bit too much off topic for golang-dev :).

Nix is kindof a different thing; it's a distro in its own right, where
as I think Manlio was talking about something that would be distro
agnostic.

That said, folks definitely do use Nix in the wild to wrangle the
foreign dependencies for their projects during development, and it works
well in practice due to a number of features missing from most system
package managers (but present in most language package managers):

* You can have multiple versions of a package installed at once,
isolated from one another.
* You can build/install packages as an unprivileged user (though things
like setuid binaries won't work, for obvious reasons -- but it's good
enough for most things).

-Ian

[Nicolas Mailhot]

Le vendredi 15 mars 2019 à 14:26 -0400, David Golden a écrit :

 

Hi David,

 

> > I don't think you realize how hard that would be to implement, even
> > if we wanted to (and we don't).

> >

>

> We may be misunderstanding each other. My understanding of GOPROXY is
> that it can be "file://path/to/local/stuff" and that already your
> idea for how to build Go modules is to patch them and inject them
> into a local GOPROXY directory -- that's how you can provide a
> locally-patched version of the upstream transparently.

 

So far so good

 

> I believe that the notary proposal also merely requires that certain
> endpoints exist, e.g. "lookup/M@V" and "/record/R". 

 

I haven't looked into detail, but yes creating a file structure in a specific directory is doable, even though it's a lot of work for something we and our users have zero use for, because security file verification is already provided by other (stronger, more complete and more generic) means.

 

> So it seemed that just as you envision packaging up the modified
> source into the GOPROXY directory, you would only need to compute the
> necessary checksum information and update the GOPROXY directory.

 

The computation is the difficult part.

 

For the result to function as notary you need something to attest the hashes are "true" (ie trust).

 

If the trust function is assured by modern cryptography, that means there is a secret somewhere, and to implement such a setup, you need the whole house of cards I detailed in my previous message. The only alternative would be to make the secret file part of the build sources. And, our builders do not protect those from inspection and copy, quite the contrary. Remember, we do free software. And besides transparency is good from a security auditing POW.

 

If the trust function is assured by the fact the files are in a specific directory, writeable only by root (because if something can hijack read-only root files on your system you have more critical things to worry about than go source hashes), the notary files are no better than the system ziphash files (that will be ro root files in a specific directory). And the whole setup becomes a very convoluted and complex way to copy the ziphash content somewhere else, just because the go commands are too dumb to realize the original is strictly equivalent to the copy from a security POW. Plus, how to you handle staging module files? You can not run their tests if the go commands refuse to read them.

 

Lastly putting even a finger in the notary grinder would lead to the mass replacing horror, because go modules have not provided a mean to mark a module as equivalent to upstream, with fixes, so a "working" notary setup implies you rename or renumber everything you produce, and then replace every single dep in other module descriptors to point to your renamings and renumberings. And, we are not going to mass replace deps.

 

Our organisation scales to many components and many contributors because it trusts implicitly the other parts of the organisation (humans and the technical bits they produce). Explicit mass replacing would add a giant centralized lock in the middle of a very complex structure, with moving parts right and left.

 

-- 

Nicolas Mailhot

[David Golden]

On Sat, Mar 16, 2019 at 4:46 AM Nicolas Mailhot <nicolas...@laposte.net> wrote:

I haven't looked into detail, but yes creating a file structure in a specific directory is doable, even though it's a lot of work for something we and our users have zero use for, because security file verification is already provided by other (stronger, more complete and more generic) means.

My understanding of what you've said is that the distribution only builds from sources that are curated.  Is that correct?  If, during a build of "Foo", you discover that it needs "Bar", but Bar is not curated, the build breaks until Bar is added as a curated module?  And every curated module is hand-vetted (i.e. creating an RPM spec file)?

So I have a couple further questions:

(a) How is the trust in the upstream source established?  Are the builds repeatable and if so, how is the trust established during repeated builds?  For example, is the upstream source manually copied into a trusted repository and frozen until updated?  Or does the build process download/patch the source fresh each time?

(b) When building a package solely from curated source RPMS, is the trust then fully established from signatures of the RPMs?  If so, then would the `GONOVERIFY=*` setting (to disable the notary) work to just bypass Go-ecosystem trust mechanisms entirely, as you're already operating on trusted source?

Depending on your answers, I wonder if the "solution" for distributions is just these two things:

* go tools need to add a "local-only" mode (as previously discussed)

* distribution build systems need to set `GONOVERIFY=*` to disable notary/go.sum functions and rely on the distro's existing trust model for curated source

Regards,

David

[Manlio Perillo]

On Sat, Mar 16, 2019 at 5:02 PM 'David Golden' via golang-dev
<golan...@googlegroups.com> wrote:
>
> [...]

> Depending on your answers, I wonder if the "solution" for distributions is just these two things:
>
> * go tools need to add a "local-only" mode (as previously discussed)
> * distribution build systems need to set `GONOVERIFY=*` to disable notary/go.sum functions and rely on the distro's existing trust model for curated source
>

Note that notary is not consulted if GONOVERIFY=off or go get is
invoked with the -insecure flag (but AFAIK, modules are not obtained
with go get).
However cmd/go will still check the go.sum. The check is not done
only with -mod=vendor module download mode flag.
A possible solution is to add another mode, like -mod=noverify.


Manlio Perillo

[David Golden]

On Sat, Mar 16, 2019 at 3:01 PM Manlio Perillo <manlio....@gmail.com> wrote:

> * go tools need to add a "local-only" mode (as previously discussed)
> * distribution build systems need to set `GONOVERIFY=*` to disable notary/go.sum functions and rely on the distro's existing trust model for curated source
>

Note that notary is not consulted if GONOVERIFY=off or go get is
invoked with the -insecure flag (but AFAIK, modules are not obtained
with go get).
However cmd/go will still check the go.sum.  The check is not done
only with -mod=vendor module download mode flag.
A possible solution is to add another mode, like -mod=noverify.

Thank you for clarifying those details.  Possibly, the distribution curation process would remove the existing go.sum files, since they would be irrelevant for checking other curated (and thus modified) modules.  But having an option to completely opt out of trust checking via another flag or environment variable would make that easier for users who don't need the trust mechanism.

[Nicolas Mailhot]

We will most certainly remove the upstream go.sums, remove indirect
requires from go.mod, remove local replaces, (maybe, not sure) remove
non-local replaces.

Ideally, at some point of the process, there should be a call to
something that checks that direct requires have not changed after the
patching. Not sure how to do it safely, without triggering all the
indirect logic and sum writing

Regards,

--
Nicolas Mailhot

[Manlio Perillo]

On Sat, Mar 16, 2019 at 8:20 PM David Golden <da...@mongodb.com> wrote:
>
> [...]

>
> Thank you for clarifying those details. Possibly, the distribution curation process would remove the existing go.sum files, since they would be irrelevant for checking other curated (and thus modified) modules.

If you remove the go.sum file, cmd/go will populate it again taking
the hash from the cache.
Since it is very fast, I assume that it take them from the
`v<semver>.ziphash` file in the download directory.

Can I assume that cmd/go has full trust on the cache content?


Manlio Perillo

[Nicolas Mailhot]

Le vendredi 15 mars 2019 à 11:13 -0400, David Golden a écrit :
> Hi, Nicholas. This has been a very long thread to follow, so
> apologies in advance if I've taken an oversimplified view of things.

Hi David

Thank you for making the effort to understand the thread

> (a) distribution patched modules will have signatures that don't
> match the published signatures in upstream repo go.sum files or the
> public notary
>
> (b) go metadata examination tools (e.g. `go list -json -mod=readonly
> ./...`) fail without internet access; i.e. they fail if they can't
> fully analyze the dependency tree even if only local information is
> requested

That's two of the main issues. The third big issue is that we need a
way to break dependency cycles for the initial build of things that
participate in a cycle. Breaking cycles often involves heavy surgery to
one of the links in the cycle (much more than the usual light massaging
and patching).

Also, I have no idea if the problems that manifest in go metadata
examination tools are also presenty in tools used later in the build
process. I haven’t managed to progress so far.

> I think the solution to (b) could be having "go list" (and other
> tools) operate in a lazy or degraded mode either when a network isn't
> available or on request by a flag (`-local-only`). If someone wants
> to use the tool to just get a list of local information, they don't
> need the tool to resolve the full, deep dependency information tree
> "just in case".

I basically agree, but that implies tooling fixes

> I think the solution to (a) could be that distributions need to
> GOPROXY a notary as well as modules -- possibly starting with a copy
> of the public notary so that any lookups can be resolved locally
> without network access.

I don't think you realize how hard that would be to implement, even if
we wanted to (and we don't).

[Jakub Cajka]

----- Original Message -----
> From: "Ian Denhardt" <i...@zenhack.net>
> To: "Jakub Cajka" <jca...@redhat.com>, "Manlio Perillo" <manlio....@gmail.com>
> Cc: "golang-dev" <golan...@googlegroups.com>
> Sent: Friday, March 15, 2019 10:00:15 PM
> Subject: Re: [golang-dev] proposal: public module authentication with the Go notary
>

My understanding has been that nixpkgs in similar to the yum/dnf/apt/deb/rpm effectively independed from any distro. NixOS is distro based on nixpkgs. And it is strictly based on hashing and verifying all the content that it builds/distributes and sync local state with it, kind of as notary will enforce on Go for a source at least. But I can be getting that wrong, it is really new thing for me that I have started to learn just recently.

[Ian Denhardt]

Quoting Jakub Cajka (2019-03-18 05:08:35)

> My understanding has been that nixpkgs in similar to the yum/dnf/apt/deb/rpm effectively independed from any distro. NixOS is distro based on nixpkgs. And it is strictly based on hashing and verifying all the content that it builds/distributes and sync local state with it, kind of as notary will enforce on Go for a source at least. But I can be getting that wrong, it is really new thing for me that I have started to learn just recently.

This is mostly correct, but the relationship with the notary is
different; what Nix does is closer to the role of go.sum. You can
add & authorize 'binary caches', which serve pre-built packages, but
that is greater in scope than what the notary does.

But what I was trying to get at was that I think Manlio was talking
about something else -- basically a registry of 'canonical' names for
things not in a language package manager. This would allow just saying
"foreign-depends = ..." without having deal with, for example, the fact
that fedora calls packages with headers in them *-devel, while debian
calls them *-dev. Whereas Nix just packages everything up again, so you
have yet another competitor.

-Ian

[Nicolas Mailhot]

Le lundi 18 mars 2019 à 14:04 -0400, Ian Denhardt a écrit :

Hi,

> But what I was trying to get at was that I think Manlio was talking
> about something else -- basically a registry of 'canonical' names for
> things not in a language package manager. This would allow just
> saying
> "foreign-depends = ..." without having deal with, for example, the
> fact
> that fedora calls packages with headers in them *-devel, while debian
> calls them *-dev. Whereas Nix just packages everything up again, so
> you have yet another competitor.

The foreign-depends concept does not work in real life because the
dependency graphs of different things are not independant. python
things will depend on C things that will depend on perl things that
will depend on Go things that will depend on python things and do on.

As soon as one of the other graphs depends back on the original graph
you get a lockup.

You can pretend things are independant for a while, by stacking up
multiple levels of vendoring, but ultimately you need to import the
whole graph in a single solver to resolve the constrains (exactly like
go did with modules).

What could work, is to define canonical names and syntax, so you could
forget different entities use different solvers, and just ask the same
thing the same way everywhere.

But, that requires some entity to classify everything into useful
atoms, and to name those atoms. This is a gigantic task. Many things
have no clear name or boundary today, and people disagree of where the
atom boundaries should be.

Regards,

--
Nicolas Mailhot

[Ian Denhardt]

Quoting Nicolas Mailhot (2019-03-18 16:20:57)

> But, that requires some entity to classify everything into useful
> atoms, and to name those atoms. This is a gigantic task. Many things
> have no clear name or boundary today, and people disagree of where the
> atom boundaries should be.

I agree; this is the hard part, and I don't really think it's worth the
trouble.

Re: your other points, you would of course need tooling for this stuff
to be any use, and probably part of that would be a language-independent
solver, yes.

[Nicolas Mailhot]

Le 2019-03-19 20:20, Florian Weimer a écrit :
> * via golang-dev:
>
>> Le 2019-03-13 21:37, Florian Weimer a écrit :
>>> But do users who run the go command currently see this?
>>>
>>> I thought that it's strictly for internal use, for building Go
>>> applications for the distribution itself. So the users here would be
>>> Fedora developers. It's kind of like golang.org/x, but using
>>> Fedora/RPM
>>> technology.
>>>
>>> I assume that Go programmers using the Fedora Go packages would still
>>> obtain whatever upstream Go would download using “go get”, even if
>>> the
>>> sources are also packaged for Fedora's own (internal) use.
>>
>> Not necessarily. This direct upstream fetching is often used in
>> addition and as a complement to distribution-proofed code, not as a
>> complete substitute.
>
> So if the system administrator installs some Go-code-containing
> package, the results of my build procedures change because a system
> version of a library is picked up, instead something from the
> Internet?
>
> That's rather surprising.

Well if you build software on a particular system and it loses company
data because your Go tooling avoided using a system Go module that
contained last-mile dataloss fixes with the kernel used by that
particular system, that's rather surprising too.

Things are not black and white.

Other packaging software deals with that by having explicit component
sources configurations, so the user can specify what it wants to target,
and easily switch targets, without replacing project by project and
component by component.

/etc/golang/module-sources.d/system.conf
[system]
src=/usr/share/golang/modules/
notary=""
enabled=true

[system-pre]
src=/usr/share/golang/prerelease-modules/
notary=""
enabled=false

/etc/golang/module-sources.d/google.conf
[google]
src=https://go.google.com/modules/
notary=https://go.google.com/notary/
CA=path-to-google-hash-signing-CA
enabled=false

[google-cloud]
src=https://cloud.google.com/go/modules/
notary=https://go.google.com/notary/
CA=path-to-google-hash-signing-CA
enabled=false

/etc/golang/module-sources.d/google.conf
[internet]
src=""
notary=https://golang.generic.com/notary/
enabled=false

~/config/golang/module-sources.d/staging.conf
[staging]
src="~/test"
notary=""
enabled=true

go build //default conf in config files

go build --with-modules=internet //temporarily enable internet pull, to
do a mixed build

go build --without-modules=system* //internet and local staging only

go build --without-modules=* --with-modules=internet // full generic
internet build

(tastes in config syntax and flag naming obviously vary, but you get the
idea)

--
Nicolas Mailhot

[Robert Engels]

Isn’t is also possible that the last mile introduces a bug that the original developer has already worked around and so causes the problem.

I find it difficult to believe that a team of 20 can review, verify and apply the patches for all software known to man...

Personally I would rather have the author/maintainer applying and validating changes not sone less knowing 3rd party.

[Nicolas Mailhot]

Le 2019-03-20 13:00, Robert Engels a écrit :
> Isn’t is also possible that the last mile introduces a bug that the
> original developer has already worked around and so causes the
> problem.

Sure, it's possible, no human creation is perfect.

> I find it difficult to believe that a team of 20 can review, verify
> and apply the patches for all software known to man...

And yet all the software you use in real life went through this
integrator process, either via free software community integration
organizations like Linux distributions, or via proprietary integration
organizations run by Microsoft, Apple, Google, Samsung, etc

The number of people available integration-side is a direct function of
the software reach. The Go reach right now is not so good because Go is
relatively new, and Go tooling makes it hard to be involved except at
the individual not-caring-about-production dev level or
big-internal-team-dedicated-to-go-with-specific-tooling level.

> Personally I would rather have the author/maintainer applying and
> validating changes not sone less knowing 3rd party.

The difference between the author/maintainer and integration people is
that the integration people need to get it to work NOW on real systems
and real hardware while the author/maintainer may not care about your
system or hardware, or is feuding with the author/maintainer of one of
the other bits you need, or finds the fix not worth the effort, or is
clueless about some things (many software authors reuse material in
their software they do not understand perfectly), or has other plans
that do not involve working on the problem you hit today.

Try to use any complex collection of software and you will hit cases
where the author/maintainer of one part is not available or helpful when
you need him to be.

That's why real-world deployments insert a last-mile fixing layer. The
last mile layer is always intended to be as minimal as possible. It
tends to grow when upstream is not helpful, and shrink otherwise. It
basically never reaches zero. That, even when the people working
integration-side are themselves the authors of the things they
integrate.

--
Nicolas Mailhot