do you use binary-only packages?

[Russ Cox]

The go command supports "binary-only packages", in which 

the compiled .a file is installed in GOROOT/pkg or GOPATH/pkg

but the corresponding source code is only a stub with relevant

imports (for linking), a special comment marking it as a binary-only

package, and perhaps documentation for exported API.

While the go command will use these packages if they exist in

GOROOT or GOPATH, 'go get' will not download them: 'go get'

is intentionally only about source code distribution.

Furthermore, binary-only packages only work when the build is

using the exact versions of whatever imported packages are

needed by the binary-only package. If there were any important

differences in a dependency between compilation and linking,

the binary-only portion might have stale information about details

of the imported package, such as type sizes, struct field offsets,

inlined function bodies, escape analysis decisions, and so on,

leading to silent memory corruption at runtime.

Binary-only packages also only work when the build is using the

exact version of the Go toolchain that was used for compilation.

This is at least enforced at link time, with the effect that if your

binary-only package only imports the standard library and you don't

use any special compilation flags, then the toolchain version check

suffices to avoid the silent memory corruption problem described in

the previous package. But if your package imports any non-standard

package for which that the eventual user might try to combine with

a different version, you're in trouble again.

Compiled Go programs also contain much richer amounts of runtime

information than compiled C programs, so that for example all the

details of type declarations, including struct definitions, are in the

compiled code, along with file names and line numbers for the compiled

code, and increasingly good debug information that was impossible to

turn off until very recently. Overall, a binary-only package obscures very little.

In short, binary-only packages:

- have never been supported by 'go get',

so you've had to go out of your way to use them,

- aren't even guaranteed to work when you do use them,

possibly leading to silent memory corruption, and

- still expose quite a lot more than most people would expect

about the original source code.

For these reasons, we're looking at removing binary-only package

support entirely.

If you use binary-only packages and think it's important to keep that

support around, please let us know either on this thread or at

golang.org/issue/28152. 

Thanks.

Russ

[Andrey Tcherepanov]

Hello,

I have to agree that binary packages In the current form are not very useful, at least to us. We have played with an idea of having sqlite (C + a Go wrapper on top) wrapped up into a binary package, so we can continue to build multiple platforms from a single build machine. As result of investigation, that deemed to have more hassle than to have all platforms built separately even with pains of CGO involved.

IMHO, precompiled (pre-built) libraries are nice to have, be that "platform-dependent binary package" or some platform-agnostic jar file. I think a success of Java as a platform in no small part could be attributed to the fact that jar's are so easy to create, distribute and use (minus obvious dependency hell that we all have to go through periodically).

So far what I can think of

• There are places that cannot distribute a library in source code format, but would like to distribute a library.
  
• Pre-built packages can be used to reduce build time (not sure how important this is with 1.10+ cache)
  
• Distribution of signed packages? I think it is not easy thing to do with "just" a source code, unless it comes in a bundled form like a signed zip file...
  

Thinking a bit about "richness of meta-info" vs "cannot distribute source code" and how Java and .NET are doing this... Would replacement of "pure" compiled-and-ready-to-be-linked  library packaging with just compiled and compressed AST (probably lightly optimized) be a better solution? 

This would give 

1. a rich (full?) meta info for optimization
   
2. a way to distribute code without source code for people who have to do it that way.
   
3. a bit of platform independence if all of suffixed files ("*_windows.go" , "*_linux.go" etc) are parsed in.
   

I think it would also give some Go version independence, covering cases when library was built with older tools, but needs to be assembled with a newer Go distribution. This might work if...

1. AST is a version independent (sorry, I don't know if it is), 
2. AST is platform-agnostic
3. has a forward compatible format or tools reading it are easy to make backward-compatible

I am probably missing something obvious here (I don't think I am the first one to think about it, and I am in no way a language expert), but wanted to share this idea.

Thanks,

   Andrey

[Ian Lance Taylor]

On Thu, Oct 18, 2018 at 1:10 PM, Andrey Tcherepanov
<xnow4f...@sneakemail.com> wrote:
>
> Would replacement of "pure"
> compiled-and-ready-to-be-linked library packaging with just compiled and
> compressed AST (probably lightly optimized) be a better solution?

Thanks. It's a fine approach but it's a considerable amount of design
and implementation, and it's not completely clear it's possible at
all. If someone wants to make implementing such a scheme a
longer-term goal, that would be interesting. But discussing that will
distract us from Russ's question, which is: should we continue to
support binary packages approximately as they exist today?

Ian

[Tharaneedharan Vilwanathan]

Hi All,

I have a quick question. I hope I am not off topic.

If this happens, by design, Go will not allow, say, a middleware company to provide binary only distribution?

Regards

dharani

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ian Lance Taylor]

On Thu, Oct 18, 2018 at 2:20 PM, Tharaneedharan Vilwanathan
<vdha...@gmail.com> wrote:
>
> If this happens, by design, Go will not allow, say, a middleware company to
> provide binary only distribution?

Yes. There would be no way to for a company to provide a binary-only
package to its users. They would have to provide source code.

Ian

[Tharaneedharan Vilwanathan]

Hi Ian,

Thanks for the clarification!

This means source-code is the only way to share the work. When it companies to sharing/selling their work on top of which others can build their app/solution, this won't work. Doesn't this seem like a big restriction? Particularly, computer industry being heavily dependent on IP rights (and where trust is low)? Wouldn't this deter such companies from adopting Go? For contrast, I have heard of providing binary only distribution even within the same company.

Just thinking aloud.

Regards

dharani

[Ian Lance Taylor]

On Thu, Oct 18, 2018 at 4:02 PM, Tharaneedharan Vilwanathan
<vdha...@gmail.com> wrote:
>
> This means source-code is the only way to share the work. When it companies
> to sharing/selling their work on top of which others can build their
> app/solution, this won't work. Doesn't this seem like a big restriction?
> Particularly, computer industry being heavily dependent on IP rights (and
> where trust is low)? Wouldn't this deter such companies from adopting Go?
> For contrast, I have heard of providing binary only distribution even within
> the same company.

The question is: is anybody actually doing this? Is anybody seriously
thinking about it?

Ian

[ancientlore]

In the past (C/C++ days) we used a lot of third-party binary-only packages like SmartHeap for memory management and LeadTools for imaging. (I like to think of that as the "Programmer's Paradise" era.) I haven't used binary-only packages in many years, but I do wonder how a company like LeadTools might make their offering available to Go developers. Presumably, they don't want to release their source code. It might be informative to get the opinions of companies like that.

Mike

[Tharaneedharan Vilwanathan]

Hi Ian,

Good question. I don't have an answer.

But this raises some questions:

Should Go be anticipating many such possibilities today and tomorrow? Or shut them off?

With this limitation, doesn't it look like Go is best suited for building end products or service but one cannot build on top of some other's work (with direct API access) without having access to source code? Are we indirectly making Go well suited for (a) building end products or service (as mentioned above) or (b) specific areas (typically research, education, non-commercial) where we can create building blocks inside/outside the company and the code will be shared as well.

One problem I see is we have a better chance of knowing who adopted Go for what but not quite on who could not adopt for what reason.

Regards

dharani

[Tamás Gulácsi]

Providing a plugin (I prefer a separate binary called through rpc, but the native .so is ok, too) is not a solution?

[Andrey Tcherepanov]

Sorry, I did not mean to hijack the discussion.

To summarize our answer on Russ's question - in a current form Go's binary packages are not useful for our team.

Thanks,

  Andrey

[Beoran]

I'd say it's a useful feature, but not in it 's current form. I rather than dropping the featture I'd like to see it enhanced so so it works as well as a pascal .unit file does as a binary package for a high level language. This also points to a solution as well, have a .gobin file that is an archive that contains all the meta information and the compiled .a and/or .so file as well.

[Sam Mortimer]

On Thursday, October 18, 2018 at 4:28:23 PM UTC-7, Ian Lance Taylor wrote:

The question is: is anybody actually doing this?  Is anybody seriously
thinking about it?

Ian

Unhelpfully, I imagine it unlikely that anyone distributing binary go packages reads golang-dev or golang-nuts.

-Sam.

[Ian Lance Taylor]

On Fri, Oct 19, 2018 at 1:14 AM, Sam Mortimer <sam.mo...@gmail.com> wrote:
>
> On Thursday, October 18, 2018 at 4:28:23 PM UTC-7, Ian Lance Taylor wrote:
>>
>> The question is: is anybody actually doing this? Is anybody seriously
>> thinking about it?
>

> Unhelpfully, I imagine it unlikely that anyone distributing binary go
> packages reads golang-dev or golang-nuts.

Is there a more likely place to reach such people?

Ian

[Scott Cotton]

Hi dharani,

I don't think the bigger questions about end products or services are so much implied by the question.  

BOPs are just one form of binary that have some problems for users and maintainenance, and there are workarounds to supplying a package with binary dependencies that don't suffer from these problems in any case (eg plugins).

Scott   

[Harald Weidner]

Hallo,

On Fri, Oct 19, 2018 at 11:45:04AM -0700, Ian Lance Taylor wrote:

> > Unhelpfully, I imagine it unlikely that anyone distributing binary go
> > packages reads golang-dev or golang-nuts.
>
> Is there a more likely place to reach such people?

I believe the best ways to reach attention on such issues are currently the
official Go blog and the the Golang Weekly Newsletter.

At least I know of IT magazines here in Germany, that watch these sources and
aquire content for their online platforms from there.

Regards,
Harald

[jc...@protonmail.com]

Currently I don't use them, but couldn't binary packages be used to distribute CGO-dependent packages for developers who might not have a C cross-compiler? I think the feature would be more commonly used if it wasn't so impractical. Rather than abandoning the feature I'd like to see it revamped to work well with modules.

[Ian Lance Taylor]

You shouldn't need to use binary packages for that. It's OK to
distribute C code as a binary, as a .syso file. The issue is there is
distributing the Go code as a binary.

Ian

[Scott Cotton]

Go can produce c-archive and c-shared binaries that can be consumed by C and then, perhaps even unbeknownst to the go programmer, the c library they use via cgo has Go in it somewhere.

I think that this situation would lead to a mix of Go binary and source in any case.  I also think it might be desirable, so that 

Go code can provide functionality that C code depends on.  I mean, a project that publishes a C library and decides to use Go for part of its implementation should not have to say "we use Go so if you use cgo, then ..."

This combination is perhaps hypothetical at this point for Go, but there are low level Rust libraries like this where the consumer doesn't need to know whether or not it is or has parts implemented in rust. 

Scott

 

[Fino]

I think Binary-only packages are very important for Go's Eco,  something like maven(Java) & sbt(scala)

- all it's dep packages should save in the same repo,  together, like a snapshot, or similar concept like docker image

- use a manifest file to list all the dep package's name and version

it's ok that when the binary release, it was compiled with a older version of Go; and after that, no body maintain it and cannot compile by newer version's Go.

if it just can run and still works good in a living OS, it should stay in a global repo and let people can download and use it.

many system don't need update for many years after release, and impossible to recompile all it's libraries against newer compilers.

BR fino

[robert engels]

I think there is a bigger concern than shipping binaries. I believe that the Go stdlib and runtime should be shipped as a shared library, and be field upgradable independent of the binaries. This is the only reasonable way to to distribute security patches in an efficient and stable manner.

This is another issue, but there is some cross-over with this discussion.

[Ian Lance Taylor]

On Mon, Oct 22, 2018 at 7:18 PM, robert engels <ren...@ix.netcom.com> wrote:
>
> I think there is a bigger concern than shipping binaries. I believe that the
> Go stdlib and runtime should be shipped as a shared library, and be field
> upgradable independent of the binaries. This is the only reasonable way to
> to distribute security patches in an efficient and stable manner.
>
> This is another issue, but there is some cross-over with this discussion.

That already works today. You can ship a shared library, but in order
for the go tool to handle it properly you also have to include the
source. Shared libraries in general are a separate issue from binary
packages.

Ian

[Robert Engels]

Wait... you can ship a Go binary that dynamically links with the Go runtime and stdlib at runtime??? Awesome. Can you point me to the docs on this. I’ve always considered the “single binary” taunted feature as somewhat of a limitation because of this.

[Robert Engels]

Is there documentation on how to do this, or was I misunderstanding your answer? Thanks.

[Ian Lance Taylor]

On Mon, Oct 22, 2018 at 8:53 PM, Robert Engels <ren...@ix.netcom.com> wrote:
>
> Wait... you can ship a Go binary that dynamically links with the Go runtime and stdlib at runtime??? Awesome. Can you point me to the docs on this. I’ve always considered the “single binary” taunted feature as somewhat of a limitation because of this.

go install -buildmode=shared std
go build -linkshared hello.go

https://golang.org/cmd/go/#hdr-Build_modes

It doesn't work on every system but it does work on GNU/Linux.

Ian

[Robert Engels]

So are there any plans for security patches for prior versions shipped as an updated shared library?

Given the amount of code in the std it seems this would be almost a requirement - if Go apps are released in the wild in binary form (e.g Linux tools)

Sent from my iPhone

[Ian Lance Taylor]

On Tue, Oct 23, 2018 at 2:11 PM, Robert Engels <ren...@ix.netcom.com> wrote:
>
> So are there any plans for security patches for prior versions shipped as an updated shared library?
>
> Given the amount of code in the std it seems this would be almost a requirement - if Go apps are released in the wild in binary form (e.g Linux tools)

The Go project does not ship shared libraries. We leave that to
distro maintainers.

The Go project's security policy is at https://golang.org/security.

Ian

[Robert Engels]

Thanks. I have a better understanding of the situation after reading the reference.

Sent from my iPhone

[fge...@gmail.com]

If you plan to change during the next release maybe there is no better
place, although if the planned timeframe is 2-4 releases in the future
- maybe revisiting the "no warning" policy and a warning on planned
deprecation would be appropriate

[fge...@gmail.com]

[smal...@gmail.com]

Well, I do need it that in our company, I need to provide the binary-package to other developers to protect my source code.

在 2018年10月19日星期五 UTC+8上午8:13:01，Tharaneedharan Vilwanathan写道：

[Wei Tang]

Well, I do think Binary-only packages are very important for some security case, such as authentication algorithm in a private enterprise。

在 2018年10月19日星期五 UTC+8上午1:16:43，Russ Cox写道：

The go command supports "binary-only packages", in which 

the compiled .a file is installed in GOROOT/pkg or GOPATH/pkg

but the corresponding source code is only a stub with relevant

imports (for linking), a special comment marking it as a binary-only

package, and perhaps documentation for exported API.

While the go command will use these packages if they exist in

GOROOT or GOPATH, 'go get' will not download them: 'go get'

is intentionally only about source code distribution.

Furthermore, binary-only packages only work when the build is

using the exact versions of whatever imported packages are

needed by the binary-only package. If there were any important

differences in a dependency between compilation and linking,

the binary-only portion might have stale information about details

of the imported package, such as type sizes, struct field offsets,

inlined function bodies, escape analysis decisions, and so on,

leading to silent memory corruption at runtime.

Binary-only packages also only work when the build is using the

exact version of the Go toolchain that was used for compilation.

This is at least enforced at link time, with the effect that if your

binary-only package only imports the standard library and you don't

use any special compilation flags, then the toolchain version check

suffices to avoid the silent memory corruption problem described in

the previous package. But if your package imports any non-standard

package for which that the eventual user might try to combine with

a different version, you're in trouble again.

Compiled Go programs also contain much richer amounts of runtime

information than compiled C programs, so that for example all the

details of type declarations, including struct definitions, are in the

compiled code, along with file names and line numbers for the compiled

code, and increasingly good debug information that was impossible to

turn off until very recently. Overall, a binary-only package obscures very little.

In short, binary-only packages:

- have never been supported by 'go get',

so you've had to go out of your way to use them,

- aren't even guaranteed to work when you do use them,

possibly leading to silent memory corruption, and

- still expose quite a lot more than most people would expect

about the original source code.

For these reasons, we're looking at removing binary-only package

support entirely.

If you use binary-only packages and think it's important to keep that

support around, please let us know either on this thread or at

golang.org/issue/28152. 

Thanks.

Russ

[Ian Lance Taylor]

On Tue, Jan 22, 2019 at 1:06 PM Wei Tang <tomja...@gmail.com> wrote:
>
> Well, I do think Binary-only packages are very important for some security case, such as authentication algorithm in a private enterprise。

I would not recommend using them in that way. A binary package should
be thought of as a slightly shrouded form of the original source code.
An attacker with access to the binary-only package can easily
reproduce a variant of the original Go source code. Comments and
local variable names will be lost, but everything else will be there,
including any certificates and all algorithms.

Ian

[Ian Denhardt]

I feel like I should stress this a bit more, because it's really
important:

It is under no circumstances reasonable to rely on lack of source code
for security. History has taught us time and time again that security
through obscurity doesn't work, and home grown/custom algorithms are
usually broken.

For authentication, you should always be using off-the shelf, open
source libraries which implement publicly known algorithms that have
been peer reviewed by experts. These will be more secure than anything
that hasn't seen the light of day, and don't rely on people not knowing
about them.

There is no good reason why there should be anything security-sensitive
in your source code.

-Ian

Quoting Ian Lance Taylor (2019-01-22 16:26:27)

[cos.p...@gmail.com]

Has this been finally decided or is there still room to save this feature?

We have invested all last year building an SDK (industrial automation for oil&gas and pharma) we have two customer on a paying beta agreement.

Many have spoken to this rather common scenario to protect IP (no, plugins do not cut it as stated by others e.g. on https://github.com/golang/go/issues/28152 ).

Just shutting a feature off like this means a huge loss for us.

Again, we wouldn't care too much about sharing source (clear or obfuscated) but we wrote all the legal agreements around the BOP concept. We would have to refactor to C which we tested (CGo that is) and it has significant performance drop. 

But more than anything it's the sheer investment I made...

Thanks for any further consideration

[Robert Engels]

I read the issue, and am unclear as to why a plug-in wouldn’t support your use case?

--