binary sizes, again

[andrey mirtchovski]

binary sizes have crept up again and i'm wondering if there's
remediation planned. the runtime cost now sits on 550k. this is for an
empty binary with no imports and func main(){} at tip:

552256 17 Sep 09:56 empty

i imagine this is the cost of all the scheduler, network performance,
and gc improvements so i'm happy with that. looking at the canonical
"hello world", it has also grown around 30%, from 1.5MB to 2.2MB:

2229408 17 Sep 09:56 hello

similar growth with the "go", "gofmt" and "godoc" binaries from pre
1.0 to now (see attached).

not complaining, but interested in in knowing if others are worried by this.

[Josh Bleecher Snyder]

> not complaining, but interested in in knowing if others are worried by this.

I am, particularly in light of
https://code.google.com/p/go/issues/detail?id=6245

-josh

[Gustavo Niemeyer]

Note that "strip"ing Go binaries should still work, so you can cut
down these numbers a bit.

> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.



--

gustavo @ http://niemeyer.net

[Gustavo Niemeyer]

I fixed it a couple of years ago: http://golang.org/issue/1242

But I wouldn't be surprised if it's broken again. I don't think we
have any good test coverage for this.


On Tue, Sep 17, 2013 at 3:29 PM, Matthew Kane <ma...@hydrogenproject.com> wrote:
> Um, stripping is supported now?

> matt kane
> twitter: the_real_mkb / nynexrepublic
> http://hydrogenproject.com

[Josh Bleecher Snyder]

> I fixed it a couple of years ago: http://golang.org/issue/1242
>
> But I wouldn't be surprised if it's broken again. I don't think we
> have any good test coverage for this.

https://twitter.com/davecheney/status/377960504477487104

-j

[Gustavo Niemeyer]

That's pretty much what I just said. Yes, it's untested, and yes it
can break your executable if there are bugs in how the ELF file is
built.

I use stripped binaries, even then, and you might too if you can live with that.

[andrey mirtchovski]

zip compresses go binaries very well (~75% on osx). how do we go about
creating self-extracting binaries again?

[Ken Allen]

UPX helps a lot depending on your target platforms.

[Rob Pike]

The topic of binary size is very much on our minds. Expect improvement
at some point, but right now we're focused on the 1.2 release.

-rob

[Kevin Gillette]

I agree, particularly for long-running single-instance apps (e.g. daemons/services), upx is great for Go binaries (do we still need goupx for this, or have the ELF peculiarities been fixed?)

However, it's less desirable for anything designed to act like a general command that can be run frequently and in multiple simultaneous contexts, since decompressed upx binaries cannot share code pages in ram.

It seems that a lot of superfluous symbols that used to be linked are no longer linked, though of course method sets are still fully linked, afaict. Adding (more) reflect-aware instrumentation could go a long way towards pruning unused method symbols out as well, since many types in a given app will never be passed to reflect, and among those, still many will never need to have their methods inspected. In a fairly simple fsnotify app (compiled with recent tip) I ran through nm, about 500 method symbols are included that are effectively dead code -- that's out of 6500 total symbols, or 1600 function symbols (including runtime internals). I expect that with non-trivial apps the proportion of unused-to-used methods will increase.

[Dave Cheney]

ldflags -s doesn't do what you want, unless that is to produce a broken binary.

I agree with Minux that -s should be removed.

[Lubos Pintes]

Stripping created a broken binary on Windows.

[Matthew Kane]

Um, stripping is supported now?

On Tue, Sep 17, 2013 at 2:22 PM, Gustavo Niemeyer <gus...@niemeyer.net> wrote:

[al...@lx.lc]

Respectfully disagree.  In some instances, producing a small binary is either desired or frankly necessary.  The -s option used to work well in Go 1, but now just produces a broken binary.  Linux strip command still seems to work, but is redundant to have to do this manually with each build. 
Perhaps the -s option could just strip less, but I feel that removing it altogether may be a mistake.

Thanks,
Alex

[d...@nanard.net]

In my case, "-s" option is an important feature to improve the security of my application. Without this, all function names are plain, in particular those related to encryption.
What should I do so ? Should I add an obfuscation step in the build process ?

Moreover, the size gain is not negligible, almost 50% of binary size, but this is less critical for me.

[Jan Mercl]

On Thu, Sep 19, 2013 at 11:16 AM, <d...@nanard.net> wrote:
> In my case, "-s" option is an important feature to improve the security of
> my application. Without this, all function names are plain, in particular
> those related to encryption.

If the security of your app depends on plain function names being
visible or not than your security model is not secure.

-j

[minux]

agree. besides, even without symbols, one can still use function signatures to determine whether the function is in std. lib and what does it do.

obscurity won't lead to security.

i'd rather not see -s is being applied to application like this, because it's a false sense of security.

[minux]

On Sep 17, 2013 6:38 PM, "Dave Cheney" <da...@cheney.net> wrote:
>
> ldflags -s doesn't do what you want, unless that is to produce a broken binary.
>
> I agree with Minux that -s should be removed.

yeah, at least for Go 1.2, we'd better first make it an noop because there isn't enough time to fix it, and i see the trend for runtine to depend more on symbol tables.

for those who wants smaller sizes, please check out -ldflags -w, which removes dwarf debugging information in a binary.

[d...@nanard.net]

Sure it is not secure... my application - a server - can't be secure as I should enter a password at startup (or make it communicate with another service physically secured) to make it really secure, and I don't want this. This is just a way to slow the break. With plain functions names, it is so easy to focus on a few functions set, not to mention the overall reverse engeneering process becoming far easier.

[minux]

On Sep 19, 2013 1:32 PM, <d...@nanard.net> wrote:
>
> Sure it is not secure... my application - a server - can't be secure as I should enter a password at startup (or make it communicate with another service physically secured) to make it really secure, and I don't want this. This is just a way to slow the break. With plain functions names, it is so easy to focus on a few functions set, not to mention the overall reverse engeneering process becoming far easier.

if it is a server, i think it will be a more serious problem that an adversary get hold of your binaries.

using function signature to determine what a (open souce) function does is well-known to the reverse engineering society, and besides, they also have more intrusive ways to break your protection.

stripping symbols just won't stop them.

[d...@nanard.net]

Stripping symbols won't stop them but will slow reverse ingeneering. With expressive function names, understanding what a function does is trivial. With obfuscation, more work is required, and reaching a complexity level, it is very costly to do it.

[Kevin Gillette]

On Thursday, September 19, 2013 11:48:17 AM UTC-6, d...@nanard.net wrote:

Stripping symbols won't stop them but will slow reverse ingeneering. With expressive function names, understanding what a function does is trivial. With obfuscation, more work is required, and reaching a complexity level, it is very costly to do it.

Not necessarily. Any Go program will almost certainly make extensive use of the stdlib -- any tool that can, as mentioned above, identify stdlib functions and methods from their signatures, and then generate a list of callees for every unidentifiable function would quickly be able to determine what a given program does. In the presence of such tooling, which is not difficult to write, breaking past the obfuscation is essentially zero-cost.

Obscurity is only useful at discouraging normal users from casual inspection. If you're intending a third party from using offline analysis to determine how your program works, any determined, moderately experienced attacker will have little trouble getting past any obstacle short of a fully encrypted binary.

[d...@nanard.net]

Ok I see. Can I considere this true for C built programs too, as they make extensive use of the OS libraries ? In other words, do you think that nowadays, the cost of reverse engineering - whatever compiler used - is so low than we can't rely on this to discourage work theft ?

[minux]

On Thu, Sep 19, 2013 at 3:31 PM, <d...@nanard.net> wrote:

Ok I see. Can I considere this true for C built programs too, as they make extensive use of the OS libraries ? In other words, do you think that nowadays, the

most commercial software are written in C/C++, and almost all of them are properly stripped.

some of them even use extensive anti-reverse techniques (packing, etc), but still, pirated copies

are floating around.

cost of reverse engineering - whatever compiler used - is so low than we can't rely on this to discourage work theft ?

unfortunately, it is true.

that's why people are now fond of SaaS solutions and the like, where the end user won't be able to get hold of the

complete software package. Practically, there is no way an end-user software product could be secure (even those

using dedicated hardwares for protection) in this sense.

[Kevin Gillette]

On Thursday, September 19, 2013 1:31:52 PM UTC-6, d...@nanard.net wrote:

Ok I see. Can I considere this true for C built programs too, as they make extensive use of the OS libraries ? In other words, do you think that nowadays, the cost of reverse engineering - whatever compiler used - is so low than we can't rely on this to discourage work theft ?

Pretty much. Always assume that, aside from properly implemented true cryptographic security coverage of whatever you want to protect (and all of those algorithms have well defined limitations), your code, assets, etc are entirely unprotected. That said, stripping binaries, and other such things don't hurt (so if you're already utilizing them in your build procedure, you might as well keep doing that) -- they just might not help.

[David Anderson]

And true cryptographic protection of a binary, short of help from the kernel and hardware, is pointless to protect a binary. By definition, you are giving the user both encrypted data and the decryption key, and expecting that they will not be able to decrypt at will.

Incidentally, this is the same problem as DRM that isn't silicon-assisted. You can't give someone a lock and the matching key and expect the lock to be unopenable.

- Dave

--

[Kevin Gillette]

On Thursday, September 19, 2013 8:53:46 PM UTC-6, David Anderson wrote:

And true cryptographic protection of a binary, short of help from the kernel and hardware, is pointless to protect a binary.

I was thinking more of an unencrypted loader/decrypter that takes a hand-typed (or other securely entered) key on run, and uses that to decrypt the binary into ram (much like what upx and friends do with compression), but obviously that too would have problems without kernel and potentially hardware support (you'd need to mark code pages as unswappable, and such). 

[Chetan Gowda]

Sorry to revive this year old thread. Just wanted add another case for supporting ldflags -s option.

I ship my binary to users and I use the -s option to strip out information like the full path of my build environment, the home directory, unix username etc. I'm still using go 1.1 specifically for this reason. I feel being able to strip this information is important to anyone who is distributing their binaries to other people.

[Dave Cheney]

Please raise an issue, golang.org/issue/new

[andrewc...@gmail.com]

A workaround is just to build in a VM with generic username and home directory.

[Chetan Gowda]

> Please raise an issue, golang.org/issue/new 

https://code.google.com/p/go/issues/detail?id=8999

> A workaround is just to build in a VM with generic username and home directory.

It's not just the username and home directory. The path for different packages are also inserted which may leak some of the project names and internal package names.