Removing unused things from the executable?

[Leff Ivanov]

Well, to be fare, I'm very new to Go and I'm coming from the C/C++ background. I compiled Go 1.4.2
compiler from sources to be able to crosscompile from my Linux box to Windows and Mac OSX, and
it works pretty fine (more to say Go has the easiest crosscompiler setup I've ever seen). But even the
helloworld executables are huge (1.0-1.5 Mb for helloworld executable). I decided to take a look at
what is compiled in, and I have several questions about it:

What is the purpose of empty ".symtab" section on Windows? Can I make Golang compiler not
generate this section for the output windows executable?

It seems that debug information can not be stripped for Mac OSX executables, is it some kind of
bug or something ("-ldflags -s" works for Windows and Linux, but doesn't work for Mac)?

By using autoanalysis in IDA Pro I can see that several runtime functions that are linked in aren't
in fact used by the code, is there something like LTO (link time optimization), that would remove
unused functions from the output executable?

There are a lot of strings in the executable that looks like /home/<user>/go/scr/<gosrcfile.go>,
these strings point to the sources of Golang's runtime and stdlib. What are these strings used
for in the executable? Can I get rid of them, or at least make them contain only the file name
without full path to the place where runtime and stdlib were built? I understand that this stuff
can be used for debugging, but I can't imagine the situation when some compiler user needs
to debug runtime or standard library.

There are a lot of strings in the executable that contains names and declarations for types and
functions, I guess it is part of the RTTI (runtime type information) spec for Golang. Where are
these string used in the executable? Can I safely get rid of them (knowing that I won't be using
functions that needs runtime type information)?

Is it possible to create my own custom and slim runtime library by cutting off things that I don't
need from original runtime library, and make Golang compiler use it instead of original one? I'm
asking about main Golang compiler, not the GCCGo one. The size of the executable doesn't
mean much these days, but I want to be able to get rid of the stuff I don't actually need in the
final executable, as Golang compiler by default seems to include too much stuff in it.

[Ian Lance Taylor]

On Mon, Jun 8, 2015 at 3:05 AM, Leff Ivanov <droid...@gmail.com> wrote:
>
> Well, to be fare, I'm very new to Go and I'm coming from the C/C++
> background. I compiled Go 1.4.2
> compiler from sources to be able to crosscompile from my Linux box to
> Windows and Mac OSX, and
> it works pretty fine (more to say Go has the easiest crosscompiler setup
> I've ever seen). But even the
> helloworld executables are huge (1.0-1.5 Mb for helloworld executable).

In part Go binaries are large because they are, by default, statically
linked.

The overall issue for binary size is http://golang.org/6853 .

> What is the purpose of empty ".symtab" section on Windows? Can I make Golang
> compiler not
> generate this section for the output windows executable?

I don't know.

> It seems that debug information can not be stripped for Mac OSX executables,
> is it some kind of
> bug or something ("-ldflags -s" works for Windows and Linux, but doesn't
> work for Mac)?

I think this is fixed in Go 1.5. I haven't checked, though.

> By using autoanalysis in IDA Pro I can see that several runtime functions
> that are linked in aren't
> in fact used by the code, is there something like LTO (link time
> optimization), that would remove
> unused functions from the output executable?

Unused functions should be removed by the linker. However, note that
it's hard for the linker to tell whether a method is used, since if
the value is converted to an interface type, such as by passing the
value to fmt.Print, then the method is retained.

> There are a lot of strings in the executable that looks like
> /home/<user>/go/scr/<gosrcfile.go>,
> these strings point to the sources of Golang's runtime and stdlib. What are
> these strings used
> for in the executable? Can I get rid of them, or at least make them contain
> only the file name
> without full path to the place where runtime and stdlib were built? I
> understand that this stuff
> can be used for debugging, but I can't imagine the situation when some
> compiler user needs
> to debug runtime or standard library.

These strings are used in backtraces, and it's quite normal to see a
backtrace for the runtime or standard library.

> There are a lot of strings in the executable that contains names and
> declarations for types and
> functions, I guess it is part of the RTTI (runtime type information) spec
> for Golang. Where are
> these string used in the executable? Can I safely get rid of them (knowing
> that I won't be using
> functions that needs runtime type information)?

These strings are used by the reflect package, which is used by basic
packages like fmt. There is no safe way to remove them.

> Is it possible to create my own custom and slim runtime library by cutting
> off things that I don't
> need from original runtime library, and make Golang compiler use it instead
> of original one? I'm
> asking about main Golang compiler, not the GCCGo one. The size of the
> executable doesn't
> mean much these days, but I want to be able to get rid of the stuff I don't
> actually need in the
> final executable, as Golang compiler by default seems to include too much
> stuff in it.

This is possible, I suppose, but there is no supported way to do it.

Hope this helps.

Ian

[mmsti...@gmail.com]

You should definitely try out https://github.com/pwaller/goupx with the --ultra-brute flag. It will compress your binaries to about a tenth of the original size, after doing everything you can do to strip it of debugging and other useless data for end users. That said, there's a lot that Go could do to reduce the binary size, but this is one of the workarounds at the moment.

[brainman]

On Monday, 8 June 2015 20:05:36 UTC+10, Leff Ivanov wrote:

> What is the purpose of empty ".symtab" section on Windows? Can I make Golang compiler not 

> generate this section for the output windows executable?

".symtab" contains pe symbol table. I don't believe it is empty. Why do you think it is empty? You can do

go build -ldflags -s main.go

to skip ".symtab" generation.

Alex

[Leff Ivanov]

In part Go binaries are large because they are, by default, statically
linked.

Yes, I understand that, but Go binaries has really too much stuff in it
in comparison to other languages. For example for 32-bit windows for
a simple hello world (printing a string) build I got 1978 functions linked
in from runtime library and standard library.

I think this is fixed in Go 1.5.  I haven't checked, though.

I'll try it when Go 1.5 will be out.

These strings are used in backtraces, and it's quite normal to see a
backtrace for the runtime or standard library

In my opinion compiler should provide a way to disable backtraces, if
developer doesn't need it in the release build.

These strings are used by the reflect package, which is used by basic
packages like fmt.  There is no safe way to remove them.

Yes, it seems Go relies on relfection and runtime type information for
everything, starting from garbage collection to interfaces stuff.

[Leff Ivanov]

That said, there's a lot that Go could do to reduce the

binary size, but this is one of the workarounds at the moment.

It is not about the size, but about the fact that Go builds too
much stuff into the output executable.

[Leff Ivanov]

".symtab" contains pe symbol table. I don't believe it is empty. Why do you think it is empty?

.idata section contain PE import table, the .symtab section is empty because it is minimal in
size (0x200), contains only zeros and there is no code that reference it in the executable. Also
.symtab section isn't a typical thing for executable, imports are usually stored in .idata, exports
are usually stored in .edata.

go build -ldflags -s main.go

to skip ".symtab" generation.

I built it with "-ldflags -s" the first time, but the .symtab section is in the executable.

 

[Jesse McNelis]

On Tue, Jun 9, 2015 at 7:12 PM, Leff Ivanov <droid...@gmail.com> wrote:
>> In part Go binaries are large because they are, by default, statically
>> linked.
>
> Yes, I understand that, but Go binaries has really too much stuff in it
> in comparison to other languages. For example for 32-bit windows for
> a simple hello world (printing a string) build I got 1978 functions linked
> in from runtime library and standard library.

Calling fmt.Println() isn't a 'simple hello world', it's a very
'complicated hello world'.
The fmt pkg is massive overkill for writing a string to stdout.

A 'simple hello world' is calling os.Stdout.WriteString( "Hello world\n").
A simple hello world in Go is comparably sized to a statically linked
simple hello world in C.

[Ian Lance Taylor]

On Tue, Jun 9, 2015 at 2:12 AM, Leff Ivanov <droid...@gmail.com> wrote:
>
>> These strings are used in backtraces, and it's quite normal to see a
>> backtrace for the runtime or standard library
>
> In my opinion compiler should provide a way to disable backtraces, if
> developer doesn't need it in the release build.

You can also get backtraces in the program itself, via runtime.Callers
and runtime.Stack. Backtraces are also used for profiling via
runtime/pprof, which is very useful in production programs.

It would be possible to provide a compiler option to disable
backtraces, but it would be unwise for anybody to actually use it in
production. While I completely agree that Go binaries are too large,
and there is a bug report about finding ways to make them smaller, it
is also true that if you want to write the smallest possible program
then Go is not your best choice.

Ian

[brainman]

On Tuesday, 9 June 2015 19:27:34 UTC+10, Leff Ivanov wrote:

> ... the .symtab section is empty because it is minimal in 

> size (0x200), contains only zeros and there is no code that reference it in the executable.

.symtab section contains pe symbol and string tables. If you used "go build -ldflags -s main.go" to build your executable, the string table will contain only dwarf section names (symbol table will not have any program symbols). That is why .symtab section is so small.

> ... .symtab section isn't a typical thing for executable,

I don't know what is typical. Go linker always creates that section.

> imports are usually stored in .idata, 

Go imports are stored in .idata.

> ... exports

> are usually stored in .edata.

Go executable does not have exports.

> I built it with "-ldflags -s" the first time, but the .symtab section is in the executable. 

Like I said above .symtab section contains pe symbol and string tables. Any long string goes there. Minimal .symtab section will still contains dwarf section names.

Alex

[Leff Ivanov]

Like I said above .symtab section contains pe symbol and string tables.

Like I said .symtab section is empty, it contains 0x200 zero bytes. There is
no such thing as "PE symbol", there are only exports and imports, which are
typically stored in .edata and .idata sections. There are no string tables either,
strings are stored in .rdata section as data, or in PE file resources as a table
of specified format stored in .rscr section.

the string table will contain only dwarf section names

There is no dwarf section in the executable, dwarf debug information is usually
stored in the overlay of PE file (appended to the end of executable file). There
are special fields in the PE, that can be used to store debug information, but to
actually be used the debug information shoud be in microsoft's format (PDB).

I don't know what is typical. Go linker always creates that section.

So it should be considered a bug, if linker always creates a useless empty
section with minimal section size and filled with zero bytes.

[Leff Ivanov]

A 'simple hello world' is calling os.Stdout.WriteString( "Hello world\n").
A simple hello world in Go is comparably sized to a statically linked
simple hello world in C.

Well, it is smaller, but still I'm getting around 800 functions linked into the
executable for a "one liner" main function. A 'simple hello world' in C can
be compiled into 2kb executable, because 'puts' ans 'exit' functions are
provided by the operating system (msvcrt.dll on windows and libc on other
operating systems). If I statically link CRT, I will get around 100-200Kb
executable, the Go's executables are 600-850Kb for different targets,
so I don't think that it is comparable size. The same goes for other
programming languages like D (around 300Kb), Nim (around 200Kb).
However Rust's executables are huge too, even bigger than Go's one,
different LISP/Scheme based system produce huge executables too.

[Jan Mercl]

On Wed, Jun 10, 2015 at 10:09 AM Leff Ivanov

> So it should be considered a bug, if linker always creates

> a useless empty section with minimal section size and filled

> with zero bytes.

I agree that producing a unused/zero filled section is not perfect. However, let's have a look what the perfectionism can give us:

This is arguably the minimal Go program.

        package main

        

        func main() {

        }

The resulting binary is 634696 bytes (Linux 64b, Go 1.4.2). Saving ~500 bytes would make it about 0,079% smaller.

-j

--

-j

[Jan Mercl]

On Wed, Jun 10, 2015 at 10:18 AM Leff Ivanov <droid...@gmail.com> wrote:


> Well, it is smaller, but still I'm getting around 800 functions linked into 

> the executable for a "one liner" main function. A 'simple hello world' in

> C can be compiled into 2kb executable, because 'puts' ans 'exit'

> functions are provided by the operating system (msvcrt.dll on windows 

> and libc on other operating systems). If I statically link CRT, I will get 

> around 100-200Kb executable, the Go's executables are 600-850Kb

> for  different targets,  so I don't think that it is comparable size. 

And neither it is comparable in functionality. Every Go binary includes the garbage collector, started even before main executes. The GC executes in more than one goroutine, IIRC, it's not a trivial conservative one. And the GC is precise, which means every object type which can be encountered on stack or in the heap must have a complete information about if and where it has pointers, including type defined by the runtime itself.

Comparing the size of all of this to a C Hello world program which has nothing of the above makes little sense.

-j

--

-j

[Milan P. Stanic]

~> go version
go version go1.3.3 linux/amd64
----------- helloworld.go ---------------------
package main

func main() {
print("Hello, world\n")
}
--------------------------------
~> go build -ldflags -s helloworld.go
~> ls -lh helloworld
-rwxr-xr-x 1 mps mps 333K Jun 10 13:58 helloworld

333K and Go version is 1.3.3 (I should upgrade, I see).