Insane memory allocation with go build

[Michael Banzon]

Hi,

I have been using go-bindata (https://github.com/jteeuwen/go-bindata) to convert some assets (templates and image files) to Go source files.

When compiling the resulting source code the go compiler uses an insane amount of memory. I compile on Win8.1 (64-bit) and Ubuntu 14.04 (64-bit). On Windows compiling the source spikes at around 1.6 GB memory use - but on Ubuntu the memory usage goes to about 7 GB.

The source file from Go-bindata is about 7 MB (a bit larger than the original assets ~700 KB).

Is there a logical reason to the:

- Amount of memory used?

- Different amount of memory used?

- Difference in source code size and compiler memory usage?

I can't really share the source/assets - but if it brings any value I can spend some time to re-create the scenario with sharable data.

--
Michael Banzon
http://michaelbanzon.com/

[Dave Cheney]

Does go-bindata produce a large array like

var asset = []byte{ 0x30, 0x01, 0x44 } ?

If so, this will cause the compiler to use a lot of memory because of the way that every constant literal is stored inside the compiler as a multi precision number. The best way to solve this is to instead keep your asset as a base64 encoded string literal

var asset = "somebase64text"

and feed the asset through a base64 reader before consuming it.

This generates a much more efficient compiled form.

[Michael Jones]

On Sun, May 18, 2014 at 5:58 PM, Dave Cheney <da...@cheney.net> wrote:

This generates a much more efficient compiled form

You mean "efficient compilation process" right? The compiled form should be a good one either way.

--

Michael T. Jones | Chief Technology Advocate  | m...@google.com |  +1 650-335-5765

[Dave Cheney]

Yes, that is correct.

Also my second example should have read

const assert = "somebase64encodedstring"

[Carlos Castillo]

By using the -nomemcpy flag to go-bindata, the data is stored in a string instead, which go 1.3 should handle better. 

However, you should probably use a different method to store large assets. My method is to append a zip file to the end of you executable post-compile, and then use either zip -A to fix the zip header and use the archive/zip package, or use a package like my gopkg.in/cookieo9/resources-go.v2 package (don't need to zip -A then). 

[Nigel Tao]

On Mon, May 19, 2014 at 10:58 AM, Dave Cheney <da...@cheney.net> wrote:
> If so, this will cause the compiler to use a lot of memory because of the
> way that every constant literal is stored inside the compiler as a multi
> precision number. The best way to solve this is to instead keep your asset
> as a base64 encoded string literal

Well, that's a workaround to a compiler bug, but the best solution is
to generate the natural looking code and fix the compiler bug.
https://code.google.com/p/go/issues/detail?id=6643

[Michael Banzon]

Thanks for the reply guys.

The memory usage is explained, and a workaround is found ;-)

Just for the curiosity:

What about the 1:5 difference in memory allocation between Windows and Linux compiler (Windows version allocating the smallest amount)? This is actually the thing that triggered the question as the builds would fail on the ("very limited") Linux box.

[Dave Cheney]

How are you measuring memory allocation ?

> --
> You received this message because you are subscribed to a topic in the
> Google Groups "golang-nuts" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/golang-nuts/tOp5mx_SJMI/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Michael Banzon]

Oh you make me feel so guilty ;-)

Until now: htop on Linux and the Job-list-thing on Windows - very low tech.

As I said, the main reason I started looking was the failing build on Linux.

[Dave Cheney]

Can you please post some examples

[Michael Banzon]

I just ran the compilation on my Windows and Linux box with a tool that properly measured allocated VM.

The difference was about 200 MB in a total of ~7.4 GB allocation (from 7 MB source file).

So, after properly adjusting  instrumentation tools it turns out the difference wasn't really that alarming. Sorry for causing the fuzz :-)

[Dave Cheney]

Thanks for confirming, that is what I expected.

[Mateusz Czapliński]

On Monday, May 19, 2014 3:50:38 AM UTC+2, Carlos Castillo wrote:

By using the -nomemcpy flag to go-bindata, the data is stored in a string instead, which go 1.3 should handle better. 

However, you should probably use a different method to store large assets. My method is to append a zip file to the end of you executable post-compile, and then use either zip -A to fix the zip header and use the archive/zip package, or use a package like my gopkg.in/cookieo9/resources-go.v2 package (don't need to zip -A then). 

Also, if you fancy, you can try using https://github.com/akavel/rsrc, like below:

rsrc -data YOUR_INPUT.dat -o youroutput.syso > youroutput.c
  Generates a .syso file with specified opaque binary blob embedded,
  together with related .c file making it possible to access from Go code.
  Theoretically cross-platform, but reportedly cannot compile together with cgo.

The generated *.syso and *.c files should get automatically recognized
by 'go build' command and linked into an executable/library, as long as
there are any *.go files in the same directory.

then in your code, declare a function like below, and just use it to retrieve the contents of your dat file (note: you should probably treat this slice as readonly):

func get_youroutput() []byte

 
Please note, that this is for now somewhat experimental, especially it is reported to have trouble if used together with cgo, unfortunately.

I'll be very happy if this helps anyone.
/Mateusz.

[dav...@gmail.com]

Hello all - I'm trying to build go programs on a Xilinx Zynq development board that has 512MB of RAM. There is no other storage. When I have all the tools in place (cross compiled and put in the /usr/local/go directories) a du -hs / reports 185MB used. I can run go doc fmt and most other go commands. BUT if I try to build the simplest (fmt.Println("Hello!")) programs, it eventually kernel panics.

I can cross compile and copy over any reasonable program on the (Zynq/ARM) target. I'd like to build on the target platform so I can eventually test and profile code natively.

Any suggestions?

[Dave Cheney]

It is possible that your go installation is incorrect. Can you please post the output of go build -x.

It /tmp on disk, or in ram?

Dave

[Dave Mazzoni]

I think the installation is correct. But my approach is wrong. I moved all the go tools over to the target/Arm system under the assumption it was the only way to run go test and go pprof on the target/Arm embedded system.

The real payoff would be to do all the heavy lifting (compiling, testing, profiling, etc) on the host OS/Linux amd64 and transfer those executables to the target.

Let me pose the question this way: is there a way I can cross-compile everything I need on the host and then transfer the files to the target to do testing and profiling? It seems I would need a way to build/cross-compile an "instrumented" executable for the target and then copy and run it there.

Make sense?

[Dave Cheney]

You can build test binaries with the -c flag, this should support the usual GOOS/GOARCH flags, but not tested.

The go tool also supports the -execcmd (from memory) flag which effectively inserts a shim, the cmd, before running your program, check out the various wrappers in the misc directory, this is who we do the darwin/arm and android builds.

Profiling will work as expected with a cross compiled binary, just scp back the binary and the profile to your workstation and use pprof as usual.

Dave

[Ian Lance Taylor]

On Mon, Sep 28, 2015 at 4:32 PM, Dave Cheney <da...@cheney.net> wrote:
>
> The go tool also supports the -execcmd (from memory) flag which effectively
> inserts a shim, the cmd, before running your program, check out the various
> wrappers in the misc directory, this is who we do the darwin/arm and android
> builds.

Note that it's -exec, not -execcmd. See
https://golang.org/cmd/go/#hdr-Compile_and_run_Go_program for more
details.

Ian

[Dave Mazzoni]

Perfect -- it's working. This is a huge win for the embedded community. 

Thanks!

[Dave Mazzoni]

Almost... While I can get go tests to cross-compile correctly using the -c option, when I reintroduce C code with CGO, I now get runtime errors on the target: 

# ./ad9361_test.go 

./ad9361_test.go: line 1: package: not found

./ad9361_test.go: line 3: syntax error: unexpected newline (expecting ")") 

Am I missing something?

[Dave Cheney]

Cross compilation of CGO packages is not supported by default as it requires a cross compiling c compiler for your target.