Out of memory using golang.org/x/tools/go/packages

[Levieux Michel]

Hi all,

I'm having a very hard time with golang.org/x/tools/go/packages. Spent most of my evening yesterday trying to understand what's happening here.

Here's my code: https://play.golang.com/p/5L1N0lSaetB

With this very simple code I would expect that the program prints detailed information about the package path I run it with. But whatever the package, or module, I try to use this with, the program is killed because it takes all the memory (~32GB) of my machine in a few seconds, nothing ever gets printed...

Here's my config:

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/michel/.cache/go-build"
GOENV="/home/michel/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/michel/.go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/michel/.go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.16"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3825502007=/tmp/go-build -gno-record-gcc-switches"

I have tried many, many things yesterday, including:

- changing the `Mode` in the config

- using go/types directly ("can't find import" for the package I'm look to parse)

- using different importers

- trying to load and parse different packages

- ...

For context, and to avoid any XY problem here, my goal is to parse a package and find an interface based on its name. Once this interface is found, I need to range over its method set and generate a structure implementing this interface, with (maybe not at the beginning but) a lot of logic, e.g. detect that an interface returns an implementation of itself, and so on.

I'm working on a tool to generate mock structures from their interface method set, as a personal project, and this is kind of the most important part of it (being able to automatically generate the mock).

If anyone would kindly help me find what I'm doing wrong, or at least point me to useful resources explaining how to fix my problem, I would be reaaaaaally delighted. This has been a problem for days now... And I can't find any relevant issue or blog as this is a peculiar context.

Thanks in advance to all that will read this and have a nice day! :D

[Brian Candler]

The problem appears to be with litter, not with x/tools/go/packages

On my machine, this hangs after "Stage B":

https://play.golang.com/p/N2MTQszvnRN

[jake...@gmail.com]

The first thing I would do is remove the call to litter, and see if that solved the issue. That would tell you immediately if the problem was the litter package or the packages package. I have so specific knowledge, but it is not impossible to imagine that you are simply trying to print something way to big for litter.

After that, using pprof might be your next step.

Have you tried it on a really tiny package?

[Robert Engels]

Since litter checks for circular references it needs to keep a ref to every object it sees. 

With a large tree you will run out of memory. 

On Jul 21, 2021, at 7:19 AM, jake...@gmail.com <jake...@gmail.com> wrote:



--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/36cd11af-ee05-4e9e-8324-51212c80d99cn%40googlegroups.com.

[Brian Candler]

But AFAICT, it should generate output as it runs.  The fact that it doesn't generate any output at all is suspicious.

[Levieux Michel]

Thx all for the response!

Giving myself a (tremendous) facepalm, it was _indeed_ the use of litter that destroyed my machine memory. Everything's working fine now.

Sorry for the unnecessary noise guys. *retreating in shame*

Have a great day!

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/2aa03542-c6a4-4ace-8c84-b188c19b2ffdn%40googlegroups.com.

[Brian Candler]

Not at all... I think this is a nice reproducible case that you should raise as an issue in the Litter repo.

[Robert Engels]

If it is pretty printing it may need to determine the lowest level for indentation - so a depth first search - causing the entire tree to be traversed and retained before it can output anything. 

On Jul 21, 2021, at 9:40 AM, Brian Candler <b.ca...@pobox.com> wrote:

But AFAICT, it should generate output as it runs.  The fact that it doesn't generate any output at all is suspicious.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/2aa03542-c6a4-4ace-8c84-b188c19b2ffdn%40googlegroups.com.

[Daniel T. Gorski]

> If anyone would kindly help me find what I'm doing wrong, or at least point me to useful resources

> explaining how to fix my problem, I would be reaaaaaally delighted.

Maybe a look at typex [1,2] helps.

[1] https://github.com/dtgorski/typex

[2] https://github.com/dtgorski/typex/blob/master/internal/packagist.go

[Brian Candler]

I had a brief look at the code and it doesn't appear to do that.  As far as I can see, it indents as it goes: where you start is the "top" level, and it increases the indentation for each level of children.

The individual dump functions write directly to the writer, not to a buffer (except when an object has a custom dumper, which means it has a "LitterDump" method, in which case the output of LitterDump *is* buffered and realigned)

[Robert Engels]

Maybe the object itself has circular references - so it fails at the first level. 

On Jul 21, 2021, at 11:25 AM, Brian Candler <b.ca...@pobox.com> wrote:

I had a brief look at the code and it doesn't appear to do that.  As far as I can see, it indents as it goes: where you start is the "top" level, and it increases the indentation for each level of children.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/5ebcea65-2833-4f5c-9ded-559d51065ac9n%40googlegroups.com.