Load function from golang.org/x/tools/go/packages throws "argument list too long" error

[adam...@uber.com]

I am trying to analyze a large application using Go's static analysis support. I am using the "packages" package to load all application packages using the Load function.  Unfortunately the application is really big and the list of patterns that the Load function takes the second parameter is really long. This causes a problem on Mac OS as in this case it leads to the "argument list too long" error popping up after the Load function is called. The reason for it is that internally, the Load function executes "go list" command passing it the list of patterns as arguments, which leads to exceeding the limit established by Mac OS (which, as far as I can tell, cannot be changed).

The usual workaround for this problem is to split the argument list and apply a given function incrementally to each list fragment - in this case it would mean calling the Load function multiple times with a different subset of the patterns list at each invocation. This, however, does not quite work, as it messes up the SSA representation that I need to generate afterwards using AllPackages function from the "ssautil" package. The AllPackages function constructs a program that contains a "file set" containing all files constituting the program. As far as I can tell, the only way to access this file set during analysis is via the program itself. And this is the gist of the problem, incremental application of the Load function creates a "file set" per increment, and the resulting program during SSA construction will assign only one to the program, making the remaining ones inaccessible during analysis.

I can't think of a viable workaround here (one that I considered is trying to use the loader package but it's deprecated and does not support modules), but perhaps someone will have an advice on how to proceed here...

Adam

[Paul Jolly]

cc golang-tools, bcc golang-nuts

Michael Matloob will be well-placed to advise here.

> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/3dad4625-1b92-4438-b753-c63153c15848%40googlegroups.com.

[Manlio Perillo]

On Wednesday, January 29, 2020 at 11:12:40 PM UTC+1, adam...@uber.com wrote:

I am trying to analyze a large application using Go's static analysis support. I am using the "packages" package to load all application packages using the Load function.

What arguments are you passing to the Load function?

It should suffice a simple "all".

 

  Unfortunately the application is really big and the list of patterns that the Load function takes the second parameter is really long.

Why are you passing a list of patterns to Load? Where are these patterns coming from?

> [...]

 

Adam

Manlio Perillo 

[adam...@uber.com]

The answer to both of the questions is that I followed this examples from the ssa package (LoadWholeProgram):

// Load, parse, and type-check the whole program.
cfg := packages.Config{Mode: packages.LoadAllSyntax}
initial, err := packages.Load(&cfg, "fmt", "net/http")
if err != nil {
    log.Fatal(err)
}

It implies that a list of packages (or package paths) needs to be passed to the Load function. Otherwise, I am not sure how would the tool know what to load, particularly that the GOPATH is set to the tool itself and not to the codebase that the tool analyses.

FYI, Michael Matloob answered this question in the golang-tools list and it seems like it is a legit problem that should be filed as a bug.

Adam