Type-checking from Memory?

[K. Alex Mills]

Hi All,

Right now, I can run the type-checker by writing the GitHub tarball contents to disk and using the packages.Load function to parse and type-check the contents.

Unfortunately, all I've been able to make work at the moment with packages.Load would entail writing the entire repository contents to disk... only to read them back into memory... and then delete them. This works, but it could be better.

I'm curious if anyone is aware of a way to parse and type-check multiple packages (and making type information available to multiple packages in the process) for source code that "lives" in memory.

I've tried using the very conspicuous Overlay field from packages.Config and passing a map from a fake filepath to the contents of the file in memory. I've found it challenging to find an acceptable combination of Dir and Overlay that will make this work (if one exists).

I've also noticed that packages provides this option for external drivers but I'm not sure it fits my use-case, since it seems like it would involve IPC.

If I'm passionate about not writing these files to disk, is there a way I can invoke the type-checker for multiple packages without reinventing a large part of what packages provides?

Thanks,

K. Alex Mills

[K. Alex Mills]

On a second reading, it turns out I do not proof-read all of my emails thoroughly enough before hitting send. My apologies for any confusion and/or alarm.

In the meantime, I have solved the problem, so I'm following up here with more context and my solution in case anyone else wants to do something similar and might find the information useful. I also encountered what may be a small documentation hole (or maybe I'm missing something).

The GitHub Vet project reads Go repositories from GitHub and performs static analysis. As part of improving the analyzers, I've begun incorporating the type-checker. 

What I've seen work so far is a setup like this, based on the packages package

 var fset token.FileSet
  config := packages.Config{
    Dir:  srcDir,
    Fset: &fset,
    Mode: packages.NeedTypesInfo | packages.NeedSyntax | packages.NeedTypes,
  }
  pkgs, err := packages.Load(&config, "./...")

Unfortunately, this approach requires that I first write the repository contents to disk, so they can be read in from srcDir. I do not want to send the repository contents to disk if I can avoid it.

But it seems from the packages.Config documentation that using the disk may not be necessary. packges.Config contains this very suggestive Overlay field which I have not been able to figure out how to use (documentation below).

// Overlay provides a mapping of absolute file paths to file contents.
// If the file with the given path already exists, the parser will use the
// alternative file contents provided by the map.
//
// Overlays provide incomplete support for when a given file doesn't
// already exist on disk. See the package doc above for more details.
Overlay map[string][]byte

(FWIW, I couldn't find the details referenced in the package doc).

I have tried providing a map[string][]byte to this argument with no success. I am using the path of each .go file as a key, and including the []byte contents of the file. Based on the what I am seeing, ISTM that there is an undocumented dependency between the paths used as keys in the Overlay and the Config.Dir field which I was unable to get a handle on using black-box hackery.

I also tried digging into the source of the packages.Load method, but there's a layer of indirection involved via a third-party driver so I found it challenging to get a handle on where Overlay is actually being used.

So I'm left wondering whether there's any way to make the packages package work without relying on the filesystem. I'm sure there is, but it's difficult (for me at least) to understand.

OTOH I was able to get the loader package to type-check from memory by parsing the files into a map[string][]*ast.File using something like the below snippet.

  var filesByPath map[string][]*ast.File

  config := loader.Config{
    Fset:        fset,
    AllowErrors: true,
    TypeChecker: types.Config{
      Importer: importer.ForCompiler(fset, "gc", nil)
    },
  }

  for dir, files := range filesByPath {
    config.CreateFromFiles(dir, files...)
  }
  prog, err := config.Load()

So here's hoping this is helpful to others who may have the same (albeit obscure) goal. I'm aware the loader package predates Go modules, so this approach may miss something important for some use-cases. For my purposes, I want to avoid downloading / type-check module dependencies anyway.

Sincerely,

K. Alex Mills