The Disintegrated Development Environment Problem

[Alessandro Arzilli]

Over the past few months we've had a few issues related to users being
unable to set breakpoints using GoLand caused by an overly complicated
development environment.

Summary of the problem:
- user develops under /home/username/project/
- user copies source into a container to build it in the directory /project/
- user asks GoLand to set a breakpoint on file.go
- GoLand tells delve to set a breakpoint on /home/username/project/file.go
- delve can't find /home/username/project/file.go because it didn't actually
partecipate in the build.

Let's call this problem the Disintegrated Development Environment (DDE)
Problem.

I think this problem also exists with other frontends and it's only being
reported for GoLand because GoLand at the moment has a better delve frontend
and, as a consequence, it get used.

Apparently GoLand uses some magic to automatically map between
files-in-the-binary and files-on-the-developer-machine but, also apparently,
this magic isn't strong enough and sometimes fails.

I think delve can help make the magic stronger and the incantation simpler.

A few definitions:

* import path: the path that would be specified in an import statement to
import a package, also the value of the ImportPath field in the output of
'go list'

* directory path: the path of the directory containing a package, also the
value of the Dir field in the output of 'go list'

I think that delve can use debug_info and the file table of debug_line to
construct a mapping between import paths and directory paths (hencefort the
IP2DP map) for each package included in a given binary and then return it to
the IDE with an API call.

An IDE that wishes to tackle the DDE problem would first get delve's IP2DP
map (delveIP2DP) and then construct a second IP2DP map (localIP2DP) by
either calling 'go list --json' or by replicating its functionality.

Then when receiving a filename from delve the following conversion would
happen:

func toLocal(path string) string {
basename := filepath.Base(path)
dir := filepath.Dir(path)
ip := delveIP2DP.findImportPathForDirectory(dir)
localDir := localIP2DP.findDirectoryForImportPath(ip)
return filepath.Join(localDir, basename)
}

And when sending a filename to delve (for example when setting a breakpoint)
this conversion would have to be done:

func toDelve(path string) string {
basename := filepath.Base(path)
dir := filepath.Dir(path
ip := localIP2DP.findImportPathForDirectory(dir)
delveDir := delveIP2DP.findDirectoryForImportPath(ip)
return filepath.Join(delveDir, basename)
}


@ Florin Patan: does this sound like something GoLand can do? There is no
point in delve implementing this feature if no IDE is going to take
advantage of this.

== How the IP2DP table can be constructed inside delve ==

Each package is a separate compilation unit with a separate debug_line
section. The import path of a package can be found in the DW_AT_name
attribute of the corresponding DW_TAG_compile_unit.

In absence of inlining we can get the package directory by simply opening
debug_line section of a package, reading a random entry in the file table
and returning the output of filepath.Dir.

I don't think we can do the same with inlining on and be 100% accurate but I
think that in practice we can use the first file table entry used in the
debug_line program and be right. We'd only make a mistake if the first
instruction emitted for a given compile unit happened to be from a function
inlined from a different package which should be relatively unlikely (go
functions almost always start with a prologue so it would have a go:nosplit
function inlining a function from a different package).

[Florin Pățan]

@Alessandro thanks for the ping on this.

I don't know if we can take advantage of this or not as we fixed all the

outstanding issues we had reported.

We already have something similar to the IP2DP that you mentioned,

but it had some issues in the presence of the "-trimpath" and other options

present in users configurations.

I'll CC my colleague who's the owner of the Debugger subsystem, Dmitry,

and let him give more feedback on this.

As for other environments, think we are the only ones that take care of these things

in particular, but I'd be curious how vim or vscode manage to work around this problem.

You'll also have to consider that delve might not always be part of the build process,

as people might want to debug existing running applications/containers, or using things

such as Bazel, where this gets even more complex (and afaik, the Bazel plugin provides

more functionality about this part when needed).

Florin

[Alessandro Arzilli]

On Sun, Nov 24, 2019 at 03:35:50PM +0200, Florin Pățan wrote:
> @Alessandro thanks for the ping on this.
>
> I don't know if we can take advantage of this or not as we fixed all the
> outstanding issues we had reported.
>
> We already have something similar to the IP2DP that you mentioned,
> but it had some issues in the presence of the "-trimpath" and other options
> present in users configurations.

I'm curious how you are obtaining this mapping without looking at the binary
for build in modules mode.

>
> I'll CC my colleague who's the owner of the Debugger subsystem, Dmitry,
> and let him give more feedback on this.
>
> As for other environments, think we are the only ones that take care of
> these things
> in particular, but I'd be curious how vim or vscode manage to work around
> this problem.

I'm pretty sure they don't.

> You'll also have to consider that delve might not always be part of the
> build process,
> as people might want to debug existing running applications/containers, or
> using things
> such as Bazel, where this gets even more complex (and afaik, the Bazel
> plugin provides
> more functionality about this part when needed).

It doesn't matter, the method I'm proposing just looks at the binary file
and should works even when attaching to a pre-existing process.

[Dmitry Neverov]

I'm curious how you are obtaining this mapping without looking at the binary
for build in modules mode.

Hi Alessandro,

at the moment we use the following heuristic for path mapping if exact match isn't found. We trim project root dir/go sdk root from a local path and try to find a unique remote file with such a suffix. If unique file is not found we still fail to map, so I it looks like information about imports from debugger can help us with more precise mapping.