x/vgo: How can I grep all of myapp's dependencies?

91 views
Skip to first unread message

Peter Waller

unread,
May 31, 2018, 5:50:25 AM5/31/18
to golang-nuts
Hi All,

This may have an easy/obvious answer, but it's not coming to me immediately.

In the old world, I had all of my dependencies in `myapp/vendor`. If I wanted to search for an error message I could just use ag/ack/ripgrep/grep -R and search through all of myapp's dependencies as they are compiled into myapp.

With vgo, `myapp`'s dependencies aren't in a subdirectory by themselves anymore, they live at `$GOPATH/src/v/...`, where they may exist with other dependencies, and multiple other versions of the same packages. I want to avoid grepping those.

One solution, I suppose, would be to set GOPATH to a new empty location, do a vgo install there, then grep that. I would like to avoid doing this if possible. Partly because this currently takes a while, makes another copy of the cod eand itself has to be kept uptodate.

Any nice ideas, or is there scope for some sort of tool to help with this? `vgo grep`, or a fuse filesystem which puts an app's dependencies in one place where they can be grepped without a copy ? :)

- Peter

Paul Jolly

unread,
May 31, 2018, 6:04:22 AM5/31/18
to pe...@pdftables.com, golan...@googlegroups.com
Hi Peter, 

You can use vgo list for something like this (although this does not preclude other tools/solutions). Taking an example:

cd `mktemp -d`
export GOPATH=$PWD
mkdir hello
cd hello
cat <<EOD >hello.go
package main // import "example.com/hello"

import (
"fmt"
)

func main() {
fmt.Println(vgo_example_compat.X, sub.Y)
}
EOD
echo >go.mod
vgo build
./hello

gives:

2 2

Now we can list the non-standard library dependencies:

vgo  list -f '{{ join .Deps  "\n"}}' ./... | xargs vgo list -f "{{if not .Standard}}{{.ImportPath}}{{end}}" . 

which gives:


Or grep their directories (recursively):

vgo  list -f '{{ join .Deps  "\n"}}' ./... | xargs vgo list -f "{{if not .Standard}}{{.Dir}}{{end}}" . | xargs grep -r const

which gives:




Paul

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Peter Waller

unread,
May 31, 2018, 6:16:16 AM5/31/18
to Paul Jolly, golang-nuts
That is interesting, thanks for the quick input!

One minor issue, that `vgo list` results in a lot of bits of path appearing twice, for example:


Which I guess means one should not do a recursive search on those paths, but only look in those exact directories. It will result in searching exactly the packages which are imported, which is a nice in many circumstances.

What if I wanted to just get a list of module paths, so $GOPATH/src/v/github.comlib/pq@.../ in this case, but not `.../oid` as above? Is that straightforward?

In the list output there are some modules where the module path itself is not listed because it is not actually a go package (or not in the package dependency graph), but I would still like to grep it - for example to include READMEs appearing at module level in the search.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.

Paul Jolly

unread,
May 31, 2018, 6:25:51 AM5/31/18
to pe...@pdftables.com, golan...@googlegroups.com
One minor issue, that `vgo list` results in a lot of bits of path appearing twice, for example:


Which I guess means one should not do a recursive search on those paths, but only look in those exact directories. It will result in searching exactly the packages which are imported, which is a nice in many circumstances.

Good catch. In which case you can do something like:

vgo  list -f '{{ join .Deps  "\n"}}' ./... | xargs vgo list -f "{{if not .Standard}}{{.Module.Path}}{{end}}" . | xargs vgo list -f "{{.Dir}}"

which gives:

/tmp/tmp.etqFtpbatu/hello

As you can see, vgo list is similar to go list. I'm not really doing anything other than manipulating that JSON. Hence the use of -f. To see the full output do something like:

$ vgo list -json .
{
        "Dir": "/tmp/tmp.etqFtpbatu/hello",
        "ImportPath": "example.com/hello",
        "Name": "main",
        "Target": "/tmp/tmp.etqFtpbatu/bin/hello",
        "Stale": true,
        "StaleReason": "target missing",
        "GoFiles": [
                "hello.go"
        ],
        "Imports": [
                "fmt",
                "github.com/myitcv/vgo_example_compat/v2",
        ],
        "Deps": [
                "errors",
                "fmt",
                "github.com/myitcv/vgo_example_compat/v2",
                "internal/bytealg",
                "internal/cpu",
                "internal/poll",
                "internal/race",
                "internal/syscall/unix",
                "internal/testlog",
                "io",
                "math",
                "math/bits",
                "os",
                "reflect",
                "runtime",
                "runtime/internal/atomic",
                "runtime/internal/sys",
                "strconv",
                "sync",
                "sync/atomic",
                "syscall",
                "time",
                "unicode",
                "unicode/utf8",
                "unsafe"
        ],
        "Module": {
                "Top": true,
                "Path": "example.com/hello",
                "Version": ""
        }
}

And for more information on (v)go list:

go help list

Peter Waller

unread,
May 31, 2018, 6:36:19 AM5/31/18
to Paul Jolly, golang-nuts
On 31 May 2018 at 11:24, Paul Jolly <pa...@myitcv.io> wrote:
To see the full output do something like:

$ vgo list -json .

Aha, a great little trick, and, obvious in hindsight :). Thanks for the good building blocks.

It's tricky this. If I start feeding the .Module.Path into vgo list, then I start finding lots of additional dependencies which are actually not used by my package, because then the vgo list appears to operate on all packages in the module, so again I end up with a list of paths which includes more modules than myapp actually uses.

Hmm.

Paul Jolly

unread,
May 31, 2018, 6:46:18 AM5/31/18
to golang-nuts
It's tricky this. If I start feeding the .Module.Path into vgo list, then I start finding lots of additional dependencies which are actually not used by my package, because then the vgo list appears to operate on all packages in the module, so again I end up with a list of paths which includes more modules than myapp actually uses.

Perhaps this does the trick?

find $(vgo  list -f '{{ join .Deps  "\n"}}' ./... | xargs vgo list -f "{{if not .Standard}}{{.Dir}}{{end}}" .) -maxdepth 1 -type f | xargs grep const 

gives:

Peter Waller

unread,
May 31, 2018, 6:52:35 AM5/31/18
to Paul Jolly, golang-nuts
On 31 May 2018 at 11:46, Paul Jolly <pa...@myitcv.org.uk> wrote:
Perhaps this does the trick?

find $(vgo  list -f '{{ join .Deps  "\n"}}' ./... | xargs vgo list -f "{{if not .Standard}}{{.Dir}}{{end}}" .) -maxdepth 1 -type f | xargs grep const 

gives:


As I understand it, this won't grep v2/goodbye.go (or v2/README) if the myapp only imports .../v2/sub/ (and not .../v2), since that path won't appear as {{.Dir}}. (I tried to express this previously, but hopefully it's clearer written with respect to your example).

Paul Jolly

unread,
May 31, 2018, 9:59:34 AM5/31/18
to pe...@pdftables.com, golan...@googlegroups.com
As I understand it, this won't grep v2/goodbye.go (or v2/README) if the myapp only imports .../v2/sub/ (and not .../v2), since that path won't appear as {{.Dir}}. (I tried to express this previously, but hopefully it's clearer written with respect to your example).

A module directory will contain all the packages in that module, yes. But it will not contain any submodules (Go submodule, not git submodule).

So if you want to exclusively grep the contents of the packages that you import use:

find $(vgo  list -f '{{ join .Deps  "\n"}}' ./... | xargs vgo list -f "{{if not .Standard}}{{.Dir}}{{end}}" .) -maxdepth 1 -type f | xargs grep const 

If you want to recursively search the modules (and all the packages contained therein) imported by your code use:

vgo  list -f '{{ join .Deps  "\n"}}' ./... | xargs vgo list -f "{{if not .Standard}}{{.Module.Path}}{{end}}" . | xargs vgo list -f "{{.Dir}}" | xargs grep -r const

If you want something else, then apologies, I'm probably missing something!


Paul

Peter Waller

unread,
May 31, 2018, 10:34:29 AM5/31/18
to Paul Jolly, golang-nuts
On 31 May 2018 at 14:59, Paul Jolly <pa...@myitcv.io> wrote:
If you want something else, then apologies, I'm probably missing something!
 
It's subtle. The issue is that a module can depend on other modules which actually aren't compiled into myapp. Consider the following:

myapp => pkga/foo

  pkga/foo => pkgb
  pkga/bar => huge tree of dependencies.

(that is, myapp imports pkga/foo but not pkga/bar).

In the old world, since myapp only imports pkga/foo, only pkga and pkgb are vendored; I run grep in /vendor/, all is good. The huge tree of dependencies is ignored.

Also neatly, vgo install avoids pkga/bar and its huge tree of dependencies since those are not imported by my program (AIUI). It doesn't bother fetching them, in particular.

Using the suggested commands:

    vgo  list -f '{{ join .Deps  "\n"}}' myapp/...

This would for example produce pkga/foo, because myapp depends on it.

Then, when I feed `pkga/foo` along with others into .Module.Path:

  `vgo list -f '{{.Module.Path}}' pkga/foo etc`

I get `pkga` (and the other modules).

Now (after `sort --unique`) I feed pkga and friends into xargs vgo list -f '{{.Dir}}', it starts resolving (and downloading) dependencies for a large tree of packages which are unused by my program, but pulled in through pkga/bar.

Also, if it completes that download, then consider that it runs something like this (protobuf taken as an example transitive dependency):

  vgo list -f "{{.Dir}}" pkga pkgb pkgc github.com/golang/protobuf

That actually outputs nothing (not even the valid paths to pkga,pkgb,pkgc), it gives only an error instead:

  vgo: import "github.com/golang/protobuf" [/home/pwaller/.local/src/v/github.com/golang/protobuf@v1.1.0]: no Go source files

Because the top level directory of golang/protobuf is not an importable package.

Not sure if I'm at fault here, or if there is a tooling improvement that could be made.

Just to try and restate my current understanding of my problem and what I'd like to get to:

* I'd like to know the root paths-on-disk of all modules compiled in to myapp.
* I don't want vgo to spend bandwidth and time downloading modules of transitive dependencies of myapp which are not compiled in to `myapp`.

Currently, I think I'm blocked on simply on translating a module name to a directory, if the root of the module is not also importable.

(I could surely hack my way through this, but really I'd love to have a clean approach).

Paul Jolly

unread,
May 31, 2018, 10:44:14 AM5/31/18
to pe...@pdftables.com, golan...@googlegroups.com
Ah sorry, I think I see where I missed what you were getting at.

You want to be using .Imports (and potentially .TestImports and .XTestImports) as opposed to .Deps (in my examples above). .Deps is the transitive set of non-test dependencies. The .*Imports fields are the direct imports.


Paul


On Thu, 31 May 2018 at 15:34, Peter Waller <pe...@pdftables.com> wrote:
On 31 May 2018 at 14:59, Paul Jolly <pa...@myitcv.io> wrote:
If you want something else, then apologies, I'm probably missing something!
 
It's subtle. The issue is that a module can depend on other modules which actually aren't compiled into myapp. Consider the following:

myapp => pkga/foo

  pkga/foo => pkgb
  pkga/bar => huge tree of dependencies.

(that is, myapp imports pkga/foo but not pkga/bar).

In the old world, since myapp only imports pkga/foo, only pkga and pkgb are vendored; I run grep in /vendor/, all is good. The huge tree of dependencies is ignored.

Also neatly, vgo install avoids pkga/bar and its huge tree of dependencies since those are not imported by my program (AIUI). It doesn't bother fetching them, in particular.

Using the suggested commands:

    vgo  list -f '{{ join .Deps  "\n"}}' myapp/...

This would for example produce pkga/foo, because myapp depends on it.

Then, when I feed `pkga/foo` along with others into .Module.Path:

  `vgo list -f '{{.Module.Path}}' pkga/foo etc`

I get `pkga` (and the other modules).

Now (after `sort --unique`) I feed pkga and friends into xargs vgo list -f '{{.Dir}}', it starts resolving (and downloading) dependencies for a large tree of packages which are unused by my program, but pulled in through pkga/bar.

Also, if it completes that download, then consider that it runs something like this (protobuf taken as an example transitive dependency):

  vgo list -f "{{.Dir}}" pkga pkgb pkgc github.com/golang/protobuf

That actually outputs nothing (not even the valid paths to pkga,pkgb,pkgc), it gives only an error instead:

  vgo: import "github.com/golang/protobuf" [/home/pwaller/.local/src/v/github.com/golang/prot...@v1.1.0]: no Go source files
Reply all
Reply to author
Forward
0 new messages