Help with custom go_rule

[duarte....@gmail.com]

I'm trying to write a custom rule for gqlgen. 

This is the code for the custom rule: https://gist.github.com/duarten/db10109f88b6e3899d51d7fdf23cd41f

The intended usage is: 

gqlgen(

    name = "gql-gen",

    autobind = "models/",

    default_package = "test",

    model = "model.go",

    parser = "generated.go",

    resolver = "resolver.go",

    schemas = ["schemas/segmentation.graphql"],

    visibility = ["//visibility:public"],

)

When executing bazel build //:gql-gen, I get the following error:

(Exit 3) gqlgen failed: error executing command
  (cd /private/var/tmp/_bazel_duarten/18d4d4fe625c2f83e91dbc80b9b94787/sandbox/darwin-sandbox/403/execroot/playground && \
  exec env - \
    CGO_ENABLED=1 \
    GOARCH=amd64 \
    GOCACHE=/tmp \
    GOOS=darwin \
    GOPACKAGESDEBUG=1 \
    GOPACKAGESPRINTGOLISTERRORS=1 \
    GOROOT=external/go_sdk \
    GOROOT_FINAL=GOROOT \
    PATH=/usr/bin:external/local_config_cc:/bin:external/go_sdk/bin \
  bazel-out/host/bin/external/com_github_99designs_gqlgen/darwin_amd64_stripped/gqlgen --config bazel-out/darwin-fastbuild/bin/package/gql-gen.yml)
Execution platform: @bazel_tools//platforms:host_platform


Use --sandbox_debug to see verbose messages from the sandbox
GOROOT=external/go_sdk GOPATH= GO111MODULE= PWD= go [list -f {{context.GOARCH}} {{context.Compiler}} -- unsafe] stderr: <<can't load package: package unsafe: cannot find package "unsafe" in any of:
 external/go_sdk/src/unsafe (from $GOROOT)
 ($GOPATH not set. For more details see: 'go help gopath')
>>
modelgen: could not determine GOARCH and Go compiler

Similarly, doing bazel run @com_github_99designs_gqlgen//:gqlgen -- <args> results in "merging failed: unable to find type github.com/99designs/gqlgen/graphql.Boolean".

However, it works when I just do  bazel-bin/external/com_github_99designs_gqlgen/linux_amd64_stripped/gqlgen <args>.

In both cases it seems some dependencies can't be satisfied. 

Can anyone shed some light into what's going on?

Thanks!

[Jay Conrod]

Based on the go list error and the fact you have the environment variables GOPACKAGESDEBUG, GOPACKAGESPRINTGOLISTERRORS set, I'm guessing your tool loads package metadata using golang.org/x/tools/go/packages. That won't work within a Bazel sandbox. There are plans to make it work eventually (see #512 and Editor and tool integration), but there's a lot of stuff to be done.

You may have more luck with go_path, which is a rule that pulls a bunch of Go dependencies into a .zip archive or file tree that follows the GOPATH layout. Most tools that are compatible with that convention should work using that.

Hope that helps.

--
You received this message because you are subscribed to the Google Groups "bazel-go-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-go-discu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-go-discuss/7c73dbc7-2b7c-4a2a-a4d6-7043b7ad56b0%40googlegroups.com.

[Duarte Nunes]

Thanks! Good to know it's in the works :)

I'm looking into go_path, but I'm missing something. I have in my root BUILD:

go_path(
    name = "gqlgen_path",
    mode = "copy",
    deps = ["@com_github_99designs_gqlgen//:gqlgen"],
    include_pkg = True,

    visibility = ["//visibility:public"],
)

Then in my rule I have:

        "_gqlgen": attr.label(
            allow_single_file = True,
            default = "gqlgen_path",
            cfg = "host",
        ),

However, in the rules implementation, the directory given by ctx.file._gqlgen.path doesn't exit.

[Duarte Nunes]

I guess depending on a label matching the rule doesn't trigger its execution. Maybe I need to depend on go_paths's output, but I'm not sure how to do that.

[Duarte Nunes]

Alright, I managed to make some progress using depset:

   ctx.actions.run(
        inputs = depset(inputs),
        outputs = out_files,
        arguments = [
            "run",
            ctx.file._gqlgen.path + "/src/github.com/99designs/gqlgen/main.go",
            "--config",
            config_file.path,
        ],
        executable = go.go,
        env = env,
    )

where "inputs" contains "ctx.file._gqlgen". This triggers the execution of the rule and I can access all of the generated files from the sandbox.

Afterwards I hit GOPATH issues. I need to provide an absolute path, but I'm not sure how to get the current directory from within Bazel. My workaround was to use run_shell instead of run:

ctx.actions.run_shell(
        tools = [go.go],
        inputs = depset(inputs),
        outputs = out_files,
        command = """
            export GOCACHE="/tmp" &&
            export GOPATH="$(pwd)/{gqlgen}" &&
            export PATH={path} &&
            go run "{gqlgen}/src/github.com/99designs/gqlgen/main.go" \
                --config {config}
        """.format(
            gqlgen = ctx.file._gqlgen.path,
            path = go.env["PATH"] + ":{}/bin".format(go.env["GOROOT"]),
            config = config_file.path,
        ),
    )

Is there a better approach or is this pretty much it?

Thanks!

[Austin Schuh]

Getting the absolute path from within Bazel is very much an anti pattern.  If you think about it, putting the absolute path in a key in the cache would cause a lot of cache misses.  When I have needed the absolute path in the past, I have used a shell script and expanded relative paths.  Sounds like you found that solution too.

Austin

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-go-discuss/CAEoxL3U5uN5Qy5sTA7onj40%2Bh29ERMcrzhBwyyciUEiwMgScJw%40mail.gmail.com.

[Duarte Nunes]

Makes sense, thanks for the explanation.

One more question: go_path seems to only include .go files, while this program in particular would need need to access .gotpl files. Is there a way to achieve that? Setting include_data in 

go_path(
    name = "gqlgen_path",
    mode = "copy",
    deps = ["@com_github_99designs_gqlgen//:gqlgen"],
    include_pkg = True,

    include_data = True,

    visibility = ["//visibility:public"],
)

doesn't seem to help.

[Jay Conrod]

By default, go_path only includes source files needed to build targets listed in deps and their dependencies. include_pkg will also include .a files. include_data will include runfiles listed in the data attributes of those targets. So make sure your .gotpl files are listed in a data attribute somewhere. go_path itself also has a data attribute you can use.

If you're consuming this in a separate rule that reads .gotpl files, it might make more sense to define an attribute on that rule for them instead.

[Duarte Nunes]

I'm not consuming .gotpl files directory, I'm using go_path to be able to run the extracted library, which internally consumes .gotpl files. I also tried using the go_path's data attribute, but I'm not sure about the exact syntax. In particular, I tried doing

go_path(
        name = "gqlgen_path",
        mode = "copy",
        deps = ["@com_github_99designs_gqlgen//:gqlgen"],

        data = native.glob(["**/*.gotpl"]),

        visibility = ["//visibility:public"],
    )

without much success. It's not clear to me what the path in data should be relative too.

If I write "data = ["@com_github_99designs_gqlgen//plugin/modelgen:models.gotpl"],", then I get 

"no such target '@com_github_99designs_gqlgen//plugin/modelgen:models.gotpl': target 'models.gotpl' not declared in package 'plugin/modelgen'; however, a source file of this name exists.  (Perhaps add 'exports_files(["models.gotpl"])' to plugin/modelgen/BUILD?) defined by /private/var/tmp/_bazel_duarten/18d4d4fe625c2f83e91dbc80b9b94787/external/com_github_99designs_gqlgen/plugin/modelgen/BUILD.bazel and referenced by '//:gqlgen_path'".

I assume the referenced BUILD.bazel file is generated by go_repository().

[Jay Conrod]

The data attribute you wrote looks correct.

Is it possible that the glob would need to cross a Bazel package boundary to find files? A Bazel package is defined by a BUILD or BUILD.bazel file, and globs can't locate files in other packages. The glob doc has more info. If the glob would need to cross a package boundary, you could define a filegroup in the package where the .gotpl files appear, then reference the filegroup in the data attribute instead of using a glob.

For the "no such target" error you mentioned, you just need an exports_files target in the build file for that directory that lists all the files that might be referenced. Source files can't be referenced outside of the Bazel package that contains them unless they're explicitly mentioned in their build file. exports_files doesn't actually do anything; it just serves as a place to mention files that aren't mentioned elsewhere for this purpose.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-go-discuss/CAEoxL3Vo1O5aO6SnnSbwC_Tw8gmjtzZ6DtNX%2BbAjhQST0AMw2Q%40mail.gmail.com.

[Duarte Nunes]

The issue is then that I have no control over that directory. To recap, I'm using go_path as a workaround for the inability to directly execute a go program in the sandbox, in the context of a custom rule. Using go_path, I'm able to execute a run_shell action to run the go main package, but that in turn depends on *.gotpl files that are not extracted. And I can't add filegroup or exports_files rules to the BUILD files since they are generated by go_repository() :(

[Jay Conrod]

go_repository lets you apply patches to a repository after build files are generated. The attributes are the same as the ones on http_archive. This seems like a good use case for that.

You can create a patch by 1) copying `$(bazel info output_base)/external/com_github_99designs_gqlgen` to a directory named "a", 2) make another copy to a directory named "b", 3) make any changes you need in "b", 4), from the parent directory of "a" and "b", run `diff -urN a b >com_github_99designs_gqlgen.patch`. You can reference the patch in `patches`, and you'll probably want to set `patch_args = ["-p1"]`.

[Duarte Nunes]

I'm now applying the patch to add exports_files directives to the BUILD.bazel files generated by go_repository, and indeed the following go_path rule succeeds: 

go_path(
        name = "gqlgen_path",
        mode = "copy",
        deps = ["@com_github_99designs_gqlgen//:gqlgen"],

        data = ["@com_github_99designs_gqlgen//plugin/modelgen:models.gotpl"],

        include_data = True,
        visibility = ["//visibility:public"],
    )

However, I still don't see the models.gotpl file in the sandbox, so gqlgen still doesn't work. Could this be a bug in go_path?

[Jay Conrod]

If you believe this is a bug, please create a minimal reproducible test case and file an issue in rules_go.

Files listed in the data attribute should appear in the go_path root directory. The attribute was mainly intended for files like README and LICENSE, so that's where those go.

[Duarte Nunes]

They do appear in the root path. For this particular case, I would need them to appear in their original path, where the program itself expects them to be. 

I opened https://github.com/bazelbuild/rules_go/issues/2187.

[Jay Conrod]

That's working as intended though. Data files are either placed in the go_path root directory (when listed in the data attribute of go_path) or in a src/<importpath> directory (data attribute of go_library or other dependency of go_path). If you need them somewhere else, go_path probably isn't the right tool for the job.

[Duarte Nunes]

Are there any other options then? I can't run gqlgen directly because it loads package metadata, and I can't run it through go_path because it depends on template files that are placed under a different directory.

[Jay Conrod]

If you need the the file to be in src/github.com/99designs/gqlgen/plugin/modelgen, then patch the data attribute of @com_github_99designs_gqlgen//plugin/modelgen:go_default_library.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-go-discuss/CAEoxL3WrBn6owMihqJS4gEsKOi%3DUdV2AmMkAY4yiMFPCSPdsbQ%40mail.gmail.com.

[Duarte Nunes]

Thanks, that gets me closer!

However, now it seems that the files listed in the data attribute of go_library are linked in every directory. For example:

...

./bazel-genfiles/gqlgen_path/src/github.com/99designs/gqlgen/cmd/directives.gotpl
./bazel-genfiles/gqlgen_path/src/github.com/99designs/gqlgen/cmd/args.gotpl
./bazel-genfiles/gqlgen_path/src/github.com/99designs/gqlgen/cmd/field.gotpl
./bazel-genfiles/gqlgen_path/src/github.com/99designs/gqlgen/cmd/type.gotpl
./bazel-genfiles/gqlgen_path/src/github.com/99designs/gqlgen/cmd/resolver.gotpl
./bazel-genfiles/gqlgen_path/src/github.com/99designs/gqlgen/cmd/generated!.gotpl
./bazel-genfiles/gqlgen_path/src/github.com/99designs/gqlgen/cmd/interface.gotpl
./bazel-genfiles/gqlgen_path/src/github.com/99designs/gqlgen/cmd/input.gotpl
./bazel-genfiles/gqlgen_path/src/github.com/99designs/gqlgen/cmd/models.gotpl
./bazel-genfiles/gqlgen_path/src/github.com/99designs/gqlgen/cmd/object.gotpl

...

./bazel-genfiles/gqlgen_path/src/github.com/99designs/gqlgen/api/directives.gotpl
./bazel-genfiles/gqlgen_path/src/github.com/99designs/gqlgen/api/args.gotpl
./bazel-genfiles/gqlgen_path/src/github.com/99designs/gqlgen/api/field.gotpl
./bazel-genfiles/gqlgen_path/src/github.com/99designs/gqlgen/api/type.gotpl
./bazel-genfiles/gqlgen_path/src/github.com/99designs/gqlgen/api/resolver.gotpl
./bazel-genfiles/gqlgen_path/src/github.com/99designs/gqlgen/api/generated!.gotpl
./bazel-genfiles/gqlgen_path/src/github.com/99designs/gqlgen/api/interface.gotpl
./bazel-genfiles/gqlgen_path/src/github.com/99designs/gqlgen/api/input.gotpl
./bazel-genfiles/gqlgen_path/src/github.com/99designs/gqlgen/api/models.gotpl
./bazel-genfiles/gqlgen_path/src/github.com/99designs/gqlgen/api/object.gotpl

...

Is this the expected behavior? 

For reference, this is the patch I'm applying:

diff -urN codegen/BUILD.bazel codegen/BUILD.bazel
--- codegen/BUILD.bazel 2019-08-20 16:36:43.000000000 -0300
+++ codegen/BUILD.bazel 2019-08-20 16:40:02.000000000 -0300
@@ -1,6 +1,16 @@
 load("@io_bazel_rules_go//go:def.bzl", "go_library", "go_test")
 
 go_library(
+    data = [
+        "directives.gotpl",
+        "args.gotpl",
+        "field.gotpl",
+        "type.gotpl",
+        "generated!.gotpl",
+        "interface.gotpl",
+        "input.gotpl",
+        "object.gotpl"
+    ],
     name = "go_default_library",
     srcs = [
         "args.go",
diff -urN plugin/modelgen/BUILD.bazel plugin/modelgen/BUILD.bazel
--- plugin/modelgen/BUILD.bazel 2019-08-20 16:36:43.000000000 -0300
+++ plugin/modelgen/BUILD.bazel 2019-08-20 16:40:38.000000000 -0300
@@ -1,6 +1,7 @@
 load("@io_bazel_rules_go//go:def.bzl", "go_library", "go_test")
 
 go_library(
+    data = ["models.gotpl"],
     name = "go_default_library",
     srcs = ["models.go"],
     importpath = "github.com/99designs/gqlgen/plugin/modelgen",
diff -urN plugin/resolvergen/BUILD.bazel plugin/resolvergen/BUILD.bazel
--- plugin/resolvergen/BUILD.bazel 2019-08-20 16:36:43.000000000 -0300
+++ plugin/resolvergen/BUILD.bazel 2019-08-20 16:41:20.000000000 -0300
@@ -1,6 +1,7 @@
 load("@io_bazel_rules_go//go:def.bzl", "go_library", "go_test")
 
 go_library(
+    data = ["resolver.gotpl"],
     name = "go_default_library",
     srcs = ["resolver.go"],
     importpath = "github.com/99designs/gqlgen/plugin/resolvergen",
diff -urN plugin/stubgen/BUILD.bazel plugin/stubgen/BUILD.bazel
--- plugin/stubgen/BUILD.bazel 2019-08-20 16:36:43.000000000 -0300
+++ plugin/stubgen/BUILD.bazel 2019-08-20 16:42:09.000000000 -0300
@@ -1,6 +1,7 @@
 load("@io_bazel_rules_go//go:def.bzl", "go_library")
 
 go_library(
+    data = ["stubs.gotpl"],
     name = "go_default_library",
     srcs = ["stubs.go"],
     importpath = "github.com/99designs/gqlgen/plugin/stubgen",

[Aarthi K]

Hi Duarte,

Were you able to get this working?

[Duarte Nunes]

Hi Aarthi,

I think I was able to get something working (I can share the code I wrote), but I've since stopped using gqlgen.

You received this message because you are subscribed to a topic in the Google Groups "bazel-go-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bazel-go-discuss/0S89JVxHB9A/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bazel-go-discu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-go-discuss/0ff5130e-9abc-46eb-ae87-5dea66e07922n%40googlegroups.com.

[Aarthi K]

That'd be great, Duarte. I got to the point of moving the .gotpl files, but my output files don't seem to be generated in the expected location.
Any code that you have would be helpful.

[Bryant Khau]

Hi Duarte,

I'm also stuck on getting .gotpl files working. Would you be able to share what you got working?