Testing out Skylark remote repositories for auto-configuration

[Damien Martin-guillerez]

Hello all,

I recently merged the changes discussed in goo.gl/fD4ZsY to enable auto-configuration. If you use Bazel at HEAD, you can now use an experimental API to do auto-configuration for using local compilers and other tooling.

A direct application of it is detecting C++ compiler and correct flags automatically (in https://github.com/bazelbuild/bazel/blob/master/tools/cpp/cc_configure.bzl). If you wish to write your own detection mechanism for your own tooling, I will be glad to hear and help you.

If you have a platform where you had to tweak the build to build for it, please test it by adding the following lines to your WORKSPACE file:

load("@bazel_tools//tools/cpp:cc_configure.bzl", "cc_configure")
cc_configure()

Tell me if that worked (or not)!

Finally, the next stop for skylark remote repositories will be support for pypi_library, docker_pull and finally a good documentation (see https://goo.gl/OZV3o0).

[Austin Schuh]

I was successfully able to use ctx.os.environ, ctx.symlink, and ctx.execute to add a user key from $HOME to the sandbox.  Thanks!  This helps a ton.

My only note was that I needed to use ctx.execute to do a mkdir to have a place to do all the symlinks in external/.  Not a big deal.

Austin

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAN3hOS_Q-A6Vf_C%2B68dkueSh3hkJOf%2BFh7rUyFQmXZ_hDa1gOA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Damien Martin-guillerez]

Great!

Normally, creating a symlink should create the directories where the symlink goes so you should not need to ctx.execute.

[Austin Schuh]

Thanks!  That makes it much cleaner.

Austin

[sha...@google.com]

I gave it a quick try for Kythe and encountered some problems. No doubt some are local, but there are definitely a few that need fixes upstream:

First hurdle (trivial fix, use +):

ERROR: .../external/bazel_tools/tools/cpp/cc_configure.bzl:228:80: Implicit string concatenation is forbidden, use the + operator.
ERROR: .../external/bazel_tools/tools/cpp/cc_configure.bzl:229:11: syntax error at '" variable"': expected ,
ERROR: com.google.devtools.build.lib.packages.BuildFileContainsErrorsException: error loading package 'external': Extension 'tools/cpp/cc_configure.bzl' has errors.

Second hurdle (another trivial fix, the script is call osx_cc_wrapper.sh.tpl, not gcc):

ERROR: .../external/bazel_tools/tools/cpp/cc_configure.bzl:254:3: no such package '@local_config_cc//': Traceback (most recent call last):
File ".../external/bazel_tools/tools/cpp/cc_configure.bzl", line 237
_tpl(ctx, "osx_gcc_wrapper.sh", {"%{cc}":...)})
File ".../external/bazel_tools/tools/cpp/cc_configure.bzl", line 220, in _tpl
ctx.template(tpl, Label("@bazel_tools//tools/cp...), ...)
Not a file: .../external/bazel_tools/tools/cpp/osx_gcc_wrapper.sh.tpl and referenced by '//external:cc_toolchain'.
ERROR: Loading failed; build aborted.

Final hurdles:
$ bazel build //kythe/cxx/common:all
Extracting Bazel installation...
Sending SIGTERM to previous Bazel server (pid=17068)... done.
..........
ERROR: No toolchain found for cpu 'k8'. Valid cpus are: [
local,
armeabi-v7a,
].

After removing my local .bazelrc with contents:

$ cat ~/.bazelrc
build --package_path %workspace%:/home/shahms/src/bazel/base_workspace
fetch --package_path %workspace%:/home/shahms/src/bazel/base_workspace
query --package_path %workspace%:/home/shahms/src/bazel/base_workspace

$ bazel build --cpu=local //kythe/cxx/common:all
ERROR: Loading of target '//tools/cpp:toolchain' failed; build aborted: no such package 'tools/cpp': BUILD file not found on package path.
ERROR: Loading failed; build aborted.

And:

$ bazel build //kythe/cxx/common:all
ERROR: /home/shahms/src/kythe/tools/defaults/BUILD:11:1: no such package 'tools/cpp': BUILD file not found on package path and referenced by '//tools/defaults:crosstool'.
ERROR: Loading of target '//tools/cpp:toolchain' failed; build aborted: no such package 'tools/cpp': BUILD file not found on package path.
ERROR: Loading failed; build aborted.

(It's worth pointing out that the BUILD-path referenced above doesn't exist)

Very happy to see progress on this front, though :-)

--Shahms

[sha...@google.com]

Strangely, if I leave our extant tools/cpp/CROSSTOOL, et al in place, the error I get is:

ERROR: cc_toolchain_suite '@local_config_cc//:toolchain' does not contain a toolchain for CPU 'armeabi-v7a'.

--Shahms

[Damien Martin-guillerez]

Oops there was some problem with merging the change. Thanks for reporting!

[sha...@google.com]

After re-syncing this morning and removing our `tools/bazel.rc` file, the autoconfig does appear to work! As it mandates GCC (and Kythe requires Clang) I'll have to use it as a base for our own autoconfigure rules, but it should go a long way towards removing our setup script :-)

--Shahms

[Damien Martin-guillerez]

\o/

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/bac3c47f-07e0-4455-bac3-3e83b2c97c0a%40googlegroups.com.

[os...@stripe.com]

Is there any documentation yet (apologies if I missed something obvious) on what functions we can call in repo rules?

I'm thinking of trying to do a git repo rule that would support ssh as an example.

Thanks.

[Damien Martin-guillerez]

Hi Oscar,

Not yet (well the class themselves are documented: https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/bazel/repository/skylark/SkylarkRepositoryContext.java, but I need to come-up with the correct paging for documentation).

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/1327ea23-e2d2-4a75-b7d5-5090859fba0b%40googlegroups.com.

[David]

I'm trying to supply a libstdc++ in bazel to use in my build. I have a new_http_archive rule that pulls down a build of libstdc++, and a BUILD file for it which wraps the pre-built `.so`s. How can I use cc_configure to use my bazel-supplied version of libstdc++?

[László Csomor]

+Marcel Hlopko 

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/ceeb45b9-e800-4b24-82a4-1e6daa7c534c%40googlegroups.com.

[dapi...@uber.com]

The best I can currently do is hardcode an rpath to by libstdc++ like this:
build --linkopt -Wl,-rpath,$ORIGIN/../../../_solib_k8/_U@libstdcpp_Urepo_S_S_Clibstdcpp___Uexternal_Slibstdcpp_Urepo_Srpath_Susr_Slib_Sx86_U64-linux-gnu
I can do that in the CROSSTOOL or in .bazelrc, but I need a way to add it procedurally and not hardcore the path. Or a better way than this altogether :)

[David]

Actually, if you want to support supplying a C++ standard library from `new_http_archive`, you'd need a special rule like `cc_standard_library` which adds an implicit dependency on all other `cc_` targets. As far as I can tell, supplying your own standard library as a `new_http_archive` is just not supported by bazel right now.

[Nikolay Semenov]

Hi All, I'm facing similar issues and wondering if anything has changed since the last message on this thread.

I need to cross-compile my code and later run on old platform. I also need to run UT on the build machine (even older).

The project and some third-party libs (like boost) which also expect newer libstdc++.

So far I figured out a way to insert rpath to libstdc++ *before* automatically appended rpaths to boost libs (otherwise they can't resolve GLIBCXX_... symbols - linker load host's libstdc++) in the CROSTOOL file:

  feature {

    name: "stdlib"

    flag_set {

      action: "c++-link-executable"

      flag_group {

        flag: "-Wl,-rpath=$ORIGIN/../../../_solib_...."     ### have to repeat this to be able to run tests located in different subdirs 

        flag: "-Wl,-rpath=$ORIGIN/../../_solib_...."

        flag: "-l:libstdc++.so"

        flag: "-l:libstdc++.so.X"

        flag: "-l:libstdc++.so.X.0.YY"

      }

    }

  }

  feature {

    name: "fastbuild"

    implies: "stdlib"

  }

I'm looking for a better/portable solution for this and have a few questions:

 - is there a way to get absolute path to my libstdc++.so in CROSSTOOL via %package()% or something similar (also I'm having hard time figuring out the list of fields of CROSSTOOL where this macro(?) works);

 - is there a way to insert libstc++ into "deps" of all cc_test/cc_binary via some "feature" or do I have to create and use my own "cpp_test/cpp_binary" rules and append "@libstdc++" (I have to add this manually to force bazel to create symlinks to libstdc++.so*);

 - is there a way to create/register global rule similar to ".stripped" ? I need to make binaries/libraries produced in the build portable - basically to "chrpath" to some predefined path where libstc++.so/boost*.so are going to be installed on the target machine.

Thanks for your help/recommendations/links-to-examples in advance.

[Nikolay Semenov]

Is this not the right channel to ask?

[László Csomor]

It is, but maybe nobody who looked at the thread knows the answer.

--
László Csomor | Software Engineer | laszlo...@google.com

Google Germany GmbH | Erika-Mann-Str. 33 | 80636 München | Germany
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/eb6d06c7-5408-4931-ace7-426ff29e8d66%40googlegroups.com.

[Nikolay Semenov]

I got to a partial solution solving most of the issues: create new rules cpp_*, wrapping native.cc_* rules, and add dependency onto the selected toolchain's libstdc++ and generated config.h. This also helps declaring second ".rpath-fixed" library/binary.