bazel install via github clone

[Jeff Abrahamson]

TL;DR - I install bazel via git clone and compile.  It claims to build and pass its own tests, but using it with another google package (my only test) leads to failure.  Assuming the other package is not broken at HEAD, I assume I've erred in my bazel install.  But I'm unsure how to proceed.

The install instructions for bazel talk about some package management options and then mentions install via github.  Good, so I git clone, ./compile.sh, and all seems ok.

But now I want to make bazel usable without a dependency on my home directory.  Naively, I try

sudo cp output/bazel /usr/local/bin/bazel
sudo cp -r base_workspace/ /usr/local/bazel/

(with appropriate modifications to the generated .bazelrc) which causes the compilation of cctz (a google package using bazel) to fail.  Perhaps I should add tools?

sudo cp -r tools/ /usr/local/bazel/

The cctz build fails differently.

Perhaps I should try building out of the build directory.  So I unmodify my .bazelrc and then in cctz's base directory, I type

/home/jeff/src/test-stuff/bazel/output/bazel test ...

and I'm told about repositories that can't be fetched.

Note that I've killed the bazel server after each test.  (Aside: It would be nice to have an rc option that prevents the server from remaining persistent.  That's great in an environment like google where it is the base of all builds.  It's less useful where bazel will be invoked only rarely to build and install a needed package, then one switches back to some other build chain.)

I don't believe that cctz is at fault -- indeed, I'm assuming that it is not and that this is surely something about the way I've installed bazel.  (I'm aware that I've omitted some details, like log files, above.  This is because, in the absence of intuition of what might be important, I'm hesitant to make this mail even longer than it already is.)

I'm a bit uncertain what direction to take how to debug this.  Any pointers?

Jeff Abrahamson
+33 6 24 40 01 57

+44 7920 594 255    <-- only when I'm in the UK

http://www.ml-week.com/fr/

http://jeff.purple.com/
http://blog.purple.com/jeff/

[Damien Martin-guillerez]

+Lukács T. Berki Isn't that a consequence of the bazel_tools change?

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAM4Y7zyE13SzHcsVvS2nTO8DQqXxuMiTQDdjWDedA1oq-z5Z-g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Lukács T. Berki]

We have a command line option to prevent the Bazel server from being persistent: --batch. If you add "startup --batch" to your .bazelrc, it should work as you'd expect.

It's kind of hard to say anything definitive without seeing the error messages. Try removing the tools/ directory from the cctz tree so that Bazel uses the one from base_workspace.

(Yes, the tools directory is a terrible mess. I am had at work fixing it.)

--

Lukács T. Berki | Software Engineer | lbe...@google.com | 

Google Germany GmbH | Maximillianstr. 11-15 | 80539 München | Germany | Geschäftsführer: Matthew Scott Sucherman, Paul Terence Manicle | Registergericht und -nummer: Hamburg, HRB 86891

[Jeff Abrahamson]

I do this:

mkdir /tmp/google
cd /tmp/google
git clone https://github.com/bazelbuild/bazel.git
cd bazel
./compile.sh

cd ..
git clone https://github.com/google/cctz.git
cd cctz
rm -rf tools
/tmp/google/bazel/output/bazel test ...

This is what happens:

jeff@birdsong:~ $ mkdir /tmp/google
jeff@birdsong:~ $ cd /tmp/google
jeff@birdsong:google $ git clone https://github.com/bazelbuild/bazel.git
Cloning into 'bazel'...
remote: Counting objects: 54530, done.
remote: Compressing objects: 100% (828/828), done.
remote: Total 54530 (delta 391), reused 0 (delta 0), pack-reused 53536
Receiving objects: 100% (54530/54530), 89.51 MiB | 307.00 KiB/s, done.
Resolving deltas: 100% (34479/34479), done.
Checking connectivity... done.
jeff@birdsong:google $ cd bazel
╭╴ (master=)╶╮
╰ jeff@birdsong:bazel $ ./compile.sh
INFO: You can skip this first step by providing a path to the bazel binary as second argument:
INFO:    ./compile.sh build /path/to/bazel
🍃  Building Bazel from scratch............
🍃  Building Bazel tools....
🍃  Building Bazel with Bazel.
INFO: Found 1 target...
INFO: From Building src/tools/android/java/com/google/devtools/build/android/ziputils/libziputils_lib.jar (12 files):
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
INFO: From Building src/main/java/com/google/devtools/build/skyframe/libskyframe.jar (56 files):
Note: src/main/java/com/google/devtools/build/skyframe/ReverseDepsUtil.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
INFO: From Building src/main/java/com/google/devtools/build/lib/libpackages-internal.jar (90 files):
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
INFO: From Building src/main/java/com/google/devtools/build/lib/libanalysis-exec-rules-skyframe.jar (609 files):
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
INFO: From Building src/main/java/com/google/devtools/build/lib/libquery2.jar (51 files):
Note: src/main/java/com/google/devtools/build/lib/query2/BlazeTargetAccessor.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
INFO: From Building src/main/java/com/google/devtools/build/lib/libbazel-core.jar (126 files):
Note: src/main/java/com/google/devtools/build/lib/bazel/repository/MavenJarFunction.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
Target //src:bazel up-to-date:
  bazel-bin/src/bazel
INFO: Elapsed time: 49.590s, Critical Path: 48.44s

🍃  Setting up base workspace.
INFO: You already have a .bazelrc. Make sure it contains the  following package paths:

build --package_path %workspace%:/tmp/google/bazel/base_workspace
fetch --package_path %workspace%:/tmp/google/bazel/base_workspace
query --package_path %workspace%:/tmp/google/bazel/base_workspace



Build successful! Binary is here: /tmp/google/bazel/output/bazel
╭╴ (master=)╶╮
╰ jeff@birdsong:bazel $ cd ..
jeff@birdsong:google $ git clone https://github.com/google/cctz.git
Cloning into 'cctz'...
remote: Counting objects: 220, done.
remote: Total 220 (delta 0), reused 0 (delta 0), pack-reused 220
Receiving objects: 100% (220/220), 90.67 KiB | 0 bytes/s, done.
Resolving deltas: 100% (132/132), done.
Checking connectivity... done.
jeff@birdsong:google $ cd cctz
╭╴ (master=)╶╮
╰ jeff@birdsong:cctz $ rm -rf tools
╭╴ (master *=)╶╮
╰ jeff@birdsong:cctz $ /tmp/google/bazel/output/bazel test ...
INFO: Reading 'startup' options from /home/jeff/.bazelrc: --batch
ERROR: /tmp/google/bazel/base_workspace/tools/build_rules/go/toolchain/BUILD:24:1: no such package '@golang-darwin-amd64//': error loading package 'external': The repository named '@golang-darwin-amd64' could not be resolved and referenced by '//tools/build_rules/go/toolchain:go_tool'.
ERROR: Loading failed; build aborted.
INFO: Elapsed time: 1.245s
ERROR: Couldn't start the build. Unable to run tests.
╭╴ (master *=)╶╮
╰ 1, jeff@birdsong:cctz $

My ~/.bazelrc (er, this should eventually be /etc/bazelrc, no?) is this:

build --package_path %workspace%:/tmp/google/bazel/base_workspace
fetch --package_path %workspace%:/tmp/google/bazel/base_workspace
query --package_path %workspace%:/tmp/google/bazel/base_workspace
startup --batch

Hmm, so I'm on linux and it's looking for a MacOS package?

Jeff Abrahamson
+33 6 24 40 01 57

+44 7920 594 255    <-- only when I'm in the UK

http://www.ml-week.com/fr/     prochaine edition, 2016 T1

http://jeff.purple.com/
http://blog.purple.com/jeff/

[Lukács T. Berki]

This is actually a bit of a mess caused by our migration to tools embedded into Blaze (as opposed to having a base workspace installed). 

You can either remove ~/.bazelrc or call Bazel as "bazel --bazelrc=/dev/null test ...". Please sync to HEAD in the Bazel repository -- there was a bit of another snafu which just got fixed in 2fe7decc75f3... . Sorry about the confusion :(

[Jeff Abrahamson]

HEAD as seen here is 7d1928e802c92726530fcc0756b9cb9fe76b7a3f.

Jeff Abrahamson
+33 6 24 40 01 57

+44 7920 594 255    <-- only when I'm in the UK

http://www.ml-week.com/fr/     prochaine edition, 2016 T1

http://jeff.purple.com/
http://blog.purple.com/jeff/

[Lukács T. Berki]

Yep, the fix should be about eight commits back:

https://github.com/bazelbuild/bazel/commit/2fe7decc75f331a36449cb9bab1b42b56bb40832

[Jeff Abrahamson]

Bazel rebuilds successfully at HEAD = 7d1928e802c92726530fcc0756b9cb.

...
INFO: Elapsed time: 31.698s, Critical Path: 30.84s

🍃  Setting up base workspace.
INFO: You already have a .bazelrc. Make sure it contains the  following package paths:

build --package_path %workspace%:/tmp/google/bazel/base_workspace
fetch --package_path %workspace%:/tmp/google/bazel/base_workspace
query --package_path %workspace%:/tmp/google/bazel/base_workspace



Build successful! Binary is here: /tmp/google/bazel/output/bazel

(Colour added.)  Note that, as per request, I removed everything but the batch line from ~/.bazelrc:

startup --batch

Three runs of bazel in cctz.  First, forgot to re-add tools directory, then realized there was a bazel server running.  Then fails again.

╭╴ (master *=)╶╮
╰ jeff@birdsong:cctz $ /tmp/google/bazel/output/bazel test ...
INFO: Reading 'startup' options from /home/jeff/.bazelrc: --batch

Extracting Bazel installation...
INFO: Found 9 targets and 2 test targets...
INFO: From Compiling external/gtest/googletest/src/gtest-death-test.cc:
external/gtest/googletest/src/gtest-death-test.cc:76:36: fatal error: src/gtest-internal-inl.h: No such file or directory
compilation terminated.
ERROR: /home/jeff/.cache/bazel/_bazel_jeff/95e8c9d3a6bb84d9bfbc625fe12d8152/external/gtest/BUILD:2:1: C++ compilation of rule '@gtest//:gtest' failed: gcc failed: error executing command /usr/bin/gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer '-std=c++0x' -iquote . -iquote ... (remaining 33 argument(s) skipped).
INFO: Elapsed time: 5.212s, Critical Path: 2.52s

Executed 0 out of 2 tests: 2 fail to build.
╭╴ (master *=)╶╮
╰ 1, jeff@birdsong:cctz $ git reset --hard HEAD
HEAD is now at 715a222 Changed .gitignore to use glob to catch all bazel files
╭╴ (master=)╶╮
╰ jeff@birdsong:cctz $ git pull
Already up-to-date.
╭╴ (master=)╶╮
╰ jeff@birdsong:cctz $ /tmp/google/bazel/output/bazel test ...

INFO: Reading 'startup' options from /home/jeff/.bazelrc: --batch

Unhandled exception thrown during build; message: Unrecoverable error while evaluating node 'FILE_STATE:[/]/[home/jeff/.cache/bazel/_bazel_jeff/95e8c9d3a6bb84d9bfbc625fe12d8152/external]' (requested by nodes 'FILE:[/tmp/google/cctz]/[bazel-cctz/external]')
INFO: Elapsed time: 0.328s
java.lang.RuntimeException: Unrecoverable error while evaluating node 'FILE_STATE:[/]/[home/jeff/.cache/bazel/_bazel_jeff/95e8c9d3a6bb84d9bfbc625fe12d8152/external]' (requested by nodes 'FILE:[/tmp/google/cctz]/[bazel-cctz/external]')
    at com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run(ParallelEvaluator.java:974)
    at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$2.run(AbstractQueueVisitor.java:496)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
    at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:210)
    at com.google.devtools.build.lib.skyframe.ExternalFilesHelper.maybeHandleExternalFile(ExternalFilesHelper.java:136)
    at com.google.devtools.build.lib.skyframe.FileStateFunction.compute(FileStateFunction.java:47)
    at com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run(ParallelEvaluator.java:938)
    ... 4 more
╭╴ (master=)╶╮
╰ 37, jeff@birdsong:cctz $ # Kill running bazel server (!)
╭╴ (master=)╶╮
╰ jeff@birdsong:cctz $ /tmp/google/bazel/output/bazel test ...

INFO: Reading 'startup' options from /home/jeff/.bazelrc: --batch

ERROR: /tmp/google/cctz/bazel-cctz/external/bazel_tools/src/main/protobuf/BUILD:1: Extension file not found: 'tools/build_rules/genproto.bzl'.
ERROR: package contains errors: bazel-cctz/external/bazel_tools/src/main/protobuf.
ERROR: error loading package 'bazel-cctz/external/bazel_tools/src/main/protobuf': Extension file not found: 'tools/build_rules/genproto.bzl'.
INFO: Elapsed time: 0.572s

ERROR: Couldn't start the build. Unable to run tests.

╭╴ (master=)╶╮
╰ 1, jeff@birdsong:cctz $

 

Jeff Abrahamson
+33 6 24 40 01 57

+44 7920 594 255    <-- only when I'm in the UK

http://www.ml-week.com/fr/     prochaine edition, 2016 T1

http://jeff.purple.com/
http://blog.purple.com/jeff/

[Lukács T. Berki]

Adding Kristina and Nathan for the exception. They have been working with ExternalFilesHelper so it must surely be interesting to them.

Don't delete the "tools" directory in the cctz repository. Bazel still needs it for the Skylark files, as you discovered. I sort of remember getting a compile error with cctz the vaguely looked to be my fault. Let me just take a closer look...

[Lukács T. Berki]

Actually, that NullPointerException seems to be reproducible. That's pretty bad. Let me take an _even_ closer look :)

[Lukács T. Berki]

Filed https://github.com/bazelbuild/bazel/issues/594 . For now, "bazel test //test/..." is a workaround.