git clone (through git_repository) works on Ubuntu but fails on OSX

[Svend Johannsen]

Hi again :-)

I'm on Bazel commit 96f9ea0e0c7a7ed2ab6f48aed2e701b33a24cada from August 5th.

I have an issue where building a project with a git_repository dependency succeeds on Ubuntu but fails on OSX. It is perfectly possible that I somehow misconfigured git and this is not an issue with Bazel. (But I think it's unlikely.)

git clone https://sic-phabricator.us.oracle.com/diffusion/BD/buildmedep.git

cd buildmedep

bazel build java/com/example:level-two

Succeeds on Ubuntu. But fails on OSX with the following error:

ERROR: /Users/skjohann/workspace/ws0722-bad/badger/mac_wrk_spc/buildmedep/java/com/example/BUILD:8:1: no such package '@level3//java/com/example': https://sic-phabricator.us.oracle.com/diffusion/BDD/buildmedepdep.git: cannot open git-upload-pack and referenced by '//buildmedep/java/com/example:level-two'.

The error just tells me that the dependency is missing.

My ~/.gitconfig is identical between Ubuntu and OSX. I don't even know if Bazel relies on it.

If I look at the cached files on Ubuntu (where it works just fine) the git config for the dependency is:

$ cat /home/skjohann/.cache/bazel/_bazel_skjohann/280ef1f038df3cf197c91c3edf3b59bc/external/level3/.git/config

[core]

        repositoryformatversion = 0

        filemode = true

        logallrefupdates = true

[remote "origin"]

        url = https://sic-phabricator.us.oracle.com/diffusion/BDD/buildmedepdep.git

        fetch = +refs/heads/*:refs/remotes/origin/*

[branch "master"]

        remote = origin

        merge = refs/heads/master

But if I look at the cached files on OSX (where it fails to build) the git config for the dependency is missing the branch:

$ cat /private/var/tmp/_bazel_skjohann/2b8678a0db94673b85f3f88e3acbacc0/external/level3/.git/config

[core]

        repositoryformatversion = 0

        filemode = true

        logallrefupdates = true

        precomposeunicode = true

[remote "origin"]

        url = https://sic-phabricator.us.oracle.com/diffusion/BDD/buildmedepdep.git

        fetch = +refs/heads/*:refs/remotes/origin/*

The missing branch is probably an issue. The precomposeunicode flag is also set on OSX, but that seems reasonable.

I've attached a copy of my source code. "buildmedep" and "buildmedepdep" are git repositories. The WORKSPACE in "buildmedep" is:

git_repository(

    name = "level3",

    commit = "7b358927c4d3b344a257dbe69733dc8192672be3",

    remote = "https://sic-phabricator.us.oracle.com/diffusion/BDD/buildmedepdep.git",

)

And the BUILD file in "buildmedep" is:

java_library(

    name = "level-two",

    srcs = glob(["*.java"]),

    deps = ["@level3//java/com/example:level-three"],

    visibility = ["//visibility:public"],

)

[Brian Silverman]

I'm pretty sure the missing branch is a symptom, not the root cause. The git code in Bazel first creates the repository and then does a fetch, which means if the fetch fails you end up with an empty git repository without any remote tracking branches.

The actual jgit error buried in your Bazel output is "cannot open git-upload-pack". Looking around online, it seems like a common cause for that error is jgit not picking up proxy settings which git from the command line does. Considering that I can't get to that repository, that seems like a likely candidate. Do you know if you're relying on a proxy to access that repository?

If that doesn't point you at the issue, do you have this problem with all git repositories or just this one? Can you try cloning the problematic repository onto your OSX machine outside of Bazel and then giving Bazel a file:// remote?

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/a810a018-a7f0-4b96-bf09-b29b551a346f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Svend Johannsen]

Hi Brian,

Thanks for the reply. There shouldn't be a need for a proxy to access this git repository.

The git repository is hosted on a machine with self signed certificate. This may be an issue (although I'm not sure why it would only be an issue on OSX.)

Do you know if bazel relies on git being installed on the system, or does it use a private copy of git?

[Brian Silverman]

Bazel uses jgit, which is completely independent of the command line tools. The certificate does sound like a potential problem. How do you get the command line tools tools to accept it?

It would not surprise me if your java on Ubuntu has different certificate configuration or something than it does on OSX.

Actually knowing what error jgit is reporting would be helpful. I'm not sure if the information is buried somewhere normally, but compiling with this (temporary) patch will get it out on the console. What output do you get from a Bazel build with this applied?
diff --git a/src/main/java/com/google/devtools/build/lib/bazel/repository/GitCloneFunction.java b/src/main/java/com/google/devtools/build/lib/bazel/repository/GitCloneFunction.java
index 2c8e50a..8dd89d2 100644
--- a/src/main/java/com/google/devtools/build/lib/bazel/repository/GitCloneFunction.java
+++ b/src/main/java/com/google/devtools/build/lib/bazel/repository/GitCloneFunction.java
@@ -95,8 +95,7 @@ public class GitCloneFunction implements SkyFunction {
           new IOException("Invalid branch, tag, or commit: " + e.getMessage()),
           Transience.PERSISTENT);
     } catch (GitAPIException e) {
-      throw new RepositoryFunctionException(
-          new IOException(e.getMessage()), Transience.TRANSIENT);
+      throw new RuntimeException(e);
     } finally {
       if (git != null) {
         git.close();

In case the inline patch gets mangled, it's also at http://pastebin.com/LHAceY6F.

[Svend Johannsen]

Nice thanks for the patch, this was very helpful.

You were right about the proxy.

java.lang.RuntimeException: Unrecoverable error while evaluating node 'GIT_CLONE:https://sic-phabricator.us.oracle.com/diffusion/BDD/buildmedepdep.git -> /private/var/tmp/_bazel_skjohann/152112983e106425cb88daf25fc01b89/external/level3 (7b358927c4d3b344a257dbe69733dc8192672be3) submodules: false' (requested by nodes 'GIT_REPOSITORY:@level3')

............

Caused by: java.lang.RuntimeException: org.eclipse.jgit.api.errors.TransportException: https://sic-phabricator.us.oracle.com/diffusion/BDD/buildmedepdep.git: cannot open git-upload-pack

............

Caused by: org.eclipse.jgit.api.errors.TransportException: https://sic-phabricator.us.oracle.com/diffusion/BDD/buildmedepdep.git: cannot open git-upload-pack

............

Caused by: org.eclipse.jgit.errors.TransportException: https://sic-phabricator.us.oracle.com/diffusion/BDD/buildmedepdep.git: cannot open git-upload-pack

............

Caused by: java.net.UnknownHostException: http://www-proxy.us.oracle.com

............

It turns out jgit is reading the proxy settings from System Preferences -> Network -> Advanced -> Proxies. Once I added the same exceptions I have in .bash_profile (where the git cli reads its settings from) - everything works as expected.

[Lukács T. Berki]

Sorry for the delay -- I was OOO for a long while, then it took a while until I reached this e-mail.

This is a bug in Bazel related to the fact that //external: labels are special-cased. Bazel, in its youthful innocence, tries to parse things like "//external:" as a repository-local label, then it doesn't realize that said labels refer to a bound label. I have a change to fix this and it should hit the Github tree in a day or two (depending on how the code review goes)

--

You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/779ea487-a162-4904-aa3f-71fcaac8bdea%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Lukács T. Berki | Software Engineer | lbe...@google.com | 

Google Germany GmbH | Dienerstrasse 12 | 80331 München | Germany | Geschäftsführer: Graham Law, Christine Elizabeth Flores | Registergericht und -nummer: Hamburg, HRB 86891

[Svend Johannsen]

Nice, thanks for fixing this. I'll sync up and try it out.

---

From: lbe...@google.com
Date: Mon, 24 Aug 2015 14:16:39 +0200
Subject: Re: git clone (through git_repository) works on Ubuntu but fails on OSX
To: svend.j...@gmail.com
CC: bazel-...@googlegroups.com; kcho...@google.com

[Lukács T. Berki]

Sure :) The commit fixing this is

https://github.com/google/bazel/commit/ed5955c5a16996a3b630d7f95206d67d0d0c6380