Temporarily disable GerritForge CI?

[Dave Borowitz]

It looks like GerritForge CI has been failing for a few days due to, I dunno, a buck caching bug or something. At this point all its -1s are kind of counterproductive.

Luca, it would be great if you could look at it at some point, but I don't want to put too much pressure on you for this volunteer work you're doing for us :)

In the meantime, what could we do to reduce the noise? Maybe temporarily revoke the user's permission to set Verified?

[Michael Zhou]

Yes I think it's reasonable to disable it until we find a solution.

Quoting Luca from https://gerrit-review.googlesource.com/#/c/76199/

It isn't the first time that a BUCK change would need to manually emptying out all the ~/.gerritcodereview content. I wonder if I should just do it automatically by checking the git log of a commit.

So that might be it, although IMHO it's really a Buck issue.

[lucamilanesio]

Rather than disabling the CI altogether (a bit drastic !) I would opt for merging a workaround (see [1]).

[1] https://gerrit-review.googlesource.com/#/c/76673/

[lucamilanesio]

Looking at [2] the workaround seems to work.

[2] https://gerrit-ci.gerritforge.com/job/Gerrit-verifier-default/6908/console

[lucamilanesio]

GerritForge CI seems stable again :-)

As Dave predicted, the overhead of removing the Buck cache is not that high, only around 4 minutes.

We can leave it as-is at the moment, waiting to discover why a fully populated Buck cache causes this JGit compilation error.

Luca.

[Michael Zhou]

Thanks Luca! In the meantime my change to update Buck (https://gerrit-review.googlesource.com/#/c/76656/) also fails on the CI only. The CI doesn't use buckd / watchman which is different from my local environment.

[Luca Milanesio]

True, we don't use buckd / watchman on CI ... does this mean that after your change, the use of them will become mandatory?

Luca.

[lucamilanesio]

Failure is quite strange:

java.lang.IllegalStateException: Raw data claims to come from [lib/jgit/org.eclipse.jgit], but we tried rooting it at [org.eclipse.jgit].
	at com.facebook.buck.parser.ParsePipeline.parseBuildTargetFromRawRule(ParsePipeline.java:556)

Needs some digging into Buck to see what has introduced the problem ...

Luca.

[David Ostrovsky]

Am Dienstag, 19. April 2016 10:44:27 UTC+2 schrieb lucamilanesio:

Failure is quite strange:

java.lang.IllegalStateException: Raw data claims to come from [lib/jgit/org.eclipse.jgit], but we tried rooting it at [org.eclipse.jgit].
	at com.facebook.buck.parser.ParsePipeline.parseBuildTargetFromRawRule(ParsePipeline.java:556)

Needs some digging into Buck to see what has introduced the problem ...

May be related to cross-cell JGit support we've added recently.

[Michael Zhou]

Well it shouldn't be mandatory if the build without buckd passes :)

[lucamilanesio]

Are you referring to [3] ?

[3] https://gerrit-review.googlesource.com/#/c/73000/

[lucamilanesio]

Just tried now locally (MacOS X) and it fails *exactly* in the same way as in CI:

$ buck test

Not using buckd because watchman isn't installed.

[-] PROCESSING BUCK FILES...FINISHED 0.3s [100%]

[+] DOWNLOADING... (0.00 B/S, TOTAL: 0.00 B, 0 Artifacts)

[+] BUILDING...5.6s

[2016-04-19 10:23:32.010][error][command:daba45d0-2865-4995-b993-1bcf3e2e1861][tid:01][com.facebook.buck.cli.Main] Uncaught exception at top level

java.lang.IllegalStateException: Raw data claims to come from [lib/jgit/org.eclipse.jgit], but we tried rooting it at [org.eclipse.jgit].

at com.facebook.buck.parser.ParsePipeline.parseBuildTargetFromRawRule(ParsePipeline.java:556)

Please note that isn't the build itself that fails but the *test*.

@Michael have you tried to run it locally without buckd and watchman? (cleaning up your ~/.gerritcodereview and starting with a fresh clone)

On Tuesday, April 19, 2016 at 9:46:46 AM UTC+1, David Ostrovsky wrote:

[lucamilanesio]

... and it seems to break *ONLY* with your Buck update. It doesn't fail for other changes.

Possibly a Buck regression introduced with the performance improvement?

Luca.

[lucamilanesio]

We can discuss this issue but it is not related to the "Temporarily Disable GerritForge CI": does not make sense to disable the CI because a single change has (indirectly) introduced a real regression IMHO :-)

The CI is the thermometer of the fever and *is not* the cause of the illness :-) Not checking your temperature if you are ill isn't the solution to the problem.

Luca.

[David Ostrovsky]

On Tuesday, April 19, 2016 at 11:43:27 AM UTC+2, lucamilanesio wrote:

We can discuss this issue but it is not related to the "Temporarily Disable GerritForge CI": does not make sense to disable the CI because a single change has (indirectly) introduced a real regression IMHO :-)

The CI is the thermometer of the fever and *is not* the cause of the illness :-) Not checking your temperature if you are ill isn't the solution to the problem.

I can reproduce the breakage here [1] and you are right, has

nothing to do with this thread: non stable CI.  I think Michael

missed the fact that the breakage only caused by: `buck test`.

[1] http://paste.openstack.org/show/494597

[David Ostrovsky]

On Tuesday, April 19, 2016 at 2:00:03 PM UTC+2, David Ostrovsky wrote:

On Tuesday, April 19, 2016 at 11:43:27 AM UTC+2, lucamilanesio wrote:

We can discuss this issue but it is not related to the "Temporarily Disable GerritForge CI": does not make sense to disable the CI because a single change has (indirectly) introduced a real regression IMHO :-)

The CI is the thermometer of the fever and *is not* the cause of the illness :-) Not checking your temperature if you are ill isn't the solution to the problem.

I can reproduce the breakage here [1] and you are right, has

nothing to do with this thread: non stable CI.

The commit that broke `buck test` is: [1]. I filed upstream issue: [2].

* [1] https://github.com/facebook/buck/commit/9f45799db1cd25b02dadc7cc982f2264c1c13153

* [2] https://github.com/facebook/buck/issues/717

[lucamilanesio]

Thanks David for deep diving into the real cause: so actually the JGit cell separation *is* the original cause that appeared only because of this "extra check" they introduced.

It would be interesting to see their comments to the issue you've just raised :-)

In the meantime ... let's close this topic of "CI being unstable" :-) ... at least for now ;-)

[David Ostrovsky]

On Tuesday, April 19, 2016 at 3:05:37 PM UTC+2, lucamilanesio wrote:

Thanks David for deep diving into the real cause: so actually the JGit cell separation *is* the original cause that appeared only because of this "extra check" they introduced.

It would be interesting to see their comments to the issue you've just raised :-)

In the meantime ... let's close this topic of "CI being unstable" :-) ... at least for now ;-)

lol