In jenkins test harness (only) node() {} commands crash the executor

[Michael Carter]

Last of the thorns in my side before I release this code internally today.   All my tests via pipeline suddenly stopped working a week ago.   One thing I did try was having a few minute pause at the beginning of this command so I could log in via the http console.   Then tired creating the job via the UI and same results.  As soon as the node part start it crashes the executor and all jobs just spin.  

Any echo commands and such on the job run fine.   As long as there is no node() {} command

Script running
node() {
  echo "hello"
}


  44.247 [jobname-enableTest #1] [Pipeline] Start of Pipeline
  47.466 [jobname-enableTest #1] [Pipeline] node
  49.717 [id=47] SEVERE hudson.model.Executor#finish1: Executor threw an exception
java.lang.NoSuchMethodError: com.google.common.util.concurrent.MoreExecutors.sameThreadExecutor()Lcom/google/common/util/concurrent/ListeningExecutorService;
 at org.jenkinsci.plugins.workflow.support.concurrent.Futures.addCallback(Futures.java:90)
 at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.runInCpsVmThread(CpsFlowExecution.java:900)
 at org.jenkinsci.plugins.workflow.cps.CpsBodyInvoker.start(CpsBodyInvoker.java:153)
 at org.jenkinsci.plugins.workflow.cps.CpsBodyInvoker.start(CpsBodyInvoker.java:56)
 at org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask$PlaceholderExecutable.run(ExecutorStepExecution.java:844)
 at hudson.model.ResourceController.execute(ResourceController.java:97)
 at hudson.model.Executor.run(Executor.java:428)

Not sure what I did to cause the problem but I'm sure this is just some library I'm missing as it seems to be the root of all my issues so far.   Just not sure where to look or where to start.

[Jesse Glick]

On Wed, Jul 15, 2020 at 7:42 AM Michael Carter
<mikeyca...@gmail.com> wrote:
> java.lang.NoSuchMethodError: com.google.common.util.concurrent.MoreExecutors.sameThreadExecutor()Lcom/google/common/util/concurrent/ListeningExecutorService;
> at org.jenkinsci.plugins.workflow.support.concurrent.Futures.addCallback(Futures.java:90)

> …

>
> Not sure what I did to cause the problem

Somehow pulled in an incompatible version of `guava`. Jenkins bundles
11.x, and without a lot of advanced acrobatics¹ you cannot override
this. If your plugin bundles some third-party libraries, they may be
depending on a newer version of Guava and Maven picks that version for
tests, breaking everything. Check

mvn dependency:tree

and use an `<exclusion>` as necessary. Unless the third-party library
actually _requires_ newer Guava features, in which case you have
problems.


Also, did your tests stop working with no code changes? If `master`
was blue/green until now, check whether your `Jenkinsfile` is using,
say,

buildPlugin(configurations: recommendedConfigurations())

which is nondeterministic. If tests broke inside a PR (I would
recommend a PR for all nontrivial changes, even if you are a sole
maintainer, just for trackability and CI), link to that from any post
to the mailing list.


¹Read: https://www.jenkins.io/doc/developer/plugin-development/dependencies-and-class-loading/#pluginfirstclassloader-and-its-discontents

[Michael Carter]

Tried excluding the 27.1-jre version out of one of the apis I'm using so it can pick up version 27.1-jre of guava from the war.   Didn't work.

Tried adding directly:

        <dependency>
          <groupId>com.google.guava</groupId>
          <artifactId>guava</artifactId>
          <version>27.1-jre</version>
          <scope>test</scope>
        </dependency>

Didn't work either.   I think the thread is crashing.  Any way to turn on more debugging?   Doing -X on maven doesn't put the jenkins harness into debug mode or anything.

As for did the test stop working without code change.   Not to that test but I as doing a lot of work in other areas before re-testing pipeline so probably something I did... problem is backtracking what I added from it last working is near impossible at this point.

[Jesse Glick]

On Wed, Jul 15, 2020 at 9:17 AM Michael Carter
<mikeyca...@gmail.com> wrote:
> so it can pick up version 27.1-jre of guava from the war.

From `jenkins.war`? As mentioned, that bundles 11, not 27. Keep
editing `pom.xml` until

$ mvn dependency:tree | fgrep guava
[INFO] | +- com.google.guava:guava:jar:11.0.1:provided

> problem is backtracking what I added from it last working is near impossible at this point.

git bisect

[Michael Carter]

+- org.jenkins-ci.main:jenkins-core:jar:2.235.1:provided
|  +- com.google.guava:guava:jar:27.1-jre:provided

Guess that's my problem... somehow when I upgraded the pom.xml to pull the 2.235.1/2.222.4 it must have broke then.

At least this gives me a place to start looking.

[Michael Carter]

... and that did the trick:

        <dependency>
          <groupId>com.google.guava</groupId>
          <artifactId>guava</artifactId>

          <version>11.0.1</version>
          <scope>test</scope>
        </dependency>

Output:

  62.511 [jobname-PipelineManualFailTest #1] [Pipeline] node
  62.763 [jobname-PipelineManualFailTest #1] Running on Jenkins in ....workspace/jobname
  62.964 [jobname-PipelineManualFailTest #1] [Pipeline] {
  63.568 [jobname-PipelineManualFailTest #1] [Pipeline] stage
  63.668 [jobname-PipelineManualFailTest #1] [Pipeline] { (Example Build)
  64.121 [jobname-PipelineManualFailTest #1] [Pipeline] sh

Guess that's what had me all turned around when I saw the guava jar coming from the jenkins-core I just assumed it was the correct one I should be using.

Thanks for all the help everyone!

[Jesse Glick]

On Wed, Jul 15, 2020 at 2:09 PM Michael Carter
<mikeyca...@gmail.com> wrote:
> ... and that did the trick:
>
> <dependency>
> <groupId>com.google.guava</groupId>
> <artifactId>guava</artifactId>
> <version>11.0.1</version>
> <scope>test</scope>
> </dependency>

Not recommended. Better to find out which other dependency was pulling
it a newer version of Guava and add an `<exclusion>` to suppress that.

[Michael Carter]

jenkins-core:jar appears to be the one... 

[Michael Carter]

So my ignorance might be showing but I think I tracked down where the 27.1-jre is coming from as if I do an exclude on jenkins-core:jar and pipeline-model-definition it goes away... well all versions of gauva go away.

I'm guessing it's part of this?

      <jenkins.version>2.235.1</jenkins.version>

      <jenkins.bom>bom-2.235.x</jenkins.bom>

          <dependency>
            <groupId>io.jenkins.tools.bom</groupId>
            <artifactId>${jenkins.bom}</artifactId>
            <version>${jenkins.bomver}</version>
            <scope>import</scope>
            <type>pom</type>
          </dependency>

[Jesse Glick]

On Wed, Jul 15, 2020 at 4:25 PM Michael Carter
<mikeyca...@gmail.com> wrote:
> if I do an exclude on jenkins-core:jar and pipeline-model-definition it goes away... well all versions of gauva go away.

Try excluding only from `pipeline-model-definition`.

Without seeing your POM it is hard to help here, this is all just
general advice.