Jenkins 2.204.3 LTS RC testing started

[Oliver Gondža]

Hello everyone,

Latest LTS RC was made public and it is ready to be tested. Final
release is scheduled for 2020-02-26.

Please, report your findings in this thread.

Download bits from
http://mirrors.jenkins-ci.org/war-stable-rc/2.204.3/jenkins.war

Thanks!
--
oliver

[Mark Waite]

On Friday, February 14, 2020 at 5:05:22 AM UTC-7, ogondza wrote:

Hello everyone,

Latest LTS RC was made public and it is ready to be tested. Final
release is scheduled for 2020-02-26.

Please, report your findings in this thread.

I've encountered a problem on my Jenkins test instances that are running the 2.204.3 release candidate inside Docker.  Some of the folders fail to open when I click them and I receive the following stack track in an "Oops" page on one instance:

org.apache.commons.jelly.JellyTagException: jar:file:/var/jenkins_home/war/WEB-INF/lib/jenkins-core-2.204.3-SNAPSHOT.jar!/hudson/model/View/index.jelly:42:43: <st:include> org.apache.commons.jelly.JellyTagException: jar:file:/var/jenkins_home/war/WEB-INF/lib/jenkins-core-2.204.3-SNAPSHOT.jar!/lib/hudson/projectView.jelly:84:48: <j:forEach> java.nio.CharBuffer.rewind()Ljava/nio/CharBuffer;
	at org.apache.commons.jelly.impl.TagScript.handleException(TagScript.java:726)
	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:281)

In the other test instance, the message appears in the console log and does not appear in the Jenkins Oops page.  In the console log, it reports:

2020-02-15 16:37:04.540+0000 [id=164374]        INFO    j.b.MultiBranchProject$BranchIndexing#run: Bugs-Pipeline-Checks/jenkins-bugs-multibranch-pipeline-bitbucket #20200215.093700 branch indexing action completed: SUCCESS in 3.6 sec

2020-02-15 16:39:26.329+0000 [id=39]    SEVERE  hudson.triggers.SafeTimerTask#run: Timer task com.cloudbees.jenkins.Cleaner@7ed56677 failed

java.lang.NoSuchMethodError: java.nio.CharBuffer.rewind()Ljava/nio/CharBuffer;

        at hudson.Util.rawEncode(Util.java:886)

        at hudson.model.AbstractItem.getShortUrl(AbstractItem.java:576)

        at hudson.model.AbstractItem.getUrl(AbstractItem.java:537)

I don't know why there is a difference in behavior.  I don't know if the issue is related to something in my local environment, the Docker image definition that I'm using, or something completely different.

The folders which fail to open are different in the two instances, but the stack traces seem to consistently be associated with CharBuffer.rewind().

Later in the stack trace, it reports:

Caused by: java.lang.NoSuchMethodError: java.nio.CharBuffer.rewind()Ljava/nio/CharBuffer;
	at hudson.Util.rawEncode(Util.java:886)

Duplicating the problem:

If others would like to duplicate the problem, they can try the following steps:

1. Install git large file support on your Linux computer (or download from git-lfs.github.com)
2. Iniitalize git lfs with
   $ git lfs install
   Git LFS initialized.
3. Clone my docker-lfs repository
   $ git clone https://github.com/MarkEWaite/docker-lfs
   Cloning into 'docker-lfs'...
   Resolving deltas: 100% (12717/12717), done.
   
4. Change to the docker-lfs directory
   $ cd docker-lfs
5. Checkout the lts-with-plugins-rc branch
   $ git checkout -b lts-with-plugins-rc -t origin/lts-with-plugins-rc
   Filtering content: 100% (190/190), 244.87 MiB | 5.36 MiB/s, done.
   Branch 'lts-with-plugins-rc' set up to track remote branch 'lts-with-plugins-rc' from 'origin'.
   Switched to a new branch 'lts-with-plugins-rc'
   
6. Build the docker image
   $ docker build -f Dockerfile -t markewaite/lts-rc:2.204.3 .
7. Run the docker image
   $ docker run --rm -i -e JENKINS_ADVERTISED_HOSTNAME=`hostname` -e START_QUIET=True -p 8080:8080 -t markewaite/lts-rc:2.204.3
8. Connect to the running image with a web browser
   $ python -m webbrowser http://$(hostname):8080/
9.  Open each of the folders at the root of that Jenkins server.  One of them will fail to an Oops screen (at least does on the 3 machines where I've tested) 

End of symptoms, switching to wild, unjustified speculation:

There is mention on the jetty project and on stackoverflow that code compiled with JDK 9 or later may fail in this way when running with Java 8.  I'm running Java 8 in both the failure cases.  References:

• https://github.com/eclipse/jetty.project/issues/3244
• https://stackoverflow.com/questions/48693695/java-nio-buffer-not-loading-clear-method-on-runtime
  

Was the Jenkins 2.204.3 release candidate compiled with Java 11?

Thanks!

Mark Waite

[Mark Waite]

On Saturday, February 15, 2020 at 10:29:06 AM UTC-7, Mark Waite wrote:

On Friday, February 14, 2020 at 5:05:22 AM UTC-7, ogondza wrote:

Hello everyone,

Latest LTS RC was made public and it is ready to be tested. Final
release is scheduled for 2020-02-26.

Please, report your findings in this thread.

I've encountered a problem on my Jenkins test instances that are running the 2.204.3 release candidate inside Docker.  Some of the folders fail to open when I click them and I receive the following stack track in an "Oops" page on one instance:

org.apache.commons.jelly.JellyTagException: jar:file:/var/jenkins_home/war/WEB-INF/lib/jenkins-core-2.204.3-SNAPSHOT.jar!/hudson/model/View/index.jelly:42:43: <st:include> org.apache.commons.jelly.JellyTagException: jar:file:/var/jenkins_home/war/WEB-INF/lib/jenkins-core-2.204.3-SNAPSHOT.jar!/lib/hudson/projectView.jelly:84:48: <j:forEach> java.nio.CharBuffer.rewind()Ljava/nio/CharBuffer;
	at org.apache.commons.jelly.impl.TagScript.handleException(TagScript.java:726)
	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:281)

In the other test instance, the message appears in the console log and does not appear in the Jenkins Oops page.  In the console log, it reports:

2020-02-15 16:37:04.540+0000 [id=164374]        INFO    j.b.MultiBranchProject$BranchIndexing#run: Bugs-Pipeline-Checks/jenkins-bugs-multibranch-pipeline-bitbucket #20200215.093700 branch indexing action completed: SUCCESS in 3.6 sec

2020-02-15 16:39:26.329+0000 [id=39]    SEVERE  hudson.triggers.SafeTimerTask#run: Timer task com.cloudbees.jenkins.Cleaner@7ed56677 failed

java.lang.NoSuchMethodError: java.nio.CharBuffer.rewind()Ljava/nio/CharBuffer;

        at hudson.Util.rawEncode(Util.java:886)

        at hudson.model.AbstractItem.getShortUrl(AbstractItem.java:576)

        at hudson.model.AbstractItem.getUrl(AbstractItem.java:537)

I don't know why there is a difference in behavior.  I don't know if the issue is related to something in my local environment, the Docker image definition that I'm using, or something completely different.

Same message appears when trying to evaluate reverse proxy setup for the reverse proxy administrative monitor.  It reports:

2020-02-15 17:34:13.223+0000 [id=144]   WARNING o.e.j.s.h.ContextHandler$Context#log: Error while serving http://debian9-a.markwaite.net:8080/administrativeMonitor/hudson.diagnosis.ReverseProxySetupMonitor/test

java.lang.NoSuchMethodError: java.nio.CharBuffer.rewind()Ljava/nio/CharBuffer;

        at hudson.Util.rawEncode(Util.java:886)

        at hudson.diagnosis.ReverseProxySetupMonitor.doTest(ReverseProxySetupMonitor.java:69)

        at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)

        at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:396)

Caused: java.lang.reflect.InvocationTargetException

        at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:400)

[Mark Waite]

Message does not appear if I configure the Docker image to use JDK 11 instead of JDK 8.

Replace the branch name "lts-with-plugins-rc" with "lts-jdk11-with-plugins-rc" to see the JDK 11 based configuration.

Mark Waite

[Jesse Glick]

Could you file this in JIRA please with the smallest steps to reproduce you have?

Does this occur in `master`?

[Mark Waite]

On Sun, Feb 16, 2020 at 7:29 AM Jesse Glick <jgl...@cloudbees.com> wrote:

Could you file this in JIRA please with the smallest steps to reproduce you have?

Yes, I've copied those steps into a Jira ticket.  I'm not aware of any of the steps that can be removed, so I believe those are the minimum steps to duplicate the issue.

 

Does this occur in `master`?

I don't know if it occurs with 'master'.  It does not occur with 2.204.2.  It does not occur with weekly 2.220 as downloaded from the official release site.  I'll build the master branch and try it later today.

I'll also try to confirm my suspicion that the root of the issue is that the release candidate build was compiled with Java 11 without specifically mandating '-release 8'  The links included in the earlier message seem to indicate that setting source and target are necessary but not sufficient for Java 11 to generate Java 8 byte code.

I've switched my testing of the release candidate to use Java 11 and that testing has found no issues. The problem does not occur when the same release candidate bits are run with Java 11.  It only happens with Java 8.

 

Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr32uF7ZCagBNJoRoEwW6GTWSvYDgp%2BMVyASWKmAHLePRw%40mail.gmail.com.

[Mark Waite]

On Sunday, February 16, 2020 at 7:58:03 AM UTC-7, Mark Waite wrote:

On Sun, Feb 16, 2020 at 7:29 AM Jesse Glick wrote:

Could you file this in JIRA please with the smallest steps to reproduce you have?

Yes, I've copied those steps into a Jira ticket.  I'm not aware of any of the steps that can be removed, so I believe those are the minimum steps to duplicate the issue.

 

Does this occur in `master`?

I don't know if it occurs with 'master'.  It does not occur with 2.204.2.  It does not occur with weekly 2.220 as downloaded from the official release site.  I'll build the master branch and try it later today.

I'll also try to confirm my suspicion that the root of the issue is that the release candidate build was compiled with Java 11 without specifically mandating '-release 8'  The links included in the earlier message seem to indicate that setting source and target are necessary but not sufficient for Java 11 to generate Java 8 byte code.

I've compared the manifest from Jenkins 2.204.2 with the manifest from Jenkins 2.204.3 rc.  The key difference appears to be:

Jenkins 2.204.2: Build-Jdk-Spec: 1.8

Jenkins 2.204.3 rc: Build-Jdk-Spec: 11

I think we need a rebuild of the 2.204.3 release candidate using JDK 8.

Mark Waite

[ogondza]

Mark, thanks for reporting this. I have published an RC respin[1] with Java 8, until we identify the reason why is this causing problems.

Please make sure the ongoing testing is done with ca01d0bf523d6752f54277ca4c91feef0d60acc75c42d65f65bbefe09f9e8e69 (sha256). Thanks!

[1] http://mirrors.jenkins-ci.org/war-stable-rc/2.204.3/jenkins.war

--

oliver

[Mark Waite]

Thanks!  Any idea how long the propagation delay is through the mirrors?  I tried to download the URL you provided http://mirrors.jenkins-ci.org/war-stable-rc/2.204.3/jenkins.war and the sha245sum that I see is still the original, not the new one that you provided.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/005cb301-b149-44b4-bf3a-25dbd5ca35dd%40googlegroups.com.

[Oliver Gondža]

Propagating new file usually takes about an hour or two (never really
measured), so I can only guess propagating an update takes about the
same amount of time.

On 17/02/2020 13.23, Mark Waite wrote:
> Thanks!  Any idea how long the propagation delay is through the
> mirrors?  I tried to download the URL you provided
> http://mirrors.jenkins-ci.org/war-stable-rc/2.204.3/jenkins.war and the
> sha245sum that I see is still the original, not the new one that you
> provided.
>
> On Mon, Feb 17, 2020 at 5:09 AM ogondza <ogo...@gmail.com
> <mailto:ogo...@gmail.com>> wrote:
>
> Mark, thanks for reporting this. I have published an RC respin[1]
> with Java 8, until we identify the reason why is this causing problems.
>
> Please make sure the ongoing testing is done with
> ca01d0bf523d6752f54277ca4c91feef0d60acc75c42d65f65bbefe09f9e8e69
> (sha256). Thanks!
>
> [1] http://mirrors.jenkins-ci.org/war-stable-rc/2.204.3/jenkins.war
>
> --
> oliver
>
> --
> You received this message because you are subscribed to the Google
> Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to jenkinsci-de...@googlegroups.com

> <mailto:jenkinsci-de...@googlegroups.com>.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jenkinsci-dev/005cb301-b149-44b4-bf3a-25dbd5ca35dd%40googlegroups.com

> <https://groups.google.com/d/msgid/jenkinsci-dev/005cb301-b149-44b4-bf3a-25dbd5ca35dd%40googlegroups.com?utm_medium=email&utm_source=footer>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to jenkinsci-de...@googlegroups.com

> <mailto:jenkinsci-de...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtGY783zJoVQXbz3xpa7j-bmh3nnrY20ThQYU660STX9zg%40mail.gmail.com
> <https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtGY783zJoVQXbz3xpa7j-bmh3nnrY20ThQYU660STX9zg%40mail.gmail.com?utm_medium=email&utm_source=footer>.


--
oliver

[ogondza]

Mark, how does the second RC perform in your testing?

--

oliver

[Mark Waite]

The second RC has performed flawlessly in my testing.  I've been running it for the last 9 days and have detected no issues.  My environment runs a Docker image with 30 agents including Windows, Linux, and FreeBSD.  Jobs that test for specific bugs are run in that cluster and are behaving as well with 2.204.3-RC as they did with 2.204.2.

Thanks for asking,

Mark Waite

--

You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/e8b6a596-3ce3-41f0-9ab7-fcc125526d9c%40googlegroups.com.