Jenkins 2.73.1 LTS RC testing started

96 views
Skip to first unread message

Oliver Gondža

unread,
Sep 5, 2017, 7:29:44 AM9/5/17
to jenkin...@googlegroups.com
Hello everyone,

Latest LTS RC was made public and it is ready to be tested. Release is
scheduled for 2017-09-13.

Report your findings in this thread or on the test plan wiki page.

Download bits from
http://mirrors.jenkins-ci.org/war-stable-rc/2.73.1/jenkins.war
Check community maintained LTS test plan
https://wiki.jenkins-ci.org/display/JENKINS/LTS+2.73.x+RC+Testing

Thanks
--
oliver

Mark Waite

unread,
Sep 6, 2017, 10:26:31 AM9/6/17
to jenkin...@googlegroups.com
Oliver,

I'm seeing behavioral differences in ssh between the 2.73.1 release candidate.  For example, one of my agents using an ed25519 private key starts correctly from 2.60.3, but does not start from 2.73.1 release candidate.  

I've seen two cases where JGit based ssh protocol cloning of a remote repository fails on 2.73.1 release candidate when it consistently succeeds on 2.60.3.

I don't have enough information for good bug reports on those issues yet.  I'll gather that tomorrow night.

Can someone confirm that there have been changes in ssh communication between 2.60.3 and 2.73.1?  If there have been no changes, or few changes, then I need to focus my investigation differently than if there are known changes in ssh between those two releases.

Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/44cba1b8-e653-3ab3-3383-eb63d8e004f1%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Jesse Glick

unread,
Sep 6, 2017, 1:47:36 PM9/6/17
to Jenkins Dev
On Wed, Sep 6, 2017 at 10:26 AM, Mark Waite <mark.ea...@gmail.com> wrote:
> I've seen two cases where JGit based ssh protocol cloning of a remote
> repository fails on 2.73.1 release candidate when it consistently succeeds
> on 2.60.3.

You should be able to bisect then, and identify the first version to
display the problem. From there it is probably obvious which changelog
entry to look at.

Mark Waite

unread,
Sep 6, 2017, 2:35:08 PM9/6/17
to jenkin...@googlegroups.com
Thanks for the suggestion (and the amazing power of docker to construct complex scenarios with trivial effort).

The mistake is mine.  I decided to evaluate JGit 4.8.0 at the same time I was evaluating Jenkins 2.73.1 pre-release.  The problem is either in JGit 4.8.0 or in the git client plugin calls to JGit 4.8.0.  When I switch back to the released version of the git client plugin, the problem is resolved.

I don't know why it was specifically two cases with a single git hosting vendor, and not with any of the other hosting vendors, but I'll take a bug detection any way I can get it.

No issue for the 2.73.1 release candidate.

Mark Waite
 
--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

ogondza

unread,
Sep 9, 2017, 4:25:33 AM9/9/17
to Jenkins Developers
I have reverted remoting back to version 3.10 and pushed another RC asking Mark to retest.

Relevant issue https://issues.jenkins-ci.org/browse/JENKINS-46754

--
oliver

Mark Waite

unread,
Sep 11, 2017, 6:03:10 PM9/11/17
to Jenkins Developers
I reported JENKINS-46754 - 2.73.1 RC agent won't start if using ed25519 passphrase protected key and have explored it further.

The bug is unrelated to the remoting version.  The change which Oliver made (at my request) to switch from remoting 3.11 to remoting 3.10 did not change the behavior of that bug.

I recommend we return the 2.73.1 LTS to remoting 3.11.  My initial testing was with remoting 3.11, and the issue I detected (JENKINS-46754) was only with ed25519 passphrase protected private keys on two of my machines (but not on the other four tested machines).  Agents started with rsa private keys (with or without passphrase) and DSA private keys (with or without passphrase) are unaffected.  Agents started with ed25519 private keys successfully on 4 of my 6 configurations. 

I still don't understand the root of the behavioral differences noted in JENKINS-46574.  Hours and hours of testing on multiple platforms with many different test cases has not shown any new or unexpected issues beyond the ed25519 inconsistency that I've been studying.

Those machines which show the inconsistency also show a JGit clone failure when using an ed25519 private key to access the repository.  That seems (to me) to point to a difference in a library version, but I can't explain why only 2 of the 6 tested machines behave that way.

Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

Oliver Gondža

unread,
Sep 12, 2017, 2:46:44 AM9/12/17
to jenkin...@googlegroups.com
Mark, Thanks for the extensive testing. Despite the fact his was not
confirmed as remoting problem, I am leaving the remoting version bump
reverted so the version will be 3.10 in LTS 2.73.1 and the candidate
will be reconsidered for .2. So the current RC is what we will release
on Wednesday in case people have cycles to pour some more testing into it.

On 2017-09-12 00:02, Mark Waite wrote:
> I reported JENKINS-46754 - 2.73.1 RC agent won't start if using ed25519
> passphrase protected key
> <https://issues.jenkins-ci.org/browse/JENKINS-46754> and have explored
oliver

Oleg Nenashev

unread,
Sep 13, 2017, 6:02:46 AM9/13/17
to Jenkins Developers
Hi,

Unfortunately I had no time to deploy the RC on my test instance due to conferences. So I have no testing results

Regarding Remoting, there is a regression in 3.10 - JENKINS-45755. Effectively it prevents channel from being established for SSH agents when Jenkins master's JAR Cache cannot be created (e.g. directory is not writable). I am not sure how many users this defect is going to affect, I will let my colleagues to comment.

I have created a pull request today. If we agree the defect is urgent enough, I will be able to spin 3.10.1 on Thursday, ~10AM UTC. Please respond if such release is required.

BR, Oleg

вторник, 5 сентября 2017 г., 12:29:44 UTC+1 пользователь ogondza написал:

Vincent Latombe

unread,
Sep 13, 2017, 6:11:41 AM9/13/17
to Jenkins Dev
Regarding JENKINS-45755, affected users will be those running Jenkins with a non-writable user home directory (like a service user for the issue logger).
This is a more common situation in Docker environment, in cases you have to run Jenkins as a specific uid in order to mount a nfs directory with specific permissions as JENKINS_HOME. In this case, the user doesn't even have a user home directory.

Vincent

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/790c9f18-9ba8-4a0e-9f7e-9a45f2defd7a%40googlegroups.com.

Daniel Beck

unread,
Sep 13, 2017, 6:40:48 AM9/13/17
to Jenkins Developers

> On 13. Sep 2017, at 12:11, Vincent Latombe <vincent...@gmail.com> wrote:
>
> Docker environment

Is this going to be a problem in the official Jenkins Docker images?

Vincent Latombe

unread,
Sep 13, 2017, 8:27:21 AM9/13/17
to Jenkins Dev
I created a reproducer (in the ticket) using only the official Jenkins Docker image.

Andres Rodriguez

unread,
Sep 13, 2017, 9:26:00 AM9/13/17
to jenkin...@googlegroups.com
Hi,

In case the issue is finally considered severe enough to be included in .1, I just wanted to point out that we completed out testing with the rc before remoting was retracted from 3.11 to 3.10 and found no issues, in case it helps in deciding whether (and where) a backport is needed.

Thanks 

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
Regards,

Andres Rodriguez

Jesse Glick

unread,
Sep 13, 2017, 10:06:32 AM9/13/17
to Jenkins Dev
On Wed, Sep 13, 2017 at 8:26 AM, Vincent Latombe
<vincent...@gmail.com> wrote:
>> Is this going to be a problem in the official Jenkins Docker images?
>
> I created a reproducer (in the ticket) using only the official Jenkins
> Docker image.

IIUC we can simply patch the image to either have a writable `$HOME`,
or specify an explicit `-workDir`, so that most users are not
affected—right?

Mark Waite

unread,
Sep 13, 2017, 10:11:39 AM9/13/17
to Jenkins Dev
I'd prefer that we deliver 2.73.1 LTS with remoting 3.11.  I don't see the benefit of remaining with 3.10.

The cases I tested showed no difference between remoting 3.10 and remoting 3.11, and there is a known regression in remoting 3.10 which is fixed in remoting 3.11.

Mark Waite

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr1ZVmt2DsZ0par%2B69Whq_ONnwJqng3im1ds1WVOATggUA%40mail.gmail.com.

Vincent Latombe

unread,
Sep 13, 2017, 10:16:25 AM9/13/17
to Jenkins Dev
The workaround I have applied in my product is to force set user.home (to $JENKINS_HOME) when launching Jenkins.
-workDir can't be specified for a master. This flag is only available when launching the remoting jar, not when it is called by creating a channel on master side.
 

Jesse Glick

unread,
Sep 13, 2017, 10:16:37 AM9/13/17
to Jenkins Dev
On Wed, Sep 13, 2017 at 10:11 AM, Mark Waite <mark.ea...@gmail.com> wrote:
> there is a known regression in remoting 3.10 which is fixed in
> remoting 3.11.

Which? Not JENKINS-45755.

Mark Waite

unread,
Sep 13, 2017, 10:20:34 AM9/13/17
to jenkin...@googlegroups.com
Did I misunderstand Oleg's earlier statement?  He said:

Regarding Remoting, there is a regression in 3.10 - JENKINS-45755. Effectively it prevents channel from being established for SSH agents when Jenkins master's JAR Cache cannot be created (e.g. directory is not writable). I am not sure how many users this defect is going to affect, I will let my colleagues to comment. 

Mark Waite
 
--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

Jesse Glick

unread,
Sep 13, 2017, 10:23:55 AM9/13/17
to Jenkins Dev
On Wed, Sep 13, 2017 at 10:20 AM, Mark Waite <mark.ea...@gmail.com> wrote:
> Did I misunderstand Oleg's earlier statement?

He said there was a regression in 3.10. He did not say it was fixed in
3.11. The proposed fix is still under review.

Oleg Nenashev

unread,
Sep 13, 2017, 10:24:57 AM9/13/17
to JenkinsCI Developers
Yes. Regression is in 3.10, but it has not been fixed in 3.11
Release of 3.12 is required to fix this regression.

My plan is to spin it tomorrow.

Regarding just updating LTS, note that 3.11 includes new API changes. Although they a minor, I am not sure that just bumping the version is safe enough.




--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/uv2SG7k2log/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr23ZJbhSstQaunZMsikjaB1iuOrFVV7%3DsyThOxovfm%2B4Q%40mail.gmail.com.

Mark Waite

unread,
Sep 13, 2017, 10:36:05 AM9/13/17
to jenkin...@googlegroups.com
On Wed, Sep 13, 2017 at 8:24 AM Oleg Nenashev <o.v.ne...@gmail.com> wrote:
Yes. Regression is in 3.10, but it has not been fixed in 3.11
Release of 3.12 is required to fix this regression.

My plan is to spin it tomorrow.

Regarding just updating LTS, note that 3.11 includes new API changes. Although they a minor, I am not sure that just bumping the version is safe enough.



Thanks for the clarificaiton.  I retract my preference for remoting 3.11.

Mark Waite

 


2017-09-13 15:23 GMT+01:00 Jesse Glick <jgl...@cloudbees.com>:
On Wed, Sep 13, 2017 at 10:20 AM, Mark Waite <mark.ea...@gmail.com> wrote:
> Did I misunderstand Oleg's earlier statement?

He said there was a regression in 3.10. He did not say it was fixed in
3.11. The proposed fix is still under review.

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/uv2SG7k2log/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

Daniel Beck

unread,
Sep 13, 2017, 10:39:48 AM9/13/17
to jenkin...@googlegroups.com

> On 13. Sep 2017, at 16:06, Jesse Glick <jgl...@cloudbees.com> wrote:
>
> IIUC we can simply patch the image to either have a writable `$HOME`,
> or specify an explicit `-workDir`, so that most users are not
> affected—right?
>

Right now, I don't think 2.73.1 is ready for release due to this issue. We _really_ need this fixed before we publish 2.73.1. Otherwise it will just not be usable in any serious environment.

Oleg Nenashev

unread,
Sep 13, 2017, 10:51:40 AM9/13/17
to JenkinsCI Developers
I will likely miss the beginning of the Governance meeting today unless my plane lands super-fast. My vote: I agree that the issue is serious enough to delay the release.

My proposal would be to spin 3.10.1 only with this fix. In such case we could delay LTS release till Friday and do some spot-checks of the new RC candidates with the fix.



--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/uv2SG7k2log/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/C8769C58-4123-4ED6-A8AB-986AD2CA75E1%40beckweb.net.

Jesse Glick

unread,
Sep 13, 2017, 12:20:56 PM9/13/17
to Jenkins Dev
On Wed, Sep 13, 2017 at 10:51 AM, Oleg Nenashev <o.v.ne...@gmail.com> wrote:
> My proposal would be to spin 3.10.1 only with this fix. In such case we
> could delay LTS release till Friday and do some spot-checks of the new RC
> candidates with the fix.

+1

ogondza

unread,
Sep 14, 2017, 1:42:19 AM9/14/17
to Jenkins Developers
Agreed on the meeting the release will be cut with the regression compared to 2.60 line (with remoting 3.10) and Jenkins docker container will be updated not to be impacted. The proper fix will target 2.73.2.

--
oliver

Oleg Nenashev

unread,
Sep 14, 2017, 8:12:40 AM9/14/17
to JenkinsCI Developers
Hello,

I have release Remoting 3.12 with a fix.
I hope to get it integrated into the weekly, PR: https://github.com/jenkinsci/jenkins/pull/3025

BR, Oleg

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/uv2SG7k2log/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.

Daniel Beck

unread,
Sep 14, 2017, 12:50:30 PM9/14/17
to jenkin...@googlegroups.com, Oliver Gondža

> On 14. Sep 2017, at 07:42, ogondza <ogo...@gmail.com> wrote:
>
> Agreed on the meeting the release will be cut with the regression compared to 2.60 line (with remoting 3.10) and Jenkins docker container will be updated not to be impacted. The proper fix will target 2.73.2.

We have some new information on the issue Mark discovered during testing:

https://issues.jenkins-ci.org/browse/JENKINS-46754 (latest comments)

Oliver, would you support integrating a fix for that into stable-2.73, assuming it's straightforward enough, and available today/tonight?

Daniel

Daniel Beck

unread,
Sep 14, 2017, 5:00:16 PM9/14/17
to Jenkins Developers, Oliver Gondža

> On 14. Sep 2017, at 18:50, Daniel Beck <m...@beckweb.net> wrote:
>
> We have some new information on the issue Mark discovered during testing:
>
> https://issues.jenkins-ci.org/browse/JENKINS-46754 (latest comments)
>
> Oliver, would you support integrating a fix for that into stable-2.73, assuming it's straightforward enough, and available today/tonight?

We are going to ship 2.73.1 as planned and note this issue in the changelog and upgrade guide.

Reply all
Reply to author
Forward
0 new messages