[JIRA] (JENKINS-53611) Jenkins http interface failure after upgrading from 2.121.1 LTS to 2.138.1 LTS version

2 views
Skip to first unread message

fektom@gmail.com (JIRA)

unread,
Sep 17, 2018, 7:40:01 AM9/17/18
to jenkinsc...@googlegroups.com
Tamas Fekete created an issue
 
Jenkins / Bug JENKINS-53611
Jenkins http interface failure after upgrading from 2.121.1 LTS to 2.138.1 LTS version
Issue Type: Bug Bug
Assignee: Unassigned
Components: core
Created: 2018-09-17 11:39
Environment: Java version installed: 64-bit - jdk1.8.0_60
Operating System is: SUSE Linux Enterprise Server 11 (x86_64)
VERSION = 11
PATCHLEVEL = 4
Kernel: 3.0.101-108.10-xen
Jenkins is running via Tomcat
My web browser: Chrome - Version 68.0.3440.106 (Official Build) (64-bit)
Priority: Blocker Blocker
Reporter: Tamas Fekete

Dear Colleagues,

 

I'm working in a shared responsibility company infrastructure.

We run Jenkins in Tomcat container.

Last week I downloaded the latest LTS release (2.138.1) and made an upgrade like the followings:

  • I stopped the running Jenkins service
  • I made a backup about the old war file and its deployed directory
  • I deleted the old jenkins.war file and directory where Tomcat deployes the contents
  • I put the new war file into the same directory
  • I started Jenkins service

And at the and I tried to reach the web interface but I got this error:

"java.util.MissingResourceException: Can't find resource for bundle org.jvnet.localizer.ResourceBundleHolder$ResourceBundleImpl, key ApiTokenProperty.LegacyTokenNamejava.util.MissingResourceException: Can't find resource for bundle org.jvnet.localizer.ResourceBundleHolder$ResourceBundleImpl, key ApiTokenProperty.LegacyTokenName at java.util.ResourceBundle.getObject(ResourceBundle.java:450) at java.util.ResourceBundle.getString(ResourceBundle.java:407) at org.jvnet.localizer.ResourceBundleHolder.format(ResourceBundleHolder.java:170) at jenkins.security.Messages.ApiTokenProperty_LegacyTokenName(Messages.java:57) at jenkins.security.apitoken.ApiTokenStore$HashedToken.buildNewFromLegacy(ApiTokenStore.java:375) at jenkins.security.apitoken.ApiTokenStore.addLegacyToken(ApiTokenStore.java:155) at jenkins.security.apitoken.ApiTokenStore.deleteAllLegacyAndGenerateNewOne(ApiTokenStore.java:141) at jenkins.security.apitoken.ApiTokenStore.regenerateTokenFromLegacyIfRequired(ApiTokenStore.java:135) at jenkins.security.ApiTokenProperty.setUser(ApiTokenProperty.java:134) at hudson.model.User.load(User.java:217) at hudson.model.User.<init>(User.java:162) at hudson.model.User.getOrCreate(User.java:533) at hudson.model.User.access$300(User.java:133) at hudson.model.User$AllUsers.scanAll(User.java:1164)Caused: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104)Caused: java.lang.Error at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:110) at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296) at jenkins.model.Jenkins$5.runTask(Jenkins.java:1066) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)Caused: org.jvnet.hudson.reactor.ReactorException at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:282) at jenkins.InitReactorRunner.run(InitReactorRunner.java:48) at jenkins.model.Jenkins.executeReactor(Jenkins.java:1100) at jenkins.model.Jenkins.<init>(Jenkins.java:904) at hudson.model.Hudson.<init>(Hudson.java:85) at hudson.model.Hudson.<init>(Hudson.java:81) at hudson.WebAppMain$3.run(WebAppMain.java:233)Caused: hudson.util.HudsonFailedToLoad at hudson.WebAppMain$3.run(WebAppMain.java:250)Page generated: Sep 14, 2018 1:53:26 PM CESTJenkins ver. 2.138.1"

 

We have a many times upgraded Jenkins with very old data.
The war file which is read by Tomcat also has to be renamed hudson.war because of conventions.

We are using this Jenkins through reverse proxy.

When I stopped Jenkins service and made a restore of old war file (2.121.1 LTS) with old deployed directory everything worked again.

 

This problem may relate to this new:
"Replace single per-user API token with new system of multiple, revocable, unrecoverable API tokens with usage tracking."

 

Can you tell based on these details what's wrong and how to make a working upgrade?

 

Thank you!

 

  • Tamas Fekete
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

fektom@gmail.com (JIRA)

unread,
Oct 17, 2018, 8:41:02 AM10/17/18
to jenkinsc...@googlegroups.com
Tamas Fekete updated an issue
Change By: Tamas Fekete
Dear Colleagues,

 

I'm working in a shared responsibility company infrastructure.

We run Jenkins in Tomcat container.

Last week I downloaded the latest LTS release (*2.138.1*) and made an upgrade like the followings:
- I stopped the running Jenkins service
- I made a backup about the old war file and its deployed directory
- I deleted the old jenkins.war file and directory where Tomcat deployes the contents
- I put the new war file into the same directory
- I started Jenkins service


And at the and I tried to reach the web interface but I got this error:

" java.util.MissingResourceException: Can't find resource for bundle org.jvnet.localizer.ResourceBundleHolder$ResourceBundleImpl, key ApiTokenProperty. LegacyTokenNamejava.util.MissingResourceException: Can't find resource for bundle org.jvnet.localizer.ResourceBundleHolder$ResourceBundleImpl, key ApiTokenProperty. LegacyTokenName
at java.util.ResourceBundle.getObject(ResourceBundle.java:450)
at java.util.ResourceBundle.getString(ResourceBundle.java:407)
at org.jvnet.localizer.ResourceBundleHolder.format(ResourceBundleHolder.java:170)
at jenkins.security.Messages.ApiTokenProperty_LegacyTokenName(Messages.java:57)
at jenkins.security.apitoken.ApiTokenStore$HashedToken.buildNewFromLegacy(ApiTokenStore.java:375)
at jenkins.security.apitoken.ApiTokenStore.addLegacyToken(ApiTokenStore.java:155)
at jenkins.security.apitoken.ApiTokenStore.deleteAllLegacyAndGenerateNewOne(ApiTokenStore.java:141)
at jenkins.security.apitoken.ApiTokenStore.regenerateTokenFromLegacyIfRequired(ApiTokenStore.java:135)
at jenkins.security.ApiTokenProperty.setUser(ApiTokenProperty.java:134)
at hudson.model.User.load(User.java: 217 221 )
at hudson.model.User.<init>(User.java: 162 166 )
at hudson.model.User.getOrCreate(User.java: 533 537 )
at hudson.model.User.access$300(User.java: 133 137 )
at hudson.model.User$AllUsers.scanAll(User.java: 1164 1175 )
Caused: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104)
Caused: java.lang.Error
at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:110)
at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175)
at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
at jenkins.model.Jenkins$5.runTask(Jenkins.java: 1066 1069 )
at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused: org.jvnet.hudson.reactor.ReactorException
at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:282)
at jenkins.InitReactorRunner.run(InitReactorRunner.java:48)
at jenkins.model.Jenkins.executeReactor(Jenkins.java: 1100 1103 )
at jenkins.model.Jenkins.<init>(Jenkins.java: 904 907 )
at hudson.model.Hudson.<init>(Hudson.java:85)
at hudson.model.Hudson.<init>(Hudson.java:81)
at hudson.WebAppMain$3.run(WebAppMain.java:233)
Caused: hudson.util.HudsonFailedToLoad
at hudson.WebAppMain$3.run(WebAppMain.java:250) Page generated: Sep 14, 2018 1:53:26 PM CESTJenkins ver. 2.138.1"

 

We have a many times upgraded Jenkins with very old data.
The war file which is read by Tomcat also has to be renamed hudson.war because of conventions.

We
are using this Jenkins through reverse proxy.

When I stopped Jenkins service and made a restore of old war file (2.121.1 LTS) with old deployed directory everything worked again.

 

This problem may relate to this new feature :

"Replace single per-user API token with new system of multiple, revocable, unrecoverable API tokens with usage tracking."
Can you tell based on these details what's wrong and how to make a working upgrade?

 

Thank you!

 

- Tamas Fekete

fektom@gmail.com (JIRA)

unread,
Oct 17, 2018, 8:44:01 AM10/17/18
to jenkinsc...@googlegroups.com
Tamas Fekete commented on Bug JENKINS-53611
 
Re: Jenkins http interface failure after upgrading from 2.121.1 LTS to 2.138.1 LTS version

Tried to upgrade from 2.121.1 LTS to 2.138.2 LTS and got the same result.

fektom@gmail.com (JIRA)

unread,
Oct 17, 2018, 8:51:19 AM10/17/18
to jenkinsc...@googlegroups.com

There is a page about this security improvement: https://jenkins.io/blog/2018/07/02/new-api-token-system/

The page states, that the old encrypted API token can be replaced in Jenkins 2.129+.

As the API token conversion is only available from 2.129 I thought that if I change to that version I can convert the tokens and 2.138.2 LTS will work afterwards.

However after the change to 2.129 I got the same result:

java.util.MissingResourceException: Can't find resource for bundle org.jvnet.localizer.ResourceBundleHolder$ResourceBundleImpl, key ApiTokenProperty.LegacyTokenName

at java.util.ResourceBundle.getObject(ResourceBundle.java:450)
at java.util.ResourceBundle.getString(ResourceBundle.java:407)
at org.jvnet.localizer.ResourceBundleHolder.format(ResourceBundleHolder.java:170)
at jenkins.security.Messages.ApiTokenProperty_LegacyTokenName(Messages.java:57)

at jenkins.security.apitoken.ApiTokenStore.addLegacyToken(ApiTokenStore.java:157)
at jenkins.security.apitoken.ApiTokenStore.deleteAllLegacyAndGenerateNewOne(ApiTokenStore.java:143)
at jenkins.security.apitoken.ApiTokenStore.regenerateTokenFromLegacyIfRequired(ApiTokenStore.java:137)
at jenkins.security.ApiTokenProperty.setUser(ApiTokenProperty.java:134)
at hudson.model.User.load(User.java:216)
at hudson.model.User.<init>(User.java:161)
at hudson.model.User.getOrCreate(User.java:531)
at hudson.model.User.access$300(User.java:132)
at hudson.model.User$AllUsers.scanAll(User.java:1115)


Caused: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104)
Caused: java.lang.Error
at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:110)
at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175)
at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)

at jenkins.model.Jenkins$5.runTask(Jenkins.java:1068)


at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused: org.jvnet.hudson.reactor.ReactorException
at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:282)
at jenkins.InitReactorRunner.run(InitReactorRunner.java:48)

at jenkins.model.Jenkins.executeReactor(Jenkins.java:1102)


at jenkins.model.Jenkins.<init>(Jenkins.java:904)

at hudson.model.Hudson.<init>(Hudson.java:85)
at hudson.model.Hudson.<init>(Hudson.java:81)
at hudson.WebAppMain$3.run(WebAppMain.java:233)
Caused: hudson.util.HudsonFailedToLoad
at hudson.WebAppMain$3.run(WebAppMain.java:250)

fektom@gmail.com (JIRA)

unread,
Oct 17, 2018, 8:52:01 AM10/17/18
to jenkinsc...@googlegroups.com

fektom@gmail.com (JIRA)

unread,
Oct 24, 2018, 7:27:01 AM10/24/18
to jenkinsc...@googlegroups.com

fektom@gmail.com (JIRA)

unread,
Oct 24, 2018, 7:27:02 AM10/24/18
to jenkinsc...@googlegroups.com
Tamas Fekete started work on Bug JENKINS-53611
 
Change By: Tamas Fekete
Status: Open In Progress

fektom@gmail.com (JIRA)

unread,
Oct 24, 2018, 7:38:02 AM10/24/18
to jenkinsc...@googlegroups.com
 
Re: Jenkins http interface failure after upgrading from 2.121.1 LTS to 2.138.1 LTS version

Solution found:

 

Jenkins stores either the full APItoken or part of it in the jenkins "users" subdirectory. Like: /home/<username>/.hudson/users

If you cat out the userspecific config.xml file you will find a part like:
<jenkins.security.ApiTokenProperty>
<apiToken>{<whatever is your API token here>==}</apiToken>
</jenkins.security.ApiTokenProperty>

 

If you remove theese three lines in all userspecific config.xml, Jenkins will be able to start with no problem.

Please note, that you have to remove all three lines in every single users config.xml, otherwise the solution will not work.

 

Advise: for this purpose, I used the following method:

cd /home/<username>/.hudson/users
find . -xdev -type f -name config.xml -exec ls {} \; > result.txt
for i in `cat ./result.txt`; do sed -i '/ApiTokenProperty/d' $i; done
for i in `cat ./result.txt`; do sed -i '/apiToken/d' $i; done

 

Tamas Fekete

fektom@gmail.com (JIRA)

unread,
Oct 24, 2018, 7:40:02 AM10/24/18
to jenkinsc...@googlegroups.com
Tamas Fekete resolved as Fixed
 

Solution found. Check the details in the ticket.

Change By: Tamas Fekete
Status: In Progress Resolved
Resolution: Fixed
Reply all
Reply to author
Forward
0 new messages