[JIRA] (JENKINS-58257) New secrets written to kubernetes store are not always loaded by Jenkins

0 views
Skip to first unread message

steve.bussetti@gmail.com (JIRA)

unread,
Jun 28, 2019, 1:55:02 PM6/28/19
to jenkinsc...@googlegroups.com
steve bussetti created an issue
 
Jenkins / New Feature JENKINS-58257
New secrets written to kubernetes store are not always loaded by Jenkins
Issue Type: New Feature New Feature
Assignee: James Nord
Components: kubernetes-credentials-provider-plugin
Created: 2019-06-28 17:54
Environment: Jenkins ver. 2.164.3

Kubernetes Credentials Plugin 0.4.0
Priority: Minor Minor
Reporter: steve bussetti

We have observed that sometimes Jenkins does not pick up new secrets after they have been added to kubernetes.  

We do not observe any seemingly related errors in the jenkins log and are forced to restart jenkins in order for it to pick up the new secrets.

 

Sometimes it does find the new secrets but not always and there is not a clear indicator of when it will or will not find new secrets and display them in the credential store UI.

 

Is there any way to manually have it rescan for secrets?
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

steve.bussetti@gmail.com (JIRA)

unread,
Jun 28, 2019, 2:00:03 PM6/28/19
to jenkinsc...@googlegroups.com
steve bussetti updated an issue
 
Jenkins / Bug JENKINS-58257
Change By: steve bussetti
Issue Type: New Feature Bug

jnord@cloudbees.com (JIRA)

unread,
Jun 28, 2019, 2:37:02 PM6/28/19
to jenkinsc...@googlegroups.com
James Nord commented on Bug JENKINS-58257
 
Re: New secrets written to kubernetes store are not always loaded by Jenkins

when this happens can you attache a stackdump of your jenkins instance and the jenkins logs please.

as a workaround you should be able to run the following in the groovy console

```
import com.cloudbees.jenkins.plugins.kubernetes_credentials_provider.KubernetesCredentialProvider

def kp = ExtensionList.lookupSingletom(KubernetesCredentialProvider.class)
kp.stopWatchingForSecrets()
kp.startWatchingForSecrets()
```

jnord@cloudbees.com (JIRA)

unread,
Jun 28, 2019, 2:38:02 PM6/28/19
to jenkinsc...@googlegroups.com
James Nord edited a comment on Bug JENKINS-58257
when this happens can you attache a stackdump of your jenkins instance and the jenkins logs please.

as a workaround you should be able to run the following in the groovy console
``` {code}
import com.cloudbees.jenkins.plugins.kubernetes_credentials_provider.KubernetesCredentialProvider

def kp = ExtensionList.lookupSingletom(KubernetesCredentialProvider.class)
kp.stopWatchingForSecrets()
kp.startWatchingForSecrets()
``` {code}

steve.bussetti@gmail.com (JIRA)

unread,
Jul 3, 2019, 8:26:02 AM7/3/19
to jenkinsc...@googlegroups.com

Can confirm the workaround works – will provide addl details.

kyle.cronin@adp.com (JIRA)

unread,
Aug 26, 2019, 11:36:02 AM8/26/19
to jenkinsc...@googlegroups.com

We have experienced this as well. Most likely the issue is the watch task closes due to some error like:

```
2019-08-25 06:20:05.874+0000 [id=43] INFO c.c.j.p.k.KubernetesCredentialProvider#onClose: onClose.

io.fabric8.kubernetes.client.KubernetesClientException: too old resource version: 130846525 (135483452)
at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onMessage(WatchConnectionManager.java:254)
at okhttp3.internal.ws.RealWebSocket.onReadMessage(RealWebSocket.java:323)
at okhttp3.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.java:219)
at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:105)
at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274)
at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214)
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206)
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
``` 

The plugin does not attempt to recover from a closed watch connection.

kyle.cronin@adp.com (JIRA)

unread,
Aug 26, 2019, 5:40:03 PM8/26/19
to jenkinsc...@googlegroups.com
Kyle Cronin edited a comment on Bug JENKINS-58257
The plugin does not attempt to recover from a closed watch connection. I've submitted a PR to restart the watcher if the onClose method is called with a client exception.

https://github.com/jenkinsci/kubernetes-credentials-provider-plugin/pull/31

peter.niederlag@datenbetrieb.de (JIRA)

unread,
Nov 1, 2019, 9:47:03 AM11/1/19
to jenkinsc...@googlegroups.com

ran into this as well, workaround needs a Singleton, wont work with Singletom. 

import com.cloudbees.jenkins.plugins.kubernetes_credentials_provider.KubernetesCredentialProvider

def kp = ExtensionList.lookupSingleton(KubernetesCredentialProvider.class)
kp.stopWatchingForSecrets()
kp.startWatchingForSecrets()
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

jnord@cloudbees.com (JIRA)

unread,
Nov 1, 2019, 10:22:03 AM11/1/19
to jenkinsc...@googlegroups.com
James Nord edited a comment on Bug JENKINS-58257
[~jugglefish] this is should have been fixed in kubernetes-credentials-provider-0.13

 

if it is not working for you please create a new ticket with all the logs from the provider

jnord@cloudbees.com (JIRA)

unread,
Nov 1, 2019, 10:22:03 AM11/1/19
to jenkinsc...@googlegroups.com

Peter Niederlag this is fixed in kubernetes-credentials-provider-0.13

jnord@cloudbees.com (JIRA)

unread,
Nov 1, 2019, 10:22:04 AM11/1/19
to jenkinsc...@googlegroups.com
James Nord closed an issue as Fixed
 
Change By: James Nord
Status: Open Closed
Resolution: Fixed
Released As: kubernetes-credentials-provider-0.13
Reply all
Reply to author
Forward
0 new messages