PodConfiguration cached issue for Kubernetes Elastic Agent

[Maxim Shaev]

Hi guys,

I've successfully install GoCD through helm. The sample pipeline is successfully built using Kubernetes Elastic Agent.

While I was adding a new pipeline, I needed to be added SSH keys to GoCD server, and elastic profile agent according to the documentation 

https://github.com/gocd/kubernetes-elastic-agents

But, probably the syntax of YAML format for Pod was wrong. I've got an error in the logs

2019-09-07 13:23:13,250 ERROR [161@MessageListener for CreateAgentListener] KubernetesPlugin:122 - YAML decoding problem: while parsing a block collection

 in 'reader', line 14, column 5:

        - name: gocd-agent-container-b03 ... 

        ^

expected <block end>, but found Key

 in 'reader', line 18, column 5:

        volumeMounts:

        ^

 (through reference chain: io.fabric8.kubernetes.api.model.Pod["spec"]->io.fabric8.kubernetes.api.model.PodSpec["containers"])

2019-09-07 13:23:13,258 ERROR [161@MessageListener for CreateAgentListener] KubernetesPlugin:127 - Failed to handle request cd.go.elastic-agent.create-agent

java.lang.NullPointerException: null

at cd.go.contrib.elasticagent.KubernetesInstanceFactory.setGoCDMetadata(KubernetesInstanceFactory.java:108)

at cd.go.contrib.elasticagent.KubernetesInstanceFactory.createUsingPodYaml(KubernetesInstanceFactory.java:246)

at cd.go.contrib.elasticagent.KubernetesInstanceFactory.create(KubernetesInstanceFactory.java:61)

at cd.go.contrib.elasticagent.KubernetesAgentInstances.createKubernetesInstance(KubernetesAgentInstances.java:89)

at cd.go.contrib.elasticagent.KubernetesAgentInstances.create(KubernetesAgentInstances.java:67)

at cd.go.contrib.elasticagent.KubernetesAgentInstances.create(KubernetesAgentInstances.java:38)

at cd.go.contrib.elasticagent.executors.CreateAgentRequestExecutor.execute(CreateAgentRequestExecutor.java:44)

at cd.go.contrib.elasticagent.KubernetesPlugin.handle(KubernetesPlugin.java:83)

at com.thoughtworks.go.plugin.infra.DefaultPluginManager.lambda$submitTo$0(DefaultPluginManager.java:152)

at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.executeActionOnTheService(FelixGoPluginOSGiFramework.java:303)

at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.doOn(FelixGoPluginOSGiFramework.java:245)

at com.thoughtworks.go.plugin.infra.DefaultPluginManager.submitTo(DefaultPluginManager.java:149)

at com.thoughtworks.go.plugin.access.PluginRequestHelper.submitRequest(PluginRequestHelper.java:49)

at com.thoughtworks.go.plugin.access.elastic.v5.ElasticAgentExtensionV5.createAgent(ElasticAgentExtensionV5.java:142)

at com.thoughtworks.go.plugin.access.elastic.ElasticAgentExtension.createAgent(ElasticAgentExtension.java:60)

at com.thoughtworks.go.plugin.access.elastic.ElasticAgentPluginRegistry.createAgent(ElasticAgentPluginRegistry.java:43)

at com.thoughtworks.go.server.messaging.elasticagents.CreateAgentListener.onMessage(CreateAgentListener.java:32)

at com.thoughtworks.go.server.messaging.elasticagents.CreateAgentListener.onMessage(CreateAgentListener.java:22)

at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.runImpl(JMSMessageListenerAdapter.java:86)

at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.run(JMSMessageListenerAdapter.java:66)

at java.base/java.lang.Thread.run(Unknown Source)

Therefor, agent is not be able to up.

After removing volumeMounts I still got the error.

I've removed volumeMounts in profile

• /go/admin/elastic_profiles
• double checked in cruise-config.xml, 
• restart GoCD server Pod

but still there is an issue.

Should I've reset cache, if there is one or do something else to apply the changes?

Stack:

GoCD 19.7.0

Kubernetes Elastic Agent Plugin 3.0.0-156

$ kubectl version

Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:13:54Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568", GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}

[Varsha Varadarajan]

Hello,

Did you try cancelling and rerunning the job?

Thanks,

Varsha

[Maxim Shaev]

Did you try cancelling and rerunning the job?

OMG, thanks! It works! It took the latest changes after cancelling and rerunning.
I had hoped recreating the pod should solve the issue, because didn't find any other temporary files in the Persistence Volume. It looks like, the server keeps the current stage/session in DB.

Thank you again.

[Varsha Varadarajan]

Hello,

Yes, the elastic profile is stored in the DB for the job and deleted later on after job completion. So, if you wanted an already scheduled job to use another elastic profile, you need to rerun. 

Maybe there is some improvement that can be done in case of invalid pod yaml. Not sure.

Thank you,

Varsha

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/05ac76be-8b27-4ef0-8810-04ddc9aa3152%40googlegroups.com.