upgrading agents & java version

[Jason D]

We've been holding off on the upgrade to GO 15.x due to the requirement to have all agents using JDK 1.7+  We've been using GO for a while and have accumulated about 110 agents with a mix of versions and JDKs.

What we'd like to do is script the upgrade (or replacement?) of all our agents to match the latest version of the core server and recommended JDK.  

Several questions come to mind:

• can the process be done entirely through APIs now?  If not, which parts can/cannot?  Enable, env/resource tags...?
  
• would it be better to upgrade the agents in place or do a clean wipe and replace?
• can I leave the mix of agents as is and just install a new java version and update the agent config?  
• what is the tolerance to having a mix of agents for GO?  How old can some of them be?
• would all the mappings in GO be wiped out in either case (environment, resource tags, etc...)

Looking for the cleanest and easiest way to do this.  Also, if anyone has any scripts they want to share that'd be appreciated too.

Thanks.

jason

[Aravind SV]

I think this is a really useful question. I'll answer some of these questions with ideas and opinions. Others should feel free to chip in too.

On Thu, Jul 16, 2015 at 2:19 PM, Jason D <jason...@gmail.com> wrote:

what is the tolerance to having a mix of agents for GO?  How old can some of them be?

The Go Server will ensure that the Go Agent it is communicating with is always in sync. It will refuse to use an agent, unless it is upgraded. To understand this, you need to know that what you install as a part of any installer is called the "Agent Bootstrapper". Its main job is to download a new "Agent" from the Go Server, when it upgrades.

So, the question can be rephrased a bit: "Given that the agent will update itself, how old of an agent bootstrapper can one of the latest Go Servers work with?".

Though the recommend agent bootstrapper is the latest one, most old ones will work. The last upgrade I remember is the one needed for task plugins (14.1). If you have anything newer than that, it should work. Even with 15.2, it should continue working, given that you're using Java 7 with it.

can the process be done entirely through APIs now?  If not, which parts can/cannot?  Enable, env/resource tags...?

 

would it be better to upgrade the agents in place or do a clean wipe and replace?

I would try an upgrade, rather than a clean wipe and replace. I would try it on one agent first, through the silent installation option (assuming Windows). I'm not sure it works for an upgrade. If it does, it is fine. If it does not, my next step would be to automate the uninstallation and then silent installation. A very important step before uninstallation, is to take a backup of the agent keystore and trust store as well as the guid.txt file. If you put that back before or after the upgrade, then all of the settings that you have (environments, resources, etc) should be maintained on the server, since the agent will use the values in those files and it'll look like the old agent to the server. It'll be registered and you'll not need to do anything new. Remember that you'll need to restart the agent once you replace those files.

would all the mappings in GO be wiped out in either case (environment, resource tags, etc...)

As I said, done properly, you should be able to keep everything the same, and have the Go Agents upgraded. If anyone else thinks differently, please correct me.

can I leave the mix of agents as is and just install a new java version and update the agent config?

Sure, that will work too. You'll need to change the GO_AGENT_JAVA_HOME environment variable, I guess.

 

We've been holding off on the upgrade to GO 15.x due to the requirement to have all agents using JDK 1.7+  We've been using GO for a while and have accumulated about 110 agents with a mix of versions and JDKs.

What we'd like to do is script the upgrade (or replacement?) of all our agents to match the latest version of the core server and recommended JDK.  

Looking for the cleanest and easiest way to do this.  Also, if anyone has any scripts they want to share that'd be appreciated too.

I hope I'm not building too many castles in the air. I don't have a script. But, if I had to do this and I was using Puppet/Chef/Ansible etc., this would be easy. Without that, assuming Windows, I'd come up with a defensively written Powershell script which evaluates an agent and tells you the current state of JDK version/Go Agent version and what it wants to do to update it, before doing it. It might need to know the differences between an agent running as a Windows service vs. an agent running as a script, etc.

Once you have a tested script, which works on a few agents tried manually, I'd use either psexec or Go's own "Run On All Agents" to upgrade the agents. Using "Run On All Agents" to upgrade an agent might be a little tricky, since an agent cannot probably upgrade itself, since it is "in use". But, you might need to use some kind of a scheduled task set to run once, in a minute or two after it is created, so that the "Run On All Agents" job can finish, before the agent gets upgraded. However, using psexec might give you more fine grained control.

Finally, I think it's a very prudent decision to automate it. As I was mentioning elsewhere, the Java juggernaut is getting quicker. Java 7's public updates have already stopped. Java 8 is already quite prevalent, and having a way to upgrade the Java you're using will become quite important. Go will need to require more upgrades, if Java upgrades are quicker. Otherwise, there will be security risks and incompatibilities to deal with.

Cheers,

Aravind

[Carl Reid]

Great questions and great answers. 

There are so many agent related issues but the documentation is quite sparse in this area.

I feel there needs to be a specific section of the documentation dealing with these type of scenarios and the automation questions around them. We use a lot of automation scripts to install and register agents and deal with agents that become unresponsive due to machine shut-down or in the case of Amazon based agents when the instance is terminated. 

Another area is how to create agents that can perform builds, tests etc, this is of course dependent on your technology stack however there must be some common themes which would benefit a large number of users.

I would be happy to contribute towards such a document.

Carl

[Ketan Padegaonkar]

Carl,

If you have any thoughts or ideas to improve or add to the documentation that we have. We're happy to accept any and all improvements to the documentation :)

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jason D]

I don't believe most of what Aravind has mentioned is in the doc that i can find, except the silent api install thing.

Another quetion, what's the latest version of the JDK that has been tested w/GO?

[Jason D]

Any thoughts on what version of the JDK and bitness (32/64) is best when upgrading existing agents?

On Thursday, July 16, 2015 at 1:19:57 PM UTC-5, Jason D wrote:

[Jyoti Singh]

Latest version we have tested it with is java 7u79, but many others from the community have been using Go 15.2 successfully with 7u85 and java 8 as well. 

In general, I would recommend using 64 bit java 7 but there is more (this and this) that you should know before deciding on the version to upgrade to. 

[Jason D]

For context, we are currently running GO 14.4 with the core server and most of our agents using JDK 1.6.x.  We want to upgrade to GO 15.2 but need to migrate to JDK 1.8.x to do so.  I've read the SSL issue people have had and have a couple more specific questions that will hopefully help us and others:

- do we have to upgrade the JDK for both (core server, agent) at the same time or could we update all the agents to use jdk 1.8 and then do the core server at another time?  Could we do the reverse?

- do we have to upgrade to GO 15.2 at the same time or can the upgrade to JDK 1.8.x be done first w/no issues?

- do the JDK versions between the core server and all the agents have to match exactly?

Thanks.

On Thursday, July 16, 2015 at 1:19:57 PM UTC-5, Jason D wrote:

[Aravind SV]

It might help to just download 15.2 and try it with your exact version of Java, to be very sure. Even if it is with an empty database and just a sample pipeline. I doubt that 14.4 supports Java 1.8, because they removed some of the ciphers Go was using in 7u85 (look for "Prohibit RC4 cipher suites" in this document).

Because they've removed it in 7u85, and you're going from 1.6 to 1.8, I don't see a way without upgrading everything together. If you leave it at 14.4 and upgrade to JRE 1.8, then the agents won't connect since the ciphers are gone. If you upgrade to 15.2, leaving the JRE at 1.6, then 15.2 won't work as it needs 1.7.

What you can try, if it makes sense for you, is to upgrade the server to 7u79 (not 7u85 or above). 14.4 should continue to work - especially agents. You can check that, and if not, revert to 1.6. Now, you can upgrade the server to 15.2 (since you're on JRE 7). Agents should all upgrade themselves and continue to work. Now, agents can be upgraded all the way to 1.8, in a phased manner, if you want. Both 1.6 and 1.8 agents should work.

As Jyoti said, we haven't tested 1.8 agents with 1.7 server. So, no real guarantees about JRE 8, unfortunately. I'd test this with a fresh install of 15.2, to see if it works. I'd definitely wait for Jyoti to take a look at this and respond (probably Thursday, since she's away for a day). She probably knows this best.

--

[Jyoti Singh]

Well, Aravind has explained the Go and java version limitations pretty well so I have nothing more to add there.

In terms of doing this upgrade in a phased manner, I would go with most of what Aravind recommends, with just one difference that is to do with java1.6 on agents. 

Basically, stay on v 14.4, upgrade to java7 on server first, upgrade to java7 on agents in a phased manner. Once everything is on java7, upgrade your Go server to 15.2. While, theoritically, java6 on agents can work with 15.2 server, but we had decided to drop support for it a long time ago, hence we haven't really tested 15.2 with java6. Given that, I would say say stick to java7 for now, else you might see surprises. 

>> - do the JDK versions between the core server and all the agents have to match exactly?

No, in general the versions do not have to match exactly. However, there were pretty big changes done to java in recent past, so picking up a java version is no more as simple as pick any version of java7. Any version of java7 upto 7u79 will work well with all versions of Go between 13.x to 15.x. Move to 7u85 or later versions (with RC4 disabled) only after upgrading to 15.2(latest as of now). If you wish to move to java8, it can be done only at this point, be sure to upgrade the server first. If you need to tweek the cipher suites being used by the server, you may find this useful.

Does that make sense?

Cheers,

Jyoti

--

Cheers,
Jyoti

[Jyoti Singh]

Slipped off my mind, java6 won't work for sure as 15.2 source code was compiled with java7 compatibility mode. 

--

Cheers,
Jyoti

[Jason D]

What are the requirements, instructions, gotchas for upgrading Postgres, if any, when upgrading GO from 14.4 to 15.2?

On Thursday, July 16, 2015 at 1:19:57 PM UTC-5, Jason D wrote:

[Jyoti Singh]

Requirements are the same as previous versions of the addon. You can find the instructions for upgrading here.

No Gotchas :). Before you upgrade, just get in touch with your Thoughtworks sales contact to get the 15.2 version of the addon if you do not one already. Go will not work with an older version of the addon.

--

You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Cheers,
Jyoti