confusion regarding agent serverUrl configuration and load balancer

[dle...@gmail.com]

Hello team -

Question:

Is it expected that windows go-agents running v18.10.0 can be configured to point to our Load balancer terminating SSL but linux agents running 19.9.0 cannot?

Background:

Our GoCD installation has gotten state at version 19.8.0 and we are making plans to upgrade in phases all the way up to the latest 21.2.0.

Our first plan of attack is to upgrade all our go-agents to the latest version which still plays nice with our version of the GoCD server, which I believe will be go-agent 19.9.0

As I see in the release notes for GoCD 20.2.0,  GoCD is dropping the 8154 port in favor of having users install their own reverse proxy or load balancer.  Our GoCD server is fronted with an AWS ELB terminating our SSL, and we have windows 18.10.0 go-agents configured to talk to the server through the load balancer and our 19.9.0 linux agents are talking directly to the go-server via port 8154.

I thought it would make sense to reconfigure our linux agents to talk to the load balancer, like our current windows agents are doing.

I tested this out, deleting the agents guid.txt and agent.jks file.  The server sees the new agent as pending and I can then accept it and it switches to idle,

but eventually the agent switches state to LostContact and I see this in the agent.log file:

2021-05-18 16:15:26,158 ERROR [scheduler-3] AgentController:91 - [Agent Loop] Error occurred during loop:
org.springframework.remoting.RemoteAccessException: Could not access HTTP invoker remote service at [https://mydomain.com/go/remoting/remoteBuildRepository]; nested exception is o
rg.apache.http.client.ClientProtocolException: The server returned status code 403. Possible reasons include:
   - This agent has been deleted from the configuration
   - This agent is pending approval
   - There is possibly a reverse proxy (or load balancer) that is terminating SSL. Hint: use port 8154 of the GoCD server. See https://docs.gocd.org/19.8.0/installation/configure-reverse-proxy.html
#agents-and-reverse-proxies for details.
        at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.convertHttpInvokerAccessException(HttpInvokerClientInterceptor.java:226)
        at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.invoke(HttpInvokerClientInterceptor.java:153)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
        at com.sun.proxy.$Proxy10.getCookie(Unknown Source)
        at com.thoughtworks.go.agent.AgentHTTPClientController.retrieveCookieIfNecessary(AgentHTTPClientController.java:129)
        at com.thoughtworks.go.agent.AgentHTTPClientController.work(AgentHTTPClientController.java:118)
        at com.thoughtworks.go.agent.AgentController.loop(AgentController.java:85)
        at jdk.internal.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)
        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
        at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.apache.http.client.ClientProtocolException: The server returned status code 403. Possible reasons include:
   - This agent has been deleted from the configuration
   - This agent is pending approval
   - There is possibly a reverse proxy (or load balancer) that is terminating SSL. Hint: use port 8154 of the GoCD server. See https://docs.gocd.org/19.8.0/installation/configure-reverse-proxy.html
#agents-and-reverse-proxies for details.
        at com.thoughtworks.go.agent.GoHttpClientHttpInvokerRequestExecutor.validateResponse(GoHttpClientHttpInvokerRequestExecutor.java:112)
        at com.thoughtworks.go.agent.GoHttpClientHttpInvokerRequestExecutor.doExecuteRequest(GoHttpClientHttpInvokerRequestExecutor.java:79)
        at org.springframework.remoting.httpinvoker.AbstractHttpInvokerRequestExecutor.executeRequest(AbstractHttpInvokerRequestExecutor.java:137)
        at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.executeRequest(HttpInvokerClientInterceptor.java:202)
        at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.executeRequest(HttpInvokerClientInterceptor.java:184)
        at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.invoke(HttpInvokerClientInterceptor.java:150)
        ... 17 common frames omitted

Is there a reason why the 18.10.0 windows go-agent can connect through the load balancer fine, but not the 19.9.0 linux agents?

Thanks in advance.

Doug

[dle...@gmail.com]

 Turns out we *do* have one linux agent running 19.6.0 that is configured to point to the ELB, so I'm not sure what's going on.

The one host that is not working is the one where I'm running multiple agents on.  Would that make a difference? I'm going to need to do some more digging...

[dle...@gmail.com]

Please disregard - I got things working -- posting for help was a forcing function to dig in and compare configs of which I saw a few anomalies that had accumulated over years of neglect. This was after comparing it with a clean host that I configured with multiple go-agents that worked right from the start.

Cheers.

Doug