Agent is unable to communicate with server through load balancer

[Abhi Muthukumar]

Hi,

We have a GoCd Server running behind an aws load-balancer.  Our setup goes as below.

Domain: abc.xyz.cloud

Route53 record: abc.xyz.cloud --> Load balancer

Load Balancer : any (HTTP) requests to abc.xyz.cloud at port 80 will be forwarded to port 443 and any (HTTPS) requests to abc.xyz.cloud at 443 will be forwarded to gocd-server-target-group

Target Group: gocd-server-target-group uses HTTP protocol to hit our gocd-server ec2-instance on port 8153 (as it is a HTTP request)

GoCd Agents: There are 2 gocd agents running in private subnet and tries to connect with server using the server url - https://abc.xyz.cloud/go

Load Balancer takes care of stripping the certificate and makes HTTP request to go server

I am able to launch gocd server through browser and view my pipelines.  From agent, I'm able to curl server's health end point.  But I'm not able to see the agents in UI and I see the blow error in go-agent log.

2019-05-15 13:32:25,120 ERROR [scheduler-2] AgentHTTPClientController:100 - Error occurred when agent tried to ping server:
org.springframework.remoting.RemoteAccessException: Could not access HTTP invoker remote service at [https://ci.receptiviti.cloud/go/remoting/remoteBuildRepository]; nested exception is org.apache.http.client.ClientProtocolException: The server returned status code 403. Possible reasons include:
   - This agent has been deleted from the configuration
   - This agent is pending approval
   - There is possibly a reverse proxy (or load balancer) that is terminating SSL. Hint: use port 8154 of the GoCD server. See https://docs.gocd.org/19.3.0/installation/configure-reverse-proxy.html#agents-and-reverse-proxies for details.
    at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.convertHttpInvokerAccessException(HttpInvokerClientInterceptor.java:226)
    at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.invoke(HttpInvokerClientInterceptor.java:153)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
    at com.sun.proxy.$Proxy8.ping(Unknown Source)
    at com.thoughtworks.go.agent.AgentHTTPClientController.ping(AgentHTTPClientController.java:95)
    at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)
    at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.http.client.ClientProtocolException: The server returned status code 403. Possible reasons include:
   - This agent has been deleted from the configuration
   - This agent is pending approval
   - There is possibly a reverse proxy (or load balancer) that is terminating SSL. Hint: use port 8154 of the GoCD server. See https://docs.gocd.org/19.3.0/installation/configure-reverse-proxy.html#agents-and-reverse-proxies for details.
    at com.thoughtworks.go.agent.GoHttpClientHttpInvokerRequestExecutor.validateResponse(GoHttpClientHttpInvokerRequestExecutor.java:103)
    at com.thoughtworks.go.agent.GoHttpClientHttpInvokerRequestExecutor.doExecuteRequest(GoHttpClientHttpInvokerRequestExecutor.java:70)
    at org.springframework.remoting.httpinvoker.AbstractHttpInvokerRequestExecutor.executeRequest(AbstractHttpInvokerRequestExecutor.java:137)
    at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.executeRequest(HttpInvokerClientInterceptor.java:202)
    at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.executeRequest(HttpInvokerClientInterceptor.java:184)
    at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.invoke(HttpInvokerClientInterceptor.java:150)
    ... 16 common frames omitted

Could someone please help on what is going wrong with this setup?

[Jason Smyth]

Hello,

I think this section of the stack trace you posted explains what is happening in your situation:

   - There is possibly a reverse proxy (or load balancer) that is terminating SSL. Hint: use port 8154 of the GoCD server. See https://docs.gocd.org/19.3.0/installation/configure-reverse-proxy.html#agents-and-reverse-proxies for details.

From the GoCD documentation section linked in the stack strace:

 The GoCD server requires that the agents connect to it directly without any reverse-proxies in between that perform SSL termination. This is because GoCD agent-server communication is authenticated using SSL/TLS client certificates, a reverse-proxy will be interpreted as a MITM (man-in-the-middle-attack) and the agents will not be able to connect to the server.

I suspect you will need to set up a separate AWS Load-Balancer that does not do SSL termination but rather forwards port 443 to port 8154 on the GoCD server.

Cheers,

Jason