Agents can't connect after setting up SSL

[Vasco Figueira]

Hi,

We've set up SSL with a certificate of our own CA and all is fine except for agents that keep getting:

2014-10-31 12:02:03,684 [pingThread] ERROR thoughtworks.go.agent.AgentController:107 - Error occurred when agent tried to ping server:

org.springframework.remoting.RemoteAccessException: Could not access HTTP invoker remote service at [https://cd.our.company.internal.domain:8154/go/remoting/remoteBuildRepository]; nested exception is org.apache.commons.httpclient.HttpException: Did not receive successful HTTP response: status code = 403, status message = [Access Denied]

at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.convertHttpInvokerAccessException(HttpInvokerClientInterceptor.java:212)

at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.invoke(HttpInvokerClientInterceptor.java:145)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)

at com.sun.proxy.$Proxy5.ping(Unknown Source)

at com.thoughtworks.go.agent.AgentController.ping(AgentController.java:103)

at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:273)

at org.springframework.scheduling.support.MethodInvokingRunnable.run(MethodInvokingRunnable.java:65)

at org.springframework.scheduling.timer.DelegatingTimerTask.run(DelegatingTimerTask.java:70)

at java.util.TimerThread.mainLoop(Timer.java:555)

at java.util.TimerThread.run(Timer.java:505)

Caused by: org.apache.commons.httpclient.HttpException: Did not receive successful HTTP response: status code = 403, status message = [Access Denied]

This appears to be the agent java process not trusting the new certificate. I remember seeing somewhere a file named agentTruststore but I don't seem to find it (I hope I didn't dream that). We added the truststore with the new certificate to the server's /etc/go folder alongside the keystore. Is this truststore that the server gives to agents?

Thank you.

Best regards,

Vasco

[Aravind SV]

Hello Vasco,

Did you follow all the steps in  this post? http://www.go.cd/2014/06/05/using-go-cd-with-custom-certificates.html

If you've done all that, it should work.

Cheers,

Aravind

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Vasco Figueira]

Hi Aravind,

Yes I did. I I do have trusted https on the server working.

I did have an issue after following the steps mentioned therein while restarting the server, saying that it couldn't find the truststore (not the keystore) under /etc/go. Even though that wasn't documented I added a truststore with the root certificate there as well and that solved it.

Maybe that provides a hint?

I'll continue troubleshooting this, i.e. verifying the keystore and truststore are congruent.

Thank you.

Vasco

[Vasco Figueira]

I tried adding the root certificate to the server's keystore in the hope that was what is getting sent to the agents. It would explain the server requiring a truststore and the agents not having a way to establish trust.

But it does not help. Even if the keystore contains the root certificate (from the signing key) the server does not boot up if there is no /etc/go/truststore (is this documented anywhere?) and the agents continue failing to connect.

[Aravind SV]

This is the closest you can get to documentation, I believe: http://www.go.cd/documentation/developer/4/4.3.html#4-3-4-3-registration

From a 403, it looks like the server denied the request, which could mean that the registration did not go through. Is there any error in the server logs?

When an agent registers, a new key is generated for the communication, and it is put into the keystore. Then, the certificate of that is put into the agent trust store. Every agent has to then use that key, for further communication.

One thing I'm wondering is, the external communication (for a browser) can be made to use your own certificate, as you have done. But, the agents will continue to use port 8154 (as you can see from the stack trace you provided). So, maybe something to do with that?

[Vasco Figueira]

Hi Aravind,

Please find attached the log of an agent spinning up afresh and trying to connect to the server.

The server is up and running with a certificate signed by an authority of ours whose root certificate is both in /etc/go/keystore and /etc/go/truststore.

The first ERROR line is an intriguing one:

2014-11-05 16:28:49,204 [loopThread] ERROR thoughtworks.go.agent.AgentController:136 - [Agent Loop] Error occurred during loop:

org.springframework.remoting.RemoteAccessException: Could not access HTTP invoker remote service at [https://cd.core.platforms.gamesys.corp:8154/go/remoting/remoteBuildRepository]; 

Is Spring expecting somewhere an http service instead of an https one?

If I try access https://cd.core.platforms.gamesys.corp:8154/go/remoting/** I also get a 403 even after authenticating with a trusted https connection.

What am I not getting here?

Thanks in advance for any help.

Yours,

Vasco