Is SSL connection between Go agents and Go Server possible

[Cristi Campean]

Hi,

Is there any way to make the connection between the Go Agents and the Go Server on https?

We've tried to configure an Agent to connect on the Server's secure port but we got this error:

After changing the port in /etc/default/go-agent:

GO_SERVER_PORT=8154

We got this error:

108 [main] ERROR com.thoughtworks.go.agent.launcher.ServerCall  - Couldn't access Go Server with base url: http://go_server_host:8154/go/admin/agent-launcher.jar: org.apache.commons.httpclient.NoHttpResponseException: The server go_server_host failed to respond
java.lang.Exception: Couldn't access Go Server with base url: http://go_server_host:8154/go/admin/agent-launcher.jar: org.apache.commons.httpclient.NoHttpResponseException: The server go_server_host failed to respond
        at com.thoughtworks.go.agent.launcher.ServerCall.invoke(ServerCall.java:70)
        at com.thoughtworks.go.agent.launcher.ServerBinaryDownloader.headers(ServerBinaryDownloader.java:128)
        at com.thoughtworks.go.agent.launcher.ServerBinaryDownloader.downloadIfNecessary(ServerBinaryDownloader.java:105)
        at com.thoughtworks.cruise.agent.launcher.AgentLauncherImpl.launch(AgentLauncherImpl.java:78)
        at com.thoughtworks.go.agent.bootstrapper.AgentBootstrapper.go(AgentBootstrapper.java:74)
        at com.thoughtworks.go.agent.bootstrapper.AgentBootstrapper.main(AgentBootstrapper.java:56)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at com.simontuffs.onejar.Boot.run(Boot.java:306)
        at com.simontuffs.onejar.Boot.main(Boot.java:159)
Caused by: org.apache.commons.httpclient.NoHttpResponseException: The server go_server_host failed to respond
        at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1835)
        at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1590)
        at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:995)
        at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:397)
        at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170)
        at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
        at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:324)
        at com.thoughtworks.go.agent.launcher.ServerCall.invoke(ServerCall.java:49)
        ... 11 more

We've also tried to force the schema of the URL in /etc/default/go-agent like this:

GO_SERVER=https://go_server_host
GO_SERVER_PORT=8154

but we get this error:

54847 [main] ERROR com.thoughtworks.go.agent.launcher.ServerCall  - Couldn't access Go Server with base url: http://https//go_server_host:8154/go/admin/agent-launcher.jar: java.net.UnknownHostException: https
java.lang.Exception: Couldn't access Go Server with base url: http://https//go_server_host:8154/go/admin/agent-launcher.jar: java.net.UnknownHostException: https

It is a security requirement for us to connect the Go Agent to the server on https.

Thanks,
Cristi

[Sharanya Bathey]

Hi Cristi,

When a Go Agent comes up for the first time it needs to register itself with the Go server, up until the registration happens the communication between them happens over HTTP, post this all the communication between the agent and server happens over HTTPS. 

Regards,

Sharanya

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Cristi Campean]

Hi Sharanya,

Thanks, this are great news.

Is there any way to configure the HTTPS port on which the Agents should connect to the Server.

We want to run the Go Server on 80 and 443.

In /etc/default/go-agent: I can only see GO_SERVER_PORT=8153

We would like to set GO_SERVER_PORT to 80 and the secure one to 443.

Thanks,
Cristi

[Jyoti Singh]

Hi Christi,

There might be other services running on the default ports 80 and 443, hence Go runs by default on 8153 & 8154. However, if you really want to change the ports back to the default ones, or to any other port, changes are required at both server and agent side. 

As pointed out by Sharanya, up until registration, the communication between the agent & server happens over HTTP, post which the server would switch the communication over to a secured channel. Hence, setting the ssl port (GO_SERVER_SSL_PORT) on server should suffice.

The changes required for this are below:

in /etc/default/go-server :

GO_SERVER_PORT=<desired port number>
export GO_SERVER_PORT
GO_SERVER_SSL_PORT=<desired ssl port number>
export GO_SERVER_SSL_PORT

in /etc/default/go-agent:

GO_SERVER_PORT=<desired port number>
export GO_SERVER_PORT

Cheers,

Jyoti

[Jyoti Singh]

Sorry, I missed mentioning that since "go" user account under which Go server runs, is a restricted account, you might face permission issues while trying to run the server on port 80 & 443. 

You might have to run the process as "root". 

You have another option of letting go-server running on default ports and fronting it with apache, check documentation for more details.

Thanks Sachin for pointing it out.

[Cristi Campean]

Hi Jyoti,

Thanks for sorting this out.

One more question:

Actually our plan was to place a load balancer in front of GO server which will listen on 80 and 443 and to forward the requests to the server on 8080 and 8443 which is pretty standard set up in our organization.

From what I understand, if there is no way to configure the SSL port on agents, we will have to also configure our load balancer to listen on 8443 and to forward to the server on 8443, so that the server will be able to switch to a secure connection after registration.

Is that correct?

Note: we want to put a load balancer in front of the server because we want to have some sort of failover solution (if the active server crashes, just put the back-ups on the passive one and enable it on the load balancer after disabling the active one)

Thanks,
Cristi

[Jyoti Singh]

Hi Christi,

Just to be clear, Go does not support load-balancing. Meaning you cannot have two Go servers running in parallel and sharing work.

What you are speaking about is handling failover scenarios. While a live failover is not supported in Go, a solution to handle non-live failover could be worked out. 

There could be two ways to achieve failover 

- using a web-server or load-balancer fronting Go server

- DNS based switch

Fronting Go server with a web-server (or a load-balancer in your case) could help achieve failover as far as dashboard is concerned, however, the agents would continue to use https://<go-server>:8443 url, which would skip the load-balancer altogether. To be honest, I haven't tried this option so cannot tell you a workaround yet.

The other option is to use a DNS based switch. So, when primary server goes down, get the required files from backup and bring up the secondary server followed by a DNS switch. This would work well for the dashboard as well as for the agents.

Cheers,

Jyoti

[Cristi Campean]

Hi Jyoti,

I know that go doesn't support load-balancing (unfortunately) so our use case would be to easily switch between active and passive instances.

We've chose load-balancing instead of DNS switch because it's faster to change boxes behind a LB than to propagate a DNS change.

"the agents would continue to use https://<go-server>:8443 url, which would skip the load-balancer altogether. To be honest, I haven't tried this option so cannot tell you a workaround yet"

By this you mean that even though I'll put in the agents config the <server_Load_Balanced_URL> and port 80 (LB port) when the protocol will change to secure the call will be made on the server address <server_host> 8443 instead of <server_Load_Balancer_URL> 8443?

Thanks,
Cristi

[Jyoti Singh]

Hi Cristi,

>> By this you mean that even though I'll put in the agents config the <server_Load_Balanced_URL> and port 80 (LB port) when the protocol will change to secure the call will be made on the server address <server_host> 8443 instead of <server_Load_Balancer_URL> 8443?

In theory, if you do the above, the call should go to <server_Load_Balancer_URL> 8443 provided you update the go server url in /etc/default/go-agent to point to that of load balancer and have your load-balancer listening on 8443 as well. I can't think of a reason as to why should it not work. However, as I said earlier, I haven't tried this out myself or spoken to anyone who has. 

Since DNS switch option does not work for you, may be you could give this a try and share what you find.

Cheers,

Jyoti

[Cristi Campean]

Thanks Jyoti,

We'll definitely try it and will get back with the results.

Cristi

[Cristi Campean]

Hi,

We've tried this and it's working.

We have 2 servers, one running and one stopped and in front of them is a LB configured to fw the requests on 8443 to the server on 8443.

We configured the agents to connect to the LB on 80, which is forwarded to the server on 8080.

Thanks for your help Jyoti and Sharanya.

Cristi

[Jyoti Singh]

Hey Cristi,

Thats good news! Glad that you could get it working.

It would be great if you could write a blog post detailing the steps you followed to achieve this and share with the community.

Cheers,

Jyoti

[Cristi Campean]

Hi Jyoti,

By blog post you mean a blog post on the official go blog (http://www.go.cd/blog.html) or on my own blog (I don't have one :) )

If it's the first one, than sure, I can do that, but I'll need some details about the process.

If it's the second one, than there are some issues since I don't have a blog yet and I'm not sure if I'll have one in the near future. :)

Thanks,
Cristi

[Ken Mugrage]

Hello Cristi,

Anyone can submit a blog post for inclusion on http://www.go.cd/blog.html. To do so, fork the repo at http://www.go.cd/blog.html, add your post to the _posts directory and then submit a pull request. The site uses GitHub Pages with Jekyll. You can find more information on that at https://help.github.com/categories/20/articles

Pretty easy, but do post if you have questions.

Ken

--

You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Ken MugrageOpen Source Program Manager

Email

kmug...@thoughtworks.com

Telephone

+1 425 835 2056

Twitter

@kmugrage

[Sachin Sudheendra]

Correction:

To do so, fork the repo at http://www.go.cd/blog.html,

 

To fork the blog repository, you need to fork https://github.com/gocd/gocd.github.io

Sachin Sudheendra

[Ken Mugrage]

heh, typing before coffee. Thanks Sachin.

>> *Ken Mugrage**Open Source Program Manager*
>> Emailk...@thoughtworks.com
>> Telephone+1 425 835 2056
>> Twitter@kmugrage
>>
>> [image: ThoughtWorks] <http://www.thoughtworks.com/>

>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "go-cd" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to go-cd+un...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> --
> Sachin Sudheendra
>

--
*Ken Mugrage**Open Source Program Manager*
Emailk...@thoughtworks.com
Telephone+1 425 835 2056
Twitter@kmugrage

[image: ThoughtWorks] <http://www.thoughtworks.com/>

[stephen...@gmail.com]

Sharanya, that's incorrect. You can verify through a packet cap that the only things exchanged over HTTP/8153 are the following in HEAD and GET requests:

/go/admin/agent-launcher.jar

/go/admin/agent (returns an unnamed jar)

/go/admin/agent-plugins.zip

Everything thereafter (and especially registration) is through port 8154 and is encrypted... which is actually good, since the Go server sends a *private* key to the agent, rather than merely signing/returning a public cert request uploaded by the agent.

For deeper information look at the file com/thoughtworks/go/agent/service/SslInfrastructureService.java, which is where most of the agent's side of the actual registration takes place.

[Rustin Daniels]

Hi Cristi,

I've bee thinking about high availability with go for a while now and have come to the conclusion that its not viable solution for a medium to large production deployment build and deployment platforms.

I have a few questions if you don't mind.

1. Did you have to configured all your pipelines on both servers.

2. What is the topology of your production system? vlan, subnet, firewalls, dmz, gateways etc. 

3. Is the communication on secure infrastructure and is your agents highly distributed.

4. With regard to the servers, how do automatically sync the go configs across servers and their artifacts repository. 

5. Since Go uses the in-memory dh2db what effect does having two separate servers have on stage history, internals of database data like configuration,pipelines, oauth etc?

6. When having 2 servers polling upstream dependency repos you end up with twice the amount of network traffic.(sockets are on the backlog i think) this multiplied by the number of pipelines you run. I've ended up with well over million requests per day to single repo from single go server with each request responding with fairly large response payload (json=>(html))

7. You will end up having twice the storage overhead, backups etc. Eg. I have fairly large tools folder and close to 50 component build pipelines for a single project and one total cycle of all pipelines takes up to 30 gigs of disk space in matter of minutes. (I would however like to see this be changed in the future to save a substantial amount of disk space on an agent. Something along the lines of.. if the previous run checksum for an upstream dependency are equal use the previous and automatically delete the upstream artifacts in the currently executing agent run folder).

8. Lastly, why would you need ssl for agent<->server communication? Are you not running production infrastructure on private network or subnet? I would think its more important having go server and upstream repos running ssl. I can't see importance of ssl if you running go and agents in a segregated subnet or vlan even running in the cloud sometimes more secure that some enterprise networks whether on ec2 or gce firewall/routing rules should go job to securing your traffic. 

The only way i can think of an intruder could trace your traffic on build and deployment infrastructure would be if his sat next to and plugged into the router while mirroring the ports tracing with wireshark and waiting for agents to come online, which is very unlikely in the enterprise.

That's it.

Thanks for good topic and my thanks to everyone who worked on such a great product.

[Cristi Campean]

Hi Rustin,

Sorry for the late response but I was a bit busy with other stuff.

I am planning to write a blog post about all this, but in the meantime please find my answers below:

1. Did you have to configured all your pipelines on both servers.

No, only one server is running at a time, the other one is a passive one (is stopped) and the plan with it is to bring it up with the latest back-ups from the first one (not tested yet)

2. What is the topology of your production system? vlan, subnet, firewalls, dmz, gateways etc.

3. Is the communication on secure infrastructure and is your agents highly distributed.

Yes, we have the 2 servers behind a load balancer and 2 clusters of agents (1 for DEV (with 20 agents) and 1 for PRODUCTION environment (with 10 agents)) talking to the servers through the load balancer on HTTPS.
Only one server behind the load balancer is active.

4. With regard to the servers, how do automatically sync the go configs across servers and their artifacts repository.

We don't. We only took back-ups from the active server every midnight and in case of an emergency put the back-ups on the passive server and bring it up.

5. Since Go uses the in-memory dh2db what effect does having two separate servers have on stage history, internals of database data like configuration,pipelines, oauth etc?

This is also back-ed up to my knowledge, but I haven't tried it yet.

6. When having 2 servers polling upstream dependency repos you end up with twice the amount of network traffic.(sockets are on the backlog i think) this multiplied by the number of pipelines you run. I've ended up with well over million requests per day to single repo from single go server with each request responding with fairly large response payload (json=>(html))

Only one server is running.

7. You will end up having twice the storage overhead, backups etc. Eg. I have fairly large tools folder and close to 50 component build pipelines for a single project and one total cycle of all pipelines takes up to 30 gigs of disk space in matter of minutes. (I would however like to see this be changed in the future to save a substantial amount of disk space on an agent. Something along the lines of.. if the previous run checksum for an upstream dependency are equal use the previous and automatically delete the upstream artifacts in the currently executing agent run folder).

Only one server is running.

8. Lastly, why would you need ssl for agent<->server communication? Are you not running production infrastructure on private network or subnet? I would think its more important having go server and upstream repos running ssl. I can't see importance of ssl if you running go and agents in a segregated subnet or vlan even running in the cloud sometimes more secure that some enterprise networks whether on ec2 or gce firewall/routing rules should go job to securing your traffic.
The only way i can think of an intruder could trace your traffic on build and deployment infrastructure would be if his sat next to and plugged into the router while mirroring the ports tracing with wireshark and waiting for agents to come online, which is very unlikely in the enterprise.

That's true, for us it was a question raised by the security team. I think they were worried that someone with bad intentions might inject malicious commands to the agents and might mess up the PROD environment.

I hope this helped.

Thanks,
Cristi

[Rustin Daniels]

Hi Cristi,

Sounds great, I'm really looking forward to reading your article.

Cheers

Rustin