Is it possible for multiple GoCD servers to use the same database?

[Lance Nehring]

I'm wanting to deploy a GoCD server into 2 different K8S clusters and have them share a database.     My goal is to be able performance maintenance on one K8S cluster without impacting the developers.

SO the question is: can multiple GoCD servers share a database without interfering with each other?

Thanks.

[Chad Wilson]

Unfortunately not if they are both running at the same time; no. GoCD does not support active-active configurations like that.

If they otherwise share DB and storage and have the same identity, but one deployment is scaled to 0 and the other scaled to 1 replica, that is theoretically possible - but you'd need to co-ordinate very carefully, and there'd still be brief downtime when you switched instances; similar to if you just moved the server from one node to another within a single cluster while doing "maintenance"; so in my personal view not worthwhile.

-Chad

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/448aff6b-4bbf-4a5f-a3a2-3289a228dfe6n%40googlegroups.com.

[Sriram Narayanan]

On Thu, Mar 7, 2024 at 10:21 PM Lance Nehring <lance....@gmail.com> wrote:

I'm wanting to deploy a GoCD server into 2 different K8S clusters and have them share a database.     My goal is to be able performance maintenance on one K8S cluster without impacting the developers.

SO the question is: can multiple GoCD servers share a database without interfering with each other?

The two instances will have a unique server ID and should not point to the same database. Please do not attempt this even by making them share the same identity since they will be writing to the database.

For higher availability, you'll need to think more about the agents connecting to the server on the second K8s cluster. Agents can continue with their job when a server is unavailable and can reconnect to the server once it is back online.

You can leverage this behaviour to achieve some form of HA by:

- keeping postgres external to the cluster

- stopping the GoCD server on the first cluster

- starting the GoCD server with the same configuration on the second cluster

- either setting up an LB in front of the first with a fail over to the  second K8s cluster 

OR setting up DNS round robin to refer to the second K8s

OR setting a really low TTL for the DNS name and making it point to the second cluster for a switch over.

[Lance Nehring]

Thank you.  The steps you are describing is almost exactly what I do today.  However, I think GoCD could see more widespread use, if it were possible to run multiple servers simultaneously.  If I had time, I'd try to help with such a feature.  Even with this limitation, I'd never go back to Jenkins.

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CANiY96basJe4Jiteh-GNsX%2BF5OJzDuCh82Dg-nBLmS%3DsPY7DiA%40mail.gmail.com.

[Chad Wilson]

Unfortunately moving to an active-active state would be a huge architectural change for GoCD, and unfortunately not consistent with its original design - let alone its current maintainer/community state (alive, but not thriving).

Previously, GoCD had a commercial "business continuity" extension to make "active-passive" setups easier to automate and reduce "switchover" time. This was a commercial extension later integrated into the core of GoCD during its "full" open sourcing in 2020, however post-integration it had a critical security defect, and was disabled and later removed due to lack of interest/usage.

My current recommendation is generally to use the features of Kubernetes/container orchestration, load balancers and/or cloud providers to have an external database and both "shared" artfact file/cloud storage (AWS EBS, EFS and the like) and database in a way that makes switching the location the server runs as fast as possible, however making this even more seamless could also do with some work.

In an ideal state of seamless active/passive switches, GoCD would have a mode to be able to automatically turn on maintenance mode during shutdown, gracefully shutdown after n-seconds (even if jobs are still running on some agents), then start in an alternate location with limited human intervention (utilising the ability for agents to continue in the background without a server temporarily).

-Chad

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAMj%3DmE8akBXupyv4%2B9HbXt-n%3DFZaBfYOi%2B8jJx4hS5Ochnq4QA%40mail.gmail.com.