Failover, HotStandby of GoCD

[Supreme Overlord]

Hey Guys,

We are currently using GoCD 19.2. It's running in one region with multiple agents. It is using the H2DB format and not a relational database.

I want to provide a hot-standby in another region. My question is:

1. What is the best way to go about this?

2. What files, folders and other necessary objects need to be copied from GOCD Server 1 in region1 to GoCD Server 2 in region2.

I will appreciate any assistance even if it's recommended we go the relational dbase route.

Best,

SupremeOverlord

[Supreme Overlord]

Any pointers guys?

[Ashwanth Kumar]

I don't know what kind of workloads you're running or plan on running on GoCD that require a Hot Standby. But here are some of my notes if you want to setup a quick failover for GoCD.

The Easy Way

You should look into this addon from Thoughtworks. I have not personally used that addon so I'll wait for someone else in the group to share their experience with it. 

The Hard Way

Usually I would start with the latest copy of the backup. It contains the following (at a high level):

1. DB Backup (assuming you're using H2DB)
2. config.xml backup (contains the entire pipelines, references to packages or pipeline as code references, etc.)

Another artifact (pun intended) for switching GoCD instances is the server side artifacts for all the pipelines so your VSM works as intended across all the pipelines. If you're on AWS (for example), doing something like `aws s3 sync` with S3 or using EFS (shared network filesystem) for the artifacts storage would solve that problem. Also make sure you're running the same version of GoCD server to avoid any DB migrations related issues. (Shameless plug, if you're worried about the cost implications of storing lot of artifacts check out GoCD Janitor).

PS: I personally have not tried this on a failover scenario, but have used the above technique to migrate from one GoCD instance to an another instance especially when migrating EC2 instances. 

Caveats (must read):

1. Since we're dealing with last good backup, you're most likely to be behind on the latest state for the pipelines. (Few pipelines could have run since the last backup). You need to be wary of the consequences of such runs on your infrastructure or whatever workloads the pipelines run. If the operation is not idempotent it might have undesirable effects. This is applicable on both CI and CD pipelines in my experience. 
2. Extension of this DB state issue is, As soon as you launch the new GoCD Server instance -- a lot of pipelines might start firing because of latest commits or changes in the materials across the board. This is usually not a problem unless you've some time critical workloads that needs to run very often. 
3. Final issue is with new pipelines created since the backup might be missing and needs to be recreated.
4. Almost all the issues could be addressed if you're using an external DB like Postgres. The config.xml might still be old so you might not get the latest created pipelines even with Postgres. I don't know if this behaviour is changed in the recent versions of GoCD.

GoCD is not built for hot failover. You can mostly optimize your MTTR with the above setup. 

Thanks,

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/54a094e8-1cf1-4988-9cb0-79d27f586e83n%40googlegroups.com.