Tomcat deployment, load balancers, and connection draining?

[Michael Martinek]

Greetings,

I'm looking to do a rotational deployment with my configuration. I've got 5 instances running that run Tomcat, and all five are in an instance group, and a backend service that a load balancer uses.

What I'd like to do is be able to do a send a deployment to Tomcat but gracefully remove servers from rotation until they start up successfully again. The idea is to keep anyone from getting a 500+ error while visiting the site during a deployment. Connection draining seems to affect the entire backend service (all five instances).

Essentially the script I'm writing queries the available webservers and splits it up into two groups based on percentage (meant to scale later). One group handles traffic while the second group is upgraded, Tomcat is restarted, and loops until they report a 200 or times out as a failed deployment. Then the first group is upgraded.

Absolute worst case, I figure I can add a firewall rule on the instance to forward the request to the web servers not undergoing upgrade yet.. then remove the firewall rule upon completion/failure. My hope is to be able to have it managed through the HTTP/HTTPS load balancer.

Has anyone done something similar or have any thoughts?

Thanks in advance,

Michael

[Faizan (Google Cloud Support)]

Hello Michael,

We do have a feature to Update Managed Instance Groups[1]. However, this feature is in Alpha phase and is not covered by any SLA. Also, to get access you need to request to whitelist your account.

Another way I can think of to achieve your setup is by following the steps below:

1. Create two backends(backend1 and backend2). Backend1 will be the main active group while backend2 is the new backup group. Set the initial serving capacity for backend2 to 0%[2].

2. Once Backend2 is healthy, set the their capacity to 100%, so it can start receiving requests, I'll suggest to wait for few minutes before backend2 start replying to incoming requests.

3. Disable autohealing and autoscaling for the instance group serving backend1(if enabled).

3. Drain the instances of the Backend1 by having them fail the health check by blocking traffic from the 130.211.0.0/22 IP range using IP tables[3].

4. Once all the active connections on Backend1 close naturally and all the instances are unhealthy, set the capacity for the Backend1 to 0% or remove the backend service.

I hope that helps.

Faizan

[1] https://cloud.google.com/compute/docs/instance-groups/updating-managed-instance-groups

[2] https://cloud.google.com/compute/docs/load-balancing/http/backend-service#backend_service_components 

[3] https://cloud.google.com/compute/docs/load-balancing/health-checks#handling_unhealthy_instances

[Michael Martinek]

These instances aren't managed so unfortunately that option is out. However, I do like the other idea and think that will work just fine. Thanks for your help!