health status while decommissioning

[Andrei Matei]

Hey load balancing friends,

I'd like to change the health checking behavior for a node going through the decommissioning process. At the moment, the /health?ready=1 endpoint (not /health) will return an error while the node is draining or decommissioning. I'd like to change it to only return an error while draining, not while decommissioning, thus letting load balancers that care continue to use a decommissioning node. Wondering if anyone thinks otherwise.

The decommissioning process can be arbitrarily long and it ends with the node draining too. Technically, the server doesn't really have a decommissioning process. The cockroach quit --decommission client comes along, the server sets a flag in its liveness record and then lets the replication queues and allocators across the cluster do their thing and move replicas away. Meanwhile, the client continuously pools the server's status until there's no more replicas left, at which point it goes to the regular cockroach quit path, which does the draining.

While decommissioning, the respective server might still hold leases, etc. You can also restart the node, and it stays in this decommissioning state forever (until you recommission it, which is a thing). So, I don't see a particular reason to claim that it isn't "ready". Additionally:

- we have the server.shutdown.drain_wait cluster setting which causes the server to declare itself unready for this long before even starting the draining process. This was added specifically to give balancers plenty of time to find out about the server going away.
- I hear that some people have figured out that a decommissioning node can serve as a SQL-only gateway. I'm not saying that we should particularly support such out of the box thinking, but I also don't think we should artificially work against it through this health check behavior.

Besides this seeming the right thing to me, the less code needs to look at the decommissioning flag, the better.

The unready on decommissioning behavior was added in 22911. The review had no discussion about the decommissioning case specifically.

Thanks for reading!

- a_m

[Alfonso Subiotto Marques]

Your proposed change makes sense to me. I think that PR overlooked the case that a decommissioning node could still be healthy.

--

Alfonso