Seeing all Bazel remote operations timeout after 21 second timeouts

[carpen...@gmail.com]

This is with bazel 6.2. I have just looked at a chrome:\\tracing log for a successful but very slow build. What I see is after a point in the build every remote http cache operation times out after 21 seconds. 

I see that it all starts with a BulkTransfer error:

[2023-10-09T03:04:19.615Z] stderr: WARNING: Remote Cache: 2 errors during bulk transfer:
[2023-10-09T03:04:19.615Z] stderr: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection timed out: no further information: xxx/1.2.3.4:443

The after that bazel starts compiling and attempting to upload every operation. 

But every remote operation takes 21 seconds to within half a second. I see a few more instances in of the BulkTransfer but where near enough to account for every slow upload/download operation.

What is the intent of bazel in this mode? 

Why is every remote cache taking 21 seconds?

Why doesn't it back off all remote_cache operations?

The 21 seconds doesn't match the remote timeout as our remote timeout is set to 60 seconds, so my assumption is this comes from TCP.