Potential issue with leases

[Frantisek Svoboda]

Hello,

this may be a stupid question but it would really help me if anyone could confirm my observations are correct. I am encountering a strange behavior running etcd server version 3.3.8 and go client github.com/coreos/etcd v3.3.25+incompatible. It seems that some keys get randomly deleted over time. The etcd client uses KeepAliveOnce on leases attached to those keys and according to the application logs, it does so frequently enough to make sure the keys are preserved, however, it seems that the data sometimes simply disappear (the subsequent get request returns 0 key-values).

Is it possible that this issue is related to the following commit and that upgrading the client version would help?
https://github.com/etcd-io/etcd/pull/14357/commits/295044fba22837beb65a8b0481f2327fe019a4fe

BTW I do see the following messages when running etcd with debug logs enabled in this setup:
2024-07-31 15:38:18.750060 D | etcdserver/api/v3rpc: failed to receive lease keepalive request from gRPC stream ("rpc error: code = Canceled desc = context canceled")

Thanks in advance for any response, regards,

Frantisek Svoboda