Backgrounder - Our device connects to an AWS static IP. We use dnsmasq on device to provide lookup services for downstream devices. Currently we are planning to use a long. DNS TTL on AWS to avoid too many DNS lookups from on field devices. The on-field devices use a grpc connection to maintain long standing tcp connections. We do have multiple availability zones and so a DNS resolution does return 4 IP addresses
Problem - When an IP address fails(on AWS) the grpc client will retry and re-resolve. But because we have dnsmasq on device it will send a cached address - which is potentially faulty.
Solution - This can be resolved by flushing the dnsmasq cache on device. But is there a way to flush the dnsmasq cache on device on connection failure only? grpc under the hood uses c-ares which in our case goes to the dnsmasq proxy on device.
Any solutions/thoughts. Someone must have encountered this problem before?