We are switching to distributed cache cluster (memcached) which is located on intranet and managed by another team.
But the latency increased from ms to s.
It seems the network latency since the cache is not local to the application. This is concerning because we use the cache cluster to cache the ML models stored on network disk.
Wonder what I am missing here? Or I am doing some anti-pattern? What is the best practice of deployment of the distributed cache cluster? How to minimize the network latency?
Please advise. Thank you!