Do you have any anti-malware programs running that intercept the email
connections for filtering purposes? I found that some of them time out
or give faulty connections.
They usually intercept the email client by setting up domain redirects
in the %systemroot%system32/drivers/etc/hosts file and creating a
proxy mail server listening on 127.0.0.1. You would think that it
would be a perfect solution but I think what happens is the proxy
accepts the authentication but when the real server is slow setting up
the connection to the proxy's network side and it times out or
otherwise fails in some way and Agent sees it as a server failure in a
delayed fashion.
I don't have a solution for it but that's my theory.
If you do have mail malware filtration as described, try temporarily
turning it off for a time and see if Agent likes it better.
FWIW, I run Agent barefoot and get occasional server failures without
any proxy involved. It just happens, servers get busy or spammed and
get slow or whatever servers do.