If everything is closed off by a firewall you can consider it protected. However, I use a Mozilla mail client on my laptop to check certain mailboxes (my laptop is outside our server firewall), and for that I need to use secure POP3. It's mainly a development tool, but in a production setup, it can be useful for troubleshooting. I have test accounts for troubleshooting so I don't accidentally download PHI to my Mozilla mail client. Even without PHI, I use secure POP3/SMTP to protect the auth credentials.
-tom