You mentioned the LDAP authentication. The chance is quite high that the long query time is due to a slow LDAP system. Do you also face slow login times as user?
Do you know about the caching feature in the LDAP plugin? I suggest to have a look at it.