Have a read of:
There are different configurations mod_wsgi can be run as. In some (especially embedded mode), subsequent requests can be handled by different processes, and so until all processes have been primed with code, it will appear that it is being loaded very time.
You should ideally be using the recommended daemon mode configuration. This way it is set number of processes that persist, and so once loaded into the processes you are good to go.
What mode of mod_wsgi are you using? Can you show the mod_wsgi configuration?
Graham