It might be due to
caching, i.e. when you're opening the page "manually" your browser doesn't download
embedded resources (images, scripts, styles, fonts, sounds, etc.) and it normally takes >90% of all the content.
Try running your browser in
incognito mode, you should see similar response times.
Selenium by default uses "clean" session, however it's possible to specify
profile directory if you want to test specific scenarios like returning users.