Here is what i got answered in a webmaster.stackexchange thread. Sounds very interesting:
There are only two page load speed metrics that really matter:
HTML delivery time is important because it is the factor that Googlebot most directly sees. It controls how quickly Googlebot is able to crawl your site. Algorithmic ranking penalties that Google applies are almost all based on this metric.
Time until the the page is usable is important for users. Users turn away from a site that isn't quickly usable. This has indirect SEO consequences because Google notices when users are not satisfied with a site.
So focus on users. They are much more picky than Google. You have three seconds. That breaks down to:
Since TTFB is a component of HTML page delivery, you need to optimize it to be a portion of the one second allotted for HTML delivery. Get it down to 200 to 500 ms.
All assets that are not needed for the page to start working should be lazy loaded. Most JavaScript should be loaded asynchronously. Images below the fold should have their load deferred.
With this in mind, the PageSpeed Insights score can be completely ignored. I do not believe that Google uses this score directly in rankings at all. The tool and the score can be useful. It can tell you which optimizations might be available to you. It may be able to help you prioritize which optimizations to perform. However it is a mistake to pay attention to the score only. Instead focus on making the site fast for users as your primary goal.
Also keep in mind that items beyond your control will effect your PageSpeed Insights score. I have a site that fully loads its pages in 1.2 seconds and gets a 100 for its score. However, when I enable AdSense on the page, PageSpeed insights reports that it takes 10 seconds to fully load and the score goes down to 63. This is despite the fact that the base page is fully usable after 1.2 seconds and the ads lazy load afterwards.