Improve your Google rank by speeding up your website
It matters because Google is judging page speed as a ranking factor and because users want fast loading pages. Google measures your page speed from the the time it is called to when the page is initially loaded. This means you want to get to the page load event as quickly as possible. That initial page load time is what Google is using to judge your webpage (and let’s not forget your users waiting for it to load). Google actively promotes and recommends prioritizing above the fold content so getting any resources at all (js, css, images, etc.) out of the critical rendering path is well worth the effort. If it makes your users happy, and it makes Google happy, you should probably do it.