Last month, Consumer Reports came up with some interesting battery test results with the latest MacBook Pro models. Results ranged from 3.75 hours to 19.5 hours. Many in the Apple community thought that something was off. There was either a software bug or a benchmarking issue â it turns out it was both.
Apple worked with Consumer Reports over the holidays to find out why the battery test results varied so much. Itâs hard to get the exact same battery life every time. Thatâs why benchmark lovers donât run a test just once. They run it multiple times before finding the average result.
And yet, battery life shouldnât vary from 1x to 5x. Many early buyers have already been complaining about the battery life of the new MacBook Pro. Thatâs why Apple removed the âbattery time remainingâ estimate in a macOS update.
But it doesnât explain Consumer Reportsâ benchmark. It turns out that Consumer Reports turns off Safariâs local cache for their browsing test. It means that the browser will fetch a websiteâs data from the internet every time. When the cache is on, Safari stores images on your hard drive so that you donât have to load them up every time.
âConsumer Reports uses a hidden Safari setting for developing web sites which turns off the browser cache. This is not a setting used by customers and does not reflect real-world usage,â Apple said in a statement.
This seems deliberate, and Consumer Reports explains why in a new post today. The publication turns off the cache settings on the default browser for all laptops â not just Mac laptops. This way, itâs like Consumer Reports is browsing thousands of different websites, and not just the same ten websites over and over again.
In addition to this, Apple has also discovered that there was a bug in Safari affecting icon fetching. This bug has been fixed â this should improve benchmarks.
But it makes you think about Consumer Reportsâ methodology here. Using web browsing for battery benchmarks seems like a great idea. It imitates real life usage as most people use their computer to browse the internet, play music, write documents and spreadsheets and play movies. But deactivating default settings that positively affect battery life seems curious. Maybe Consumer Reports should consider leaving the cache on for future Mac and PC laptop benchmarks.
Apple sent the following statement to TechCrunch:
âWe appreciate the opportunity to work with Consumer Reports over the holidays to understand their battery test results. We learned that when testing battery life on Mac notebooks, Consumer Reports uses a hidden Safari setting for developing web sites which turns off the browser cache. This is not a setting used by customers and does not reflect real-world usage. Their use of this developer setting also triggered an obscure and intermittent bug reloading icons which created inconsistent results in their lab. After we asked Consumer Reports to run the same test using normal user settings, they told us their MacBook Pro systems consistently delivered the expected battery life. We have also fixed the bug uncovered in this test. This is the best pro notebook weâve ever made, we respect Consumer Reports and weâre glad they decided to revisit their findings on the MacBook Pro.â