There has been an epidemic of cheating throughout the tech world. It stinks of desperation, and largely goes unpunished. Cheaters may or may not win, depending on how their attempt plays out. But they typically don’t lose anything due to their cheating.
The Verge notes, “Oppo is the latest company to get caught cheating on benchmark tests.”
“The phones were programmed to recognize the publicly available 3DMark app from the Google Play Store by name and then allocating system resources to ensure a better score. UL, the company that makes 3DMark, tested the devices again with a private version of 3DMark and found that the Find X and F7 were scoring [up] to 41 percent higher with the publicly available app than the private one, even though the benchmarking tests were identical.”
It is trivially easy to cheat on benchmark tests. It is unclear whether this is endemic to the industry –done all the time by everyone — or if only a handful ever get caught, giving a bad name to other brands. Last month, it was Huawei that was caught cheating benchmarks across multiple phones. The P20 and P20 Pro were among the guilty parties.
This practice has been going on for a long time. Samsung was caught doing the same thing as far back as the Galaxy S4. They made the news, as reported by ExtremeTech, when they stopped cheating.
It is not just smartphone manufacturers who are complicit in this cheating of the benchmarks. Intel was caught doing the same thing as far back as 2009. Wherever there is a specs battle between powerful competitors in the tech world, it is almost certain that at least one of the participants is cheating to produce better results.
The problem with cheating is not just a battle for CPU supremacy. Less than two weeks ago, Xiaomi was caught cheating at Mi 8’s DxOMark camera benchmark according to AndroGuider.
The notable absentee in the benchmark cheating hall of fame is Apple. They are extravagant in their claims of integrity. The photos that they use to show off the iPhone’s camera are not only from the phone, but ostensibly have not been retouched in any way. They make a point of repeating this mantra on stage at every event. The benchmarks that they post are replicated by unaffiliated testers from around the world.
At the end of the day, consumers should ignore benchmark claims. Firstly, there is enough general cheating within the benchmark community that there is no reason to trust any of them. Second, even when the benchmarks are honest, they only tell a small part of the performance story.
It is seldom the part of the story that will make a difference in the hands of the end user.