Android phones cheating in benchmarks??

[htsoxznese]

My friend says benchmarks shouldn't be use because many phones cheat in benchmarking by setting the software to run faster when benchmark apps are detected. But how is this cheating? I call this "stepping up to the challenge." When you run a marathon, you don't walk. You run. When you go to a job interview, you don't under perform. You outperform. Same with phone benchmarks. Some people say what matters is real life use, so they go into and out of some apps and swipe around the home screen to see how fluid it is. But this doesn't tell you anything about how demanding apps will perform. This simple swipe test is stupid. It's like someone shows up for a job interview and you ask simple questions. Benchmarking software is the best we have to see how phones perform when they are challenged.

 

[El Presidente]

If you set the software/hardware to "run faster" than it normally would in day to day operations when benchmarks are running, it doesn't really give an accurate reflection of how the phone performs in real life (when it's running at stock speeds).

Many moons ago, when I believed benchmarks to be important, a friend and I used Quadrant to compare the scores of my Desire HD running CM 7 and his stock Sensation. I can't remember exact figures, but my Desire HD scored roughly 25% higher than his phone. However, the sensation absolutely wiped the floor with mine in terms of apps opening, web pages loading and even silly things like the length of time it took to authenticate and connect to WiFi networks.

If not faked, bench marks are ok at giving a decent overview of how a phone might perform, but they're nowhere near being the be all and end all.

 

[dibblebill]

Cheating at Android benchmarks isn't new | Android Central

There was another more recent article on it, and paparently someone's exposed the Note 3 adjusting features to boost scores. It is falsifying actual numbers to entice buyers, when they will never see that performance.

 

[kate]

Most manufacturers reportedly cheat in benchmarks:

AnandTech | They're (Almost) All Dirty: The State of Cheating in Android Benchmarks

Even if they didn't cheat, you shouldn't judge a phone only based on benchmarks.

 

[funkylogik]

Only use a benchmark/s to test your own phone against itself, like when u mess around with kernels etc. Thats it :beer:

 

[dibblebill]

Or to compare various ROM's on the same device, which is how I use Epic Citadel benchmarking and Antutu, to compare different Android images on my devices.

Which I will do to OmniROM tonight as soon as I get it working :<

 

[funkylogik]

^this

 

[boyo1991]

I think I somewhat agree with you. But only if all phones do it that way as logically thinking, if other devices do that as well, then it will act as somewhat a cancellation. You see them both at their best, the only other part of this is how much they force them back down when its done. It's a good start, but must be combined with other factors.

 

[funkylogik]

^nah my point is. Say i clear RAM and cache and run say AnTuTu for example, then flash a new kernel, run it again and compare. Or mess with GPU speed/voltages etc, its a way of getting actual figures.
How the phone FEELS is most important to me though