• After 15+ years, we've made a big change: Android Forums is now Early Bird Club. Learn more here.

S4 hard-coded to boost benchmark app scores

Hadron

Hadron

Spacecorp test pilot
VIP Member
Well we all know that benchmarks are a useful marketing tool but have little relevance to real world usage. If what Anandtech have dug out is true, it seems that's even more true with the Galaxy S4, and especially the Exynos version:

AnandTech | Looking at CPU/GPU Benchmark Optimizations in Galaxy S 4

The main points are that:

* the GPU in the Exynos version overclocks when running certain benchmark apps to a higher frequency than will be reached when running anything else, no matter how intensive

* the CPU governor behaves similarly: although it doesn't overclock, it does switch to the A15s with a fixed minimum frequency the moment you start these apps, no matter how low the load/

* the Qualcomm variants are similar: load the same apps and they run all 4 cores at max clock, even if you aren't actively running the benchmark (e.g. are just sitting looking at the benchmark app's menu)

* there's software on the phones controlling this behaviour with a hard-coded list of benchmark apps triggering it.

So it seems that at least one manufacturer has recognised the importance of benchmark scores to some customers and hard-coded to maximise performance in some popular benchmarks. You can argue about the CPU governor trickery if you like (that cpu governor setting doesn't reflect real-world behaviour, but it's not actually exceeding the maximum the system could produce if loaded hard enough), but overclocking the GPU above what you can ever get outside of the benchmark app smells to me.

Of course we don't know whether Sammy are the only ones to have thought of this wheeze, just that they seem to have been caught. But as Anand and Brian say, it would be better if the work that went into this had been devoted to optimising real world performance.

Moral: take benchmark scores with an even larger pinch of salt than ever ;).
 
Thanks man. i always suspected this. Bet AnTuTu is one of the benchmarks since a lot of reviewers use it?
I prefer Geekbench 2 but i do take them with a pinch of salt.
I think theyre only useful when comparing your own phone with itself. Like when you change the kernel or its settings :thumbup:
 
Seems that Samsung have been caught manipulating benchmark results. This soon after the Intel benchmark manipulation fuss.

It's the 90s redub!

Basically, Samsung's S4 has a built-in check to identify when it is running common benchmarking tools. When it spots these, it cranks up the GPU and CPU speeds beyond those they normally run at (at least, that's my understanding). Obviously, this means the benchmarks are measuring speeds that the phone is capable of running at, but doesn't actually run at - for me, that kinda counts as cheating, but it's a matter of opinion, I guess.

Thought this might be of interest as, unlike most phone buyers, I would guess there people on here who actually look at benchmarks before selecting which phone to buy.
 
Thread merged with one already in existence ;)

Just one more reason to not make a phone purchase decision based on benchmark scores IMO. :p
 
some of you may recall my recent foray with a clone phone.....near as i can tell, none of the benchmark apps that i tried got things right. antutu, quadrant, cpu z. they were all fooled in one area or another.
 
i am using it! i got one of those cpu control apps that you recommended. i have things turned waaaay down for most things and the battery is at least now doing a a fair job. and i have the cpu zooming for a few apps. the device actually is pretty snappy if i let the cpu motor along.
 
Doesn't surprise me, some benchmarks make the device very warm. Much more so than just playing a game and the benchmark only lasts a few minutes at most.

Also with CPU throttling it's never at highest clockspeeds on the snapdragon version for very long anyway so I don't think that's an issue, or rather it doesn't really matter. Actually I'm happy they're doing this as it saves me forcing the performance governor when running benchmarks. :D

I'm just wondering how they're changing the GPU clockspeeds on Adreno 320, it's 450MHz max so is that for benchmarks only as well or is that left alone? Didn't see any mention of that.
 
From the article I got the impression that it was easier to check the clock speed of the Exynos's GPU than most, so I'm not sure that was tested for the Adreno.
 
That's exactly why I don't bother with benchmarks. You'll even see some devs set up their kernels purposely to produce bad benchmark results so everyone stops going on about them.
 
Shotgun84 said:
That's exactly why I don't bother with benchmarks. You'll even see some devs set up their kernels purposely to produce bad benchmark results so everyone stops going on about them.
Click to expand...

A agree you shouldn't take benchmark results too seriously, however limiting max GPU clockspeeds only for benchmarks is very underhanded.

If the GPU can do 533/450MHz as advertised then I expect that in every 3d game I play, not just benchmarks.
 
Shocky said:
If the GPU can do 533/450MHz as advertised then I expect that in every 3d game I play, not just benchmarks.
Click to expand...
Sammy have issued a response in which (of course) they claim it's about optimising overall experience rather than gaming benchmarks - though they don't address why the code is called "benchmark booster" ;)

The wording is a bit misleading IMO. It implies that they throttle down the gpu for some games, whereas it's actually only unthrottled for a few apps. And if that's the case, why include benchmarks on the short list of privileged apps - irrelevant to user experience and misleading if most apps have lower gpu performance. So nice try, but I'm not convinced.

The best outcome would be if blogosphere reviewers took the hint and stopped reporting synthetic benchmark scores. Unfortunately not very likely.
 
OK, am I joining too many dots here?

Another Anandtech article on updates to the Exynos Octa notes that there was a bug in the original version of the SoC (the one used in the Exynos S4):

Part of the problem with the design was a broken implementation of the CCI-400 coherent bus interface that connect the two CPU islands to the rest of the SoC. In the case of the 5410, the bus was functional but coherency was broken and manually disabled on the Galaxy S 4. The implications are serious from a power consumption (and performance) standpoint. With all caches being flushed out to main memory upon a switch between CPU islands. Neither ARM nor Samsung LSI will talk about the bug publicly, and Samsung didn't fess up to the problem at first either - leaving end users to discover it on their own.
Click to expand...
So that means that there's a significant penalty when switching between the two core clusters. And one of the things this "benchmark booster" does is forces the system to use the A15 cores the moment you start a benchmark app. So one consequence is that for apps on the benchmark booster whitelist this bug will not be visible - part of the motivation, or just a fringe benefit?
 
I knew samsung didnt have the octocore working the way Arm demonstrated it but trying to hide that in that way is pretty low. Have they ever admitted the s3 "SDS" bug?
 
Hadron said:
The best outcome would be if blogosphere reviewers took the hint and stopped reporting synthetic benchmark scores. Unfortunately not very likely.
Click to expand...

But how do they test performance if not for benchmarks? you can't just say X is faster because it feels smooth. Nobody would take them seriously if they did that.
 
You must log in or register to reply here.
Back
Top Bottom