Are We Hearing the Whole Story In the Samsung Benchmark Issue?

About a week ago, we highlighted AnandTech's news item of Samsung enabling higher performance levels for certain applications on the global edition of the Galaxy S4. The set of applications included a couple of popular Android benchmarks such as AnTuTu and Quadrant. As the world of mobile is hot these days, AnandTech's article triggered a lot of responses. Amongst them, this editorial over at by author Sean Kalinich. In this editorial, the author sort of defends Samsung's position stating the optimisations are much rather part of a general performance profile customization than they are to purely enhance benchmark results.

Where the author might be correct that the word "cheating" is a little over the top, he does conclude by pointing fingers at Samsung for the lack of transparency. Furthermore, the author points out more general flaws of the media in this industry.

Samsung screwed up by enabling what looks like an extension for a profile that already existed in Google’s Android. Regardless of the fact that it does allow other apps to perform better, they should have not included bechmarks and tests by name. They should not have done this without some sort of explanation or disclaimer. There are a ton of developer features in Android (and in Samsung’s flavor) so why not point to those when running a benchmark and ask that developers put this information in an FAQ? This would allow someone to see performance in “stock” mode and then in a “benchmark” mode. One gives you the basic performance profile and the other allows the device to push its max limits. It is a simple solution that should have been thought of and allowed for.

Benchmark developers for mobile are also to blame here as their code is not suited to the typical power and performance profiles on most phones. AnTuTu was already shown to not have properly optimized code for all platforms so there is a big possibility that other benchmarks are also inefficient in this regard. It is possible that certain optimizations are needed to allow benchmarks to perform properly. It is not an easy task to develop code that works for all hardware build and Android versions. We know that manufacturers have to spend a considerable amount of time reworking their OS offerings whenever Google pushes out an Android update so why would we expect an app that is written for stock Google to work the same across all builds?

Lastly is the industry and tech press. This group is to blame possibly more than others as they have helped to create the environment that allows for this behavior. The push to get content out as quickly as possible means that corners are sometimes cut and not all of the information is fully investigated. Reviews have also become shorter and less detailed. The focus in on synthetic and scripted tests. These can be performed without much user interaction so the process of the user experience has been removed. There is very little attempt to do real-world testing as reviewers rush product in and out of the lab. This is further fueled by companies that have deadlines for products to be covered and published. They also know the value of the news cycle and want their product covered now for maximum impact. These two groups have been the driving force behind the change in the way the industry and products are covered. Instead of through and fair coverage now it is about scandal and being the first publication to publish.

In the end, it comes down to a single question: "Are performance enhancements triggered for specific applications, designed to measure and compare performance, faul play?". I believe it is faul play, and with Samsung targeting specific benchmark they are at fault. What do you think?


Belgium Massman says:

Another point of view on the "Samsung BenchmarkBoost" story

United Kingdom borandi says:

I've never known AnandTech to publish early. Brian and Anand wait until the review is ready - the SGS4, HTC One, iPhone and MacBook reviews are always 3-6 weeks later than everyone else. As Brian pointed out, the Samsung issue causes the GPU bin and TDP to be boosted for certain applications specifically specified in the code. That's not a performance profile - that's artificial boosting for the purpose of benchmarks with no real-world influence. Even after Samsung's response, stating several stock apps that have this 'enhancement', the only one that would ever hit this bin was the Camera app, and only when switching between filters stupidly fast = not real-world. That is a foul play. The fact that v2.5.1 was not specifically mentioned causes Sean to draw the opposite and also false conclusion - as it isn't listed in this particular program, that it must be a performance profile. I assume he then went through every single file looking for the string, or was it a case of automatic assumption based on a lack of evidence to draw a conclusion? A false positive is a false positive, not a negative: you find a defendant guilty or not guilty, not guilty or innocent. Don't take it from me, Brian and Anand spoke *at length* in the latest Podcast about the issue: If there's specific problems with their testing, despite the fact that they tested more than 3x of what was published, then why not ask them direct on twitter for confirmation? That's not to mention Sean's rant on Dino's facebook the other day either, seems like a bugbear on Sean's shoulder over AT specifically.

Please log in or register to comment.