Author: Pieter-Jan Plaisier
There’s no doubt that Australia deserves the title of Country Cup winner this year. And yet, during the competition the boys Down Under caused quite a bit of commotion. Not because of their results – those are phenomenal – but because of the attitude of sandbagging their scores until the last day of the competition (stages). Some even call it a “coward’s move!” Sandbagging has been a part of competitive overclocking for a long time, and it is still causing discomfort amongst participants. Reason enough to have a closer look at the problem and alternative solutions.
Most of the suggested solutions brought forward in the discussion forums relate to adjusting the points system. By incentivizing to submit a score early, the hope is to get the big scores out earlier in the competition. In this editorial I will try to cover the problem of sandbagging from various angles and argue that a change in points system does not address the issue at core. I will also suggest and alternative solution which distinguishes itself by not adjusting the points, but the addressing the root cause of the emotional reaction.
It’s ironic, and time doesn’t matter.
The most common suggestion to resolve the problem of sandbagging, is to introduce multiple phases during the competition timeframe. The concept is similar to the one used in the NASCAR Sprint Cup Series, or for the Points Classification in the Tour de France. Simply put: you can earn points at the end of each phase during the competition. See illustration below:
Typical overclocking competition
NASCAR-style overclocking competition
There are two fundamental problems with this suggestion. 1) The system increases the importance of sandbagging scores and 2) submission time is now part of the equation to value a benchmark score.
The irony of more sandbagging.
If you look at the illustration above, it is easy to understand my first criticism. By introducing intermediate stages with points, we increase moments for sandbagging. In the situation sketched above, we have now five instead of one occasion for sandbaggers. This is rather ironic given that the goal is to reduce sandbagging.
The intermediate stages don’t only increase the amount of sandbag opportunities, but may also motivate people even more to hold on to their best results until the sandbag moments. Currently sandbagging only occurs at the end of the competition. During the majority of the competition, there is no reason to hold back scores. In fact, you only need to hold on to your best score if you want to sandbag. In a situation with five intermediate stages, it may be more interesting to hold on to at least five results. The increased importance of sandbagging caused by the introduction of intermediate stages might lead to participants submitting even less during the competition, and hold on to the scores even tighter. After all, a bag of sand is now worth five times more!
Time is not relevant to the value of a score.
The second criticism requires more elucidation. Let’s assume that at the end of each intermediate stage, one (1) point is awarded to the best score, and ten (10) points at the end of the competition. So if you submit the best score in between the start of the competition and point A, you get a point. Also assume that the winning score of the competition stage is 300,000 points in 3DMark03 – that would be a World Record. So, the value of the “World Record” at the end of a competition is:
- 10 points, if submitted in between point D and the end of the competition
- 14 points, if submitted in between the start of the competition and point A.
According to the logic, it means a score becomes more valuable over time. That is not sensible! The value of a benchmark score relates only to the selection of hardware components and the amount of scores beaten. For example, scoring below 10 minutes in SuperPI 32M with an AMD Phenom II – something only one person has achieved so far – is impressive regardless the date it was achieved it. It was impressive in 2011 and it will be as impressive in 2015.
One could argue that next to the factual parameters (hardware components and scores beaten), there are subjective parameters that should also be taken into account when valuing a score. For example: effort, dedication, skill, knowledge, or persistence. Most of the parameters relate to the overclocker, not the hardware. Although these are hard to quantify or measure, I tend to agree they could add extra value to a certain benchmark score. Sometimes it takes weeks to beat a record that someone else set, just because the hardware does not clock that easily. In that case, breaking the record can be considered ‘more’ impressive.
What does it take to submit early in competition? From the Country Cup comments, I find that the following items determine if you can set a score early in the competition: hardware availability, cooling availability, and time
Even though the Country Cup stretched a period of a month and a half this year, many teams did not complete all the stages with the required amount of scores. For example, even the Country Cup winner Australia didn’t manage to close the AMD Aquamark stage with 5 scores. The reason is usually simple: the hardware did not arrive in time.
The question then is: “Is it reasonable to give extra points to teams who have the time and hardware to submit scores early in the competition?” I don’t believe so.
In conclusion, I am convinced the timestamp of the benchmark submission does not affect the value of the score. In addition, not submitting a score early is usually not a matter of unwillingness, but rather related to the availability of time and of course the hardware. Therefore, it is unreasonable to give more points to a score because it’s submit earlier during the competition.
A David and Goliath story.
A couple of days ago Hendra, better known as Coldest, from Jagatreview came by our Taipei office. Of course we debated the topic of sandbagging, and during the discussion we realized we overlooked another part of the sandbagging story: the perspective of the sandbagger.
After all, why would anyone want to sandbag? We all know it will piss people off, and unless we believe sandbaggers are by nature horrible people who enjoy the pain of others, it seems odd to make that choice.
During the discussion with Hendra it became apparent that the sandbag story is similar to the story of David and Goliath. Goliath, in this case, is the overclocker with near-unlimited resources. Goliath can bin more CPUs than anyone else, and has plenty of time to re-do scores if necessary. David, on the other hand, can only buy one or two CPUs, and only has two weekends to set a score. To have a chance at winning David has to outsmart Goliath, and in online overclocking competitions sandbagging is the way to do it. It’s very simple: if you know your competitor has better hardware, or can bin for better, you need to come up with a way to beat him. Not informing about your best results is a great tactic, as it prevents Goliath from looking for better hardware or re-benching his superior hardware.
“I could’ve beaten that”
One of the comments from sandbag-haters is that they could’ve beaten the winning score with their setup. The argumentation is that putting a score on the board serves as target, and motivates overclockers to push harder. That is true, but the argumentation goes both ways as we know from the David and Goliath analogy.
In this case we should actually not address the sandbagger, but the person who was beaten by the sandbagger. The question is: “if your setup could’ve beaten the winning score, why didn’t you?”
After all, a principle for all competitive environments is that each competitor does the best he or she can. This principle applies to all sports: holding back thinking your current result will be enough is not the right attitude. Think about Formula 1 for example. A driver who finishes on the 11th place during the qualifying phase because he thought the time was good enough and refused to do another lap, will be blamed by the team manager. Or the Champion’s League football, where a coach might opt to start with a B-squad to give the players of the A-squad rest, and figures qualification is a guarantee. If the team loses, and then also loses the qualification, everyone will ask why the coach didn’t play with the A-squad.
Both examples have happened many times before and the same applies to overclocking. If you get beaten by a sandbagged score that you’re sure you could’ve beaten, you should slap yourself over the head for not pushing harder. The reason for not winning is because you didn’t max out the system, not because you didn’t have the target!
Suggestion: Dark Day and Popcorn Time.
Until now I have tried to form an argument supporting the idea that sandbagging isn’t as bad as we might think. I have explained how introducing a NASCAR-style point system increases the amount of sandbagged scores, and how it imposes a logical error by rating benchmark scores by the time it was submitted. I have also pointed out that it is a necessary tactic for less resourceful overclockers to beat those with plenty of time and hardware binning possibilities. I concluded by pointing out that people who say they could’ve beaten the sandbagged score, should ask themselves why they didn’t max out the system.
Conclusion: the sandbag is a legitimate tactic which we should embrace to give the less resourceful more chances to win a competition.
Of course, all this reasoning doesn’t resolve the issue. It doesn’t matter how much ratio and logic you throw at the issue, if you are beaten by a sandbagger you will feel pissed off. We still need a solution! Because I believe we should embrace sandbagging as a legitimate practice, I suggest to address the problem from a different angle: the emotional aspect. After all, the debate is not if the best score should get the most points. We all agree on that. But being beaten by a sandbagged score triggers an emotional reaction. The emotional reaction may be caused by the dissymmetry of the competition and the point grading. Let me explain.
The competitive aspect of an overclocking competition is spread over a specific timeframe, but is asymmetric. For example: someone may compete during the first week of the competition and set all his scores, where someone else may only compete in the very last week. It is not a direct competition like MOA or AOOC. In contrary, all the benchmark scores are graded at the exact same time: at the end of the competition. My suggestion to address the issue of sandbagging relates to that last point. We should change the way the competition is “closed” and how the points are awarded, and make it as asymmetric as the competition itself. For this, we need two additional phases during the competition: “Dark Day” and “Popcorn Time”.
Dark Day is a period of 24h before the end of the competition during which competitors can still submit results, but the ranking and scores are no longer be visible. You are in fact competing in the dark; you don’t know who is submitting what scores, and you don’t know who is leading. You also don’t know who has just submitted sandbagged results. The only information you have is the ranking before Dark Day, and of course the information from your competitors as they tease in the forum or on social networks.
Popcorn Time is a period three hours after closing the competition. Instead of grading the scores at an exact time, the competition engine randomly reveals the scores submitted during Dark Day. The ranking is adjusted accordingly. Popcorn Time essentially stretches the emotional experience of competition closure over a longer period: from a single instant to three hours. Additionally, you’re not sure if a score was sandbagged or in fact achieved on the last day of the competition. Popcorn Time also allows to create a more exciting ending of the competition. It introduces a three-hour live show, and highlights more than just the scores of the teams competing for the top ranking. Envision a live updated scoreboard, an embedded chat, and of course beer with snacks.
Sandbagging is a source of frustration for many competitive overclockers. It makes the competition experience worse, and that is a problem for the competition host. Overclockers have suggested solutions to resolve this problem. In an attempt to motivate participants to submit their best score earlier in the competition, the usual suggestion is to re-arrange the points system. In this editorial, I argued that a new points system does not necessarily resolve the situation.
In addition, I am of the opinion that there are plenty of arguments to be made in favor of sandbagging. For example, it allows the less resourceful overclockers to outsmart those with lots of time for binning and re-benching. Also, being beaten by a sandbagged score that you could’ve beaten had you known the target score, is not a reason against sandbagging. In a competition, you are supposed to push as hard as possible and squeeze every point out of the system. If you stop pushing, and get beaten, well than that’s your fault. If you squeezed every bit and you still get beaten, well then the other won legitimately and there’s nothing or no one to blame.
As an alternative solution, I propose to address the emotional reaction to the end competition by introducing the principle of Dark Time and Popcorn Time. Dark Time freezes the rankings 24h before the end of the competition. None of the participants can see the new scores and the effect they have on the ranking. Popcorn Time introduces a new experience of the competition closure, as over a period of three hours after the closure the engine randomly reveals the scores submitted during the Dark Time.
The goal with Dark Time and Popcorn Time is the same as for every other suggested solution: to reduce the negative emotional experience of sandbagging. The main difference between the two solutions is that rather than criminalize sandbagging and trying to get rid it, the suggested solution embraces sandbagging as a legitimate strategy and focuses mainly on adjusting the competition experience.
As the title of this editorial suggests, this solution is just one opinion on the subject of sandbagging. Feel free to leave your opinion as comment. I look forward to hearing your opinions! Thanks for reading.
(the opinion expressed in this editorial is of the author only and does not represent that of HWBOT the organization)