(UPDATE 03/09) Windows 8 Results (Temporarily) Not Accepted At HWBOT - Benchmark Result Veracity Compromised

  • News, benchmarks, Editorials, Articles
  • 117
  • HWBOT

Author: Pieter-Jan Plaisier

As the result of weekend-time research, the HWBOT staff has decided to invalidate all benchmark records established with the Windows8 operating system. Due to severe validity problems with the Windows8 real time clock (“RTC”), benchmarks results achieved with Windows8 cannot be trusted. The main problem lies with the RTC being affected when over- or underclocking under the operating system. The operating system uses the RTC as reference clock, and benchmarks use it to reference (benchmark) time.

Update 03/09 – Revised Moderation Guidelines Concerning Windows 8, Futuremark Benchmarks Allowed (with new SI)

Following our recent news item regarding Futuremark’s solution for the Windows 8 RTC bug, we have updated our HWBOT moderation guidelines. They now include exceptions for Futuremark benchmarks with ORB link and System Info 4.20. The updated rules:

  • All new Windows 8 Benchmark Results will be blocked, including for on-going competitions
  • Existing Windows 8 based benchmark submissions will be blocked if seemingly out of line or world record or top score.
  • Existing Windows 8 based benchmark submissions will not be blocked or removed if they don’t meet the requirements above
  • Existing Windows 8 based benchmark submissions will not be blocked or removed if they were made on an AMD system (New ones will be blocked)

Note: if you recall downclocking when using an Intel based Windows 8 system, please remove that result.

Exceptions to the Windows 8 rule:

  • non-overclockable notebooks may be used in combination with Windows 8
  • 3DMark and PCMark runs on Windows 8 if an ORB Link is included, and SystemInfo 4.20 (unofficial download) is used.

Update 24/08 – Revised Moderation Guidelines Concerning Windows 8

Following the findings of Ocaholic.ch (reference), the HWBOT moderating team has revised its rules regarding the usage of Windows 8 for benchmark submission at HWBOT. The major change is that the combination of AMD and Windows 8 is also not allowed, as in contrary to previous findings the platform is affected too. The revised rules can be found below.

  • 1) All new Windows 8 Benchmark Results will be blocked, including for on-going competitions
  • 2) Existing Windows 8 based benchmark submissions will be blocked if seemingly out of line, world record, or top score.
  • 3) Existing Windows 8 based benchmark submissions will not be blocked or removed if they don’t meet the requirements above

Note: if you recall downclocking when using an Intel based Windows 8 system, please remove that result.

Exceptions to the Windows 8 rule:

  • 1) non-overclockable notebooks may be used in combination with Windows 8

Update 21/08 – Updated Moderation Guidelines Concerning Windows 8

Following up on the recent developments on the topic of Windows 8 RTC, read the updated news item here, Head of Moderating Christian Ney has updated the moderation rules regarding the usage of Windows8. Here is a list of the updated rules:

  • 1) All new Windows 8 Benchmark Results will be blocked, including for on-going competitions
  • 2) Existing Windows 8 based benchmark submissions will be blocked if
    • seemingly out of line
    • world record, or top score
  • 3) Existing Windows 8 based benchmark submissions will not be blocked or removed if they don’t meet the requirements above

Note: if you recall downclocking when using an Intel based Windows 8 system, please remove the result.

Exceptions to the Windows 8 rule:

  • 1) AMD-based systems may be used in combination with Windows 8
  • 2) We are still debating on how to address non-overclockable notebooks

We also received a couple of emails debating the topic of which is to blame, Windows 8 or Intel CPUs. Based on the information we gathered from clock drift problem on various operating systems and hardware platforms, we are still of the opinion that Windows 8 is the root cause of the problem. Windows 8 seems to be using a different source to fetch “real time” on an Intel based system compared to an AMD based system. It’s still guess why and which at the moment, but it seems to be related to the power saving features integrated in Windows 8. With Windows 7 unaffected, and Windows 8 affected on all Intel platforms (including old generations), the root of the problem lies with Windows 8. Having the Intel system unaffected when coming from and AMD Windows 8 installation indicates that the correct timers are available on Intel too, but unused by Windows 8.

Thank you for your understanding.

The HWBOT Staff.

Update 20/08 – Windows 8.1 Affected, AMD Problem-Free(?)

We have continued our testing on more platforms. Windows 8.1 is, just like its older brother, affected by at-runtime base clock frequency adjustments. Intel platforms back to LGA775 are affected. From the looks of it, AMD systems are unaffected. In fact, the clock drifting on Intel based systems seem to resolve itself when switching from an Intel based system, to AMD, and back to Intel. Later today Christian Ney, Head Moderator, will publish an update to the rules at HWBOT concerning the usage of Windows 8. Stay tuned!

The following combinations of platforms have been tested

Operating SystemPlatformRTC Bias / Clock Drifting
Windows 8.1HaswellYES
Windows 8HaswellYES
Windows 7Haswellno
Windows 8Ivy BridgeYES
Windows 8Sandy BridgeYES
Windows 8GulftownYES
Windows 8WolfdaleYES
Windows 8Trinityno
Windows 8Piledriverno

How does it work and how to spot it?

  • Boot in OS at 130MHz BCLK and 32x CPU Ratio
  • Reduce BCLK frequency to 122MHz at runtime (in OS). Set CPU ratio at 34x to maintain the same frequency of 4160 MHz
  • Check benchmark before and after the downclock

You can also easily spot the RTC problem using Windows Clock (see video below)

More information


Background Information – Remember Heaven?

At the moment of writing, we do not have the full technical what’s and how’s figured out. Since this problem affects everyone who is passionate about overclocking, it is important to provide an explanation. It is far from the complete story, but it should be enough for you to understand why we have decided to ban Windows 8 from HWBOT.

Do you remember the history of Unigine Heaven at HWBOT? About three years ago, we launched a wrapper for the back-then brand new DX11 benchmark software Unigine Heaven. The wrapper featured an easy and secure way to submit benchmark scores to HWBOT via data files. On November 2, 2010 we posted a response to an on-going discussion about downclocking in Windows affecting the benchmark score. To make a very long story short, by downclocking in Windows the Heaven benchmark time runs slower than it really is. One second in the Heaven benchmark is equal to 1,x second of real time. Because there is more time within a second, the system can render more frames. The benchmark itself is unaware of all this – for the benchmark, one second is still exactly one second. In the end, the system renders a higher amount of frames in a longer timeframe. In the result calculation for frames per second (“FPS”), the “frames” have increased but the “second” remains the same. You get a higher score.

Later that week, we published a V1.03b version of the wrapper, which fixed the downclock issue. Without going too much into detail, the wrapper uses a second source to verify the benchmark duration and takes the relative measured time difference into account when calculating the final score. Problem solved.

Windows8 – “Support for all devices”

As you know, Microsoft is trying to come up with a unified operating system and user interface for a wide range of devices, including tablets, smartphones, Xbox One, and the desktop PC. Building this unified platform is not easy. It is not just a matter of creating an interface that can be used with a multitude of input devices (finger, mouse, controller), but it also needs to support as many devices as possible. Getting the software to run out of the box on as much hardware as possible is the challenge they are facing.

Of the many aspects to fine-tune, one feature in particular is causing Windows8 to be practically useless for (competitive) overclocking: the RTC. Quoting Wikipedia, “A real-time clock (RTC) is a computer clock (most often in the form of an integrated circuit) that keeps track of the current time. Although the term often refers to the devices in personal computers, servers and embedded systems, RTCs are present in almost any electronic device which needs to keep accurate time.” Sparing you the details of development process, compared to Windows7 and previous versions, Microsoft made changes to how it measures time to be compatible with embedded or low cost PCs that do not have a fixed RTC clock. After all, having a fixed RTC clock adds cost to a platform.

Your PC system uses the RTC for many things. For example, it ensures the Windows Time on your machine is accurate. For most benchmark applications, the RTC is used as reference clock when executing the benchmark code. By synchronizing with the RTC, the benchmark knows exactly how much time has passed, and takes that value into account when calculating the performance of your system.

It’s all relative (part2)

In a previous editorial (“Critical thinking – Should Maximus V Extreme be …”), we already touched the topic of the relativity of time in our overclocking universe. “The concept of ‘time’ on a PC configuration is, if not synced via network or internet, an arbitrarily defined constant designed to ensure that the configuration is running in sync with the real world. In other words: hardware and software engineers ensure that ‘one second’ on your PC equals ‘one second’ in real time. One of the reasons why it’s so important to have the PC’s timer line up with the real world time is to ensure that your PC can produce accurate measurements and predictions.” The points we brought up in that editorial are relevant again. To ensure that the arbitrarily defined constant of ‘time’ is the same on everyone’s benchmark system, we rely on the OS and hardware. This worked quite well, until Windows8 came around.

The problem builds on the problems we faced with Heaven. When downclocking the system under Windows8, the Windows RTC is affected as well. The biggest difference between Windows7 and Windows8 is that now all benchmarks (no exception) are affected.

Examples: Benchmarks and Windows Time.

Let us make this more practical. On our Haswell test system we downclocked the BCLK frequency by about 6% from 130 MHz to 122MHz. Using a CPU ratio of respectively 32x and 34x, the resulting CPU frequency remains 4160MHz. Then we ran comparison benchmarks. Here are a couple examples:

BenchmarkResult at defaultResult after underclockDifference
Wprime32M5.6055.235+ 7.07%
Prime9513.62213.423+ 1.48%
Heaven DX111417.9821522.567+ 7.38%
SuperPI 32M7min 337min 10.799+ 5.10%
PiFast15.7814.88+ 6.05%
Aquamark3351,160371,902+ 5.91%
3DMark Ice Storm141,977150,526+ 6.02%
3DMark Cloud Gate2046221676+ 5.93%
3DMark Fire Strike47275012+ 6.03%

As you can see, even the latest 3DMark is affected by this. In Windows7, most benchmarks are unaffected by this kind of “overclocking”.

We can also demonstrate the Windows8 RTC problem with the Windows Time. Have a look at the videos below. In one of the videos, we underclocked the base clock frequency by 6%. After 5 minutes, Windows Time was already 18(!) seconds behind real time. When overclocking the base clock frequency, we can see the opposite effect. We overclock by roughly 4%, and after two minutes, Windows Time is 3 seconds ahead of real time.

Conclusive lines.

First, we would like to apologise if this editorial seems put together in a hurry. If it seems so, it is because it is. As we hope you gather from the data, we provided in the article, it is not possible for HWBOT to accept any benchmark results or records achieved using Windows8. Simply no benchmark – not even 3DMark – is unaffected by Microsoft’s RTC design decisions. As a result, it is impossible to verify the veracity of a system performance indicative in Windows8. The resulting score of any benchmark is relative to the RTC bias of that system.

In order to compare we need the RTC bias to be equal for all systems. With Windows8, we can only use question marks.

Practically speaking, it is not allowed to:

  • exploit the Windows8 RTC for benchmark purposes
  • submit any benchmark results with RTC bias

The HWBOT staff will:

  • Block any seemingly out-of-line Windows8-based benchmark results
  • Block any Windows8-based benchmark record, even if the score seems in line with the expectations

We hope that everyone is willing to adhere to these new guidelines. The HWBOT Staff is as passionate as you are about overclocking, and this kind of external problem is affecting our hobby in a way that makes we cannot easily resolve. We are friends amongst each other, and respect the effort your fellow overclockers put into overclocking results.

Thank you for your understanding.

The HWBOT Staff.


117

Please log in or register to comment.