Y-Cruncher Multi-Threaded Pi-Benchmark Updated with GUI and HWBOT Compatibility
y-cruncher is a benchmark program that computes Pi and other mathematical constants to the region of billions or even trillions of digits. The program was created to calculate the vastest Pi numbers possible and in fact holds the world record for the most ever recorded; 13.3 trillion to be exact. Previously only available sans user interface, our man Mystical has taken the time to create a more complete app that integrates a graphical user interface and the ability to submit on HWBOT.
Originally developed in 2009 as a high school project by Alexander J. Yee, y-cruncher is actually quite unique in a number of different ways. It’s the first Pi computing program that can use multiple threads for a near linear boost in calculation speed, it can also use and stress unlimited amounts of system memory and it also uses ISA extensions such as SSE and AVX available on most modern processors.
The new app created by Mystical is called the y-cruncher HWBOT Submitter. It's written in Java and requires Java 8 runtime to run. Note: y-cruncher itself has no requirements other than Windows Vista or later or above. The app includes three presets; Pi to 25 million digits, 1 billion digits and 10 billion digits. The 25 million digit run will complete in just under two seconds, whereas the 10 billion run will require 48GB of memory.
You can learn more about the y-cruncher HWBOT Submitter on the forum thread here.
I'm not sure how many of you are familiar with the y-cruncher Multi-Threaded Pi Program.
It's been around for quite a while now (since 2009). In short, it's a program that computes Pi and other constants to billions/trillions of digits. It currently holds the world record for the most digits of Pi ever computed (13.3 trillion digits) as well as a bunch of other less popular constants.
y-cruncher is also the first Pi computing program that can:
There has been some hope that it could be a SuperPi/PiFast alternative. But that never really happened.
Over the years, I've been asked numerous times why this program never became part of HWBOT. In fact, I've had many chats with Massman. But none of them really got anywhere until recently. Also, the fact that the program lacks a GUI didn't really help either. And to be fair, y-cruncher was designed as a math program with one purpose in mind - to break size records. Competitive benchmarking and user-friendliness was always secondary.
Official XtremeSystems thread here. (Though it's been a while since I've updated it.)
-----
Anyways. It took about a week of work, but I've thrown together an app complete with a GUI that can run and submit y-cruncher benchmarks to HWBOT.
This app is called the "y-cruncher HWBOT Submitter". It's written in Java and requires the Java 8 runtime to run. (y-cruncher itself has no requirements other than Windows Vista or later.)
For now, I've only enabled 3 benchmarks for HWBOT:
The 25m benchmark will go under a few seconds for modern hardware. That's too fast, so we'll probably drop that at some point. The only reason it exists in the first place is because it's fast and easy to test.
The 1b benchmark is the standard size. It requires about 5 GB of ram to run and will take a few minutes to run for most high-end systems.
The 10b benchmark will require 48 GB of ram. That basically implies a minimum of Skylake, Haswell-E, or some server. If you don't have enough memory, it's possible to run it using swap mode. But that's more complicated to setup and will be slower than doing it all in memory.
How does it work?
Anyone who's familiar with y-cruncher will know that it outputs a validation file at the end of every computation. The submitter app is a runnable .jar file that you can put in the y-cruncher folder.
When you run it, it automatically searches out all the validation files and verifies the checksums in them. The ones that are valid and match a supported HWBOT size are available for submission to HWBOT.
The submitter app is a wrapper on top of y-cruncher itself. No changes to y-cruncher were needed for this to happen. And quite frankly, I designed it this way so that I could keep all the Java networking/GUI separate from the 300,000 lines of ugly C++ that is y-cruncher.
Download:
Current y-cruncher version: 0.7.7.9495
Current HWBOT submitter version: 1.0.1.133
Version Support:
The submitter app:
Despite being written in Java, the submitter app does not run in Linux (at least I couldn't get it to run). But at the very least, validation files generated in Linux can still be submitted to HWBOT if you transfer it to Windows and run the submitter app there.
I have yet to figure out why it's broken in Linux, but it seems to involve the JavaFX library. In any case, even if someone does manage get it to run, the benchmark integration will still be broken since it uses Windows-specific command-line parameters to launch y-cruncher.
Version History:
Main Page: http://www.numberworld.org/y-cruncher/version_history_ui.html
Thanks for this - I really like Y-cruncher, especially for testing big memory configs. It seems that 25m might be a bit too easy for the current systems as we're already below 2 seconds. Maybe swap this one out for a more heavy test than 1b?
I wouldn't mind seeing a test which requires 32GB memory or more ... just to see how the high density guys clock their gear!
Just tried the submitter - I'm very impressed with the way you handled the flow from bench to submit! :celebration:
One issue though, I can't seem to submit my Pi-25m result. I'm using the "y-cruncher v0.6.9 with HWBOT Submitter"
That's some awesome news
Ugh... I tested this thing on at least 4 different computers before I felt confident enough it would work.
I'm unable to reproduce it locally, but HWBOT.org is intermittently timing out. (not sure if that's in any way relevant)
How consistently are you getting that error? Is 100%, what about on other machines? As of right now, there's no way for me to debug it unless I have it output both the datafile and the server-sent error message.
(Btw, I'm willing to share the source code of the submitter app with HWBOT if that makes anything easier.)
Java.
same here.
evri time i try to submid i have this error.
http://forum.hwbot.org/showthread.php?p=438309#post438309
Thanks. Since this is happening with more than one person and HWBOT.org is stable atm, then it's a problem with the app itself.
I'll push out an update (hopefully tonight) that will log the datafile and error message that HWBOT sends back.
ok thx.have the same error on my wifes Laptop whit win 8
good work guys!
awesome, this benchmarkm is also very stressfull. But I like it
I've just uploaded a new version (v0.9.1.61) that will output the datafile and the full error message when a submission fails.
If it fails again, please send me the following files: (zip them up and attach here)
Hopefully that will be enough for me to figure out what the hell went wrong. Unfortunately, that's the best I can do since I've been unable to repro the error on any of my own machines.
(Btw, the datafile is encrypted. But it's being sent unencrypted to HWBOT since I can't get that to work.)
stil not working on wyfes loptop.
when i am back home after work i gone try it on the other config and send all the files.
what is your e-mail to send it.pm don't like one hwbot file
Here are some error files
Right, I haven't actually fixed anything yet with v0.9.1.61. All I did was add the logging.
Thank you! That did it:
A localization bug. Decimals were being converted with commas instead of a decimal point. And because of that the HWBOT server couldn't parse them.
I just pushed out a fix for that. See if v0.9.1.62 works for you guys.
I can't guarantee that's the only localization bug. But at least we're one step closer. And that would explain why it worked on all my machines.
it works on the tyan test setup
whil also test it on wifes Laptop
edit : it also works on the wifes laptop
http://hwbot.org/submission/3173421_skulstation_y_cruncher_pi_25m_2x_xeon_5148_15sec_993ms?recalculate=true
Works here too: http://hwbot.org/submission/3173424_massman_y_cruncher_pi_25m_core_i7_4500u_5sec_787ms?recalculate=true
Will try to fire up a more capable system later today
Good work!
Mysticial, can I ask you some question?
You used Java only for GUI? So is it possible to run benchmark, save validation file and submit through java app on other machine? I think most benchers don't like 2D benchmarks that require additional software to run.
As far as I can understand benchmark run linux, problem is in java submitter?
Correct. Only the submitter app is in Java. The main app has always been in C and C++. I chose Java for the submitter app because GUI and networking is a pain-in-the-ass in C++. (I also expected it to work out-of-the-box in Linux as well.)
Yes, the validation file is the only thing that matters. So you can run the benchmark one machine, and transfer it to another machine to submit it.
For that matter, any Pi computation to one of the supported sizes can be submitted to HWBOT. It doesn't have to be one of the benchmark options. You can use the custom compute menu to fine-tune the parameters. (Hint: The default parameters are often sub-optimal. So you'll probably need to do this anyway to get the best possible time.)
In the future, I'm probably going to add one limitation in that the computation has to be contiguous. (So that you can't start a computation on a fast computer, pause it, and finish it on a slower one to make the slow one appear faster than it really is.)
Update: I finally got the datafile encryption to work. So that has been enabled along with a new version of the submitter. (v0.9.2.64) This also means that older versions of the submitter will no longer work. So you'll need to download the new version.
With this, I also switched the benchmarks to public beta.
says:
Hello
I nevertheless uses the latest version.
I do not understand?
That's strange... It works for me.
Since the version restrictions clearly aren't working consistently, I went ahead and disabled them. I added those yesterday because I wanted to blacklist v0.7.1 until it is released. (There are a few of copies floating around out there. And they are faster than v0.6.9, so it wouldn't be a fair to those with only v0.6.9.)
I guess that probably explains why there haven't been any submissions lately... Oops...
Works here - http://hwbot.org/submission/3177103_massman_y_cruncher_pi_25m_core_i7_4500u_5sec_536ms
I'll set up a small competition to test the benchmark
says:
It's good now
Hi ^^
Trying y-cruncher at the moment, congrats for the well integrated Hwbot wrapper !
25M and 1B done, i'm currently running the 10B ... see you the next decade
FX-8350 and 16GB, i put the bench on my hdd and enabled the swap mode.
Now, 54% completion for 1h11.
Just a question, y-cruncher use the disk where it is, or it use system disk ? maybe my ssd is getting raped x)
Edit : ok i see the files, it use current hdd, my ssd is safe ^^'
Are you running 10b in "Ram Only" mode or "Swap Mode"? If you're running it through the submitter, it will be in Ram Only mode and it will thrash the pagefile. If you're running it in Swap Mode directly in y-cruncher, then it will use the default path unless you specify otherwise. (Even though Swap Mode uses the disk, it is much more efficient than the pagefile.)
The default path will be wherever you double-clicked to open the program. So if that's your boot drive with the SSD, then yes it will be "raping" your SSD.
Wearing out an SSD is possible if you're not careful. But this is mainly a problem for the really big runs. (A 100b computation will incur 20+TB of writes if you have no more than 32GB of memory.)
swap mode, manually via the commande line.
But hell, it doesn't progress for a percent in 20min ... only semi-load iterations of "VST-PM" & Co.
What the point of these phases ?
The computation isn't really linear. It's divided up in to multiple blocks of unequal size. While it's possible to derive the % mark at the top level, it's almost impossible within each block. What you're seeing are the progress indicators deep within the computation algorithm itself.
For the most part, those internal progress indicators are meaningless to anyone who isn't familiar with the algorithms that are used. I used to hide them prior to v0.6.1. But when you're running a really large computation that takes weeks, seeing a static "XX%" indicator for several days straight doesn't provide much in the way of positive reinforcement that the computation is still running. So I ended up turning them on in the public versions even if hardly anyone will be able to decipher its meaning.
Fyi, in case anyone is interested, here's the link to our beta test competition: http://oc-esports.io/#!/round/ycruncher_beta_contest.
I'm off trying the 1B
10B -> 3h43min
http://hwbot.org/submission/3177754
Swap mode on a single drive is painful *_*
Is that with an SSD? Or HDD?
i put it on the HDD
The submitter says it can't detect a compatible version of Y-cruncher on all my computers.
lol System was not stabel.crach in 10b after some hours of crunching, was watching the furst belgium in the F1
whit 8 gig fb-dimm ram and swaping to a singel ssd.
almost 1TB swapped to the ssd
How are you running it? It needs to be unzipped and in the same folder as y-cruncher.exe.
(In reality, it actually parses the "Read Me.txt" file to determine the version # of y-cruncher that it's trying to interface with.)
win 10/server 2012 r2 allowed?
That's probably a question for Massman. I would guess that the answer is yes for now since it's still a beta. So now is the time to identify any exploits with the benchmark.
I actually don't know if y-cruncher is vulnerable to the base clock exploit/cheat. But I would assume yes.
In any case, it would be difficult to reliably enforce a ban on specific operating systems since the validation files do not record the OS information. So if someone is inclined enough, they could run the benchmark in Win8/10 with the base clock exploit/cheat. Then reboot the machine into Win7, fake a screenshot and submit it.
This will be fixed in v0.7.1 as it will record the OS version into the validation file.
With that said, if anyone knows how to implement a high-resolution timer (in C++) that is resistant to the base clock exploit, please let me know.
nice benchmark, nice stress test also, not so nice heat output
Hello, I have a recurring problem, each time I run into Pi-1B or even 89%, I have the right to a message:
Redundancy Check Failed: coefficient is too large.
an idea?
Everything that heavily uses 256-bit AVX will do that on Haswell. I presume not too many benchmarks use AVX.
Is this happening even when the hardware is completely stable? That error normally only shows up on hardware instability. The fact that you're seeing it so early (and consistently) suggests something is seriously broken. But nobody else seems to be hitting it...
Did you do unusual tweaks to the OS? I can only think of two possible things, but I'm not sure if they're even possible...
using 1366 here, haswell to new for me
.
that errors i hit this morning on the r3e , overclock wasnt stable.
got a hp ml150 g6 board here with two xeon L5520,,need to test but, 12gb ram only , will it do swap on 2x300gb sas raid? the 10billion test..
No changes, and even the stock has hardwares that me also displayed.
Pi-25m goes smoothly
but Pi and Pi-1b-10B, its not want to know. every time I have the right to error.
Yeah, the program doesn't really care what you're running on. In fact it doesn't even know. All it knows is a path.
If you're interested to see what a "high-end" swap mode configuration looks like, here's my 4770K doing the 10b with only 32GB of memory.
http://hwbot.org/submission/3178862_
I was going to ask for more details of your system. But then I noticed that you did manage to get a 1b score submitted.
What did you change? I'm really curious since it really shouldn't be happening so I'd like to fix it (or at least do something about it) in future versions.
Btw, someone else seems to be hitting the problem as well - which is why I'm concerned.
damm impressive..with swap!
Call me fanatical, but that entire machine is custom-built to run y-cruncher's swap mode. It's sole purpose is actually to test y-cruncher through its development.
It's got 4 versions of Windows, 3 versions of Linux, and absolutely nothing installed other than compilers and overclocking tools.
I think you'll find this community is very approving of "fanaticals"
I simply increase my virtual memory is 8192 MB 0 MB
And I can walk in a settings Y-cruncher, option 3.
I joined you the validation file of Pi-1B if cel may help you to understand, that was changing.
By against Y-cruncher, is very hot CPU .... my i7 4790K has 4.5Ghz, above 80 ° C in benchmark .....
Incidentally, his will that everyone sticks the screenshot to their score as many do that put pictures of their config. too bad.
PS: sorry for my english but I translated with Google.
I'm gonna leave this right here:
HPET detection seems to work on all 4 of the machines that I've tested on. There's an additional hardware timer that I'm interested in testing, but none of my machines are suitable for that. So that'll have to wait.
That's enough for the meantime. Too much programming means that I fall behind on my Anime...
p.s. Don't hold your breath for what's in the screenshot. This required a change in y-cruncher itself. And therefore it will need to wait until v0.7.1 is launched. (That will take a while and I definitely will not do it before the beta competition finishes since it will break speed consistency.)
what you need to do to make this datafile span which convert every
Can you please xplain why I can submit with the hwbotsubmitter but same file I think cannot be uploaded at hwbot form?
Originally, it was because the datafile wasn't encrypted. So you could just edit and submit whatever you want. But that point is moot now since I got the encryption to work.
But one problem remains:
I haven't tested this so I can't confirm it. But by the looks of it, HWBOT doesn't try to verify if the datafile is actually for the right benchmark. So you can manually submit a 25m benchmark into the 1b category.
That's a big enough loop-hole for me to simply disallow manual submissions for now. I could fix that by using different encryption keys, but that complicates things and it wasn't something I considered important enough to bother with.
That´s ok and a good reason, for me the problem is more people asked me and I could not explain - thanks
Win XP doesn't work
Support for Windows XP was dropped 4 years ago. Version 0.5.5 was the last version that could run on XP. Unfortunately, only v0.6.1 and later is supported for HWBOT.
In short, the latest versions can't run on XP because they make system calls which didn't exist prior to Vista.
As far as performance goes, you need a minimum of Win7 SP1 to be able to run the AVX binaries. So even if you could run on XP, you can expect a slow-down of at least 2x compared to Win7 SP1 and later. (Assuming you're running on Haswell.)
Thanks
I just pushed an update to the submitter app. The new features are:
Along with this are some miscellaneous UI changes. The top menu bar is largely redundant for now. It's a placeholder for later when there isn't enough space to make a button for everything.
The screenshots are sent as part of the datafile. Due to the encoding that HWBOT uses, the datafiles would be really large. So I had to turn on compression. In other words, the older versions of the submitter will no longer work. So you will need to update to this one to make any new submissions.
Hopefully this won't be too problematic for everyone. Please let me know if there are any issues with this new version. Quite a few things changed and it's possible I broke something.
Here's what the latest version looks like: http://hwbot.org/submission/3189606_
for me the new Version is not 100% ok.
i can run ,but submition butten don't work.
and manuel submitting give : Invalid data file: Unable to decrypt the datafile
Ugh...
Manual submissions have always been disabled for reasons I mentioned in the other thread.
In what way is the submission button not working? It doesn't do anything when you click? When you click it, do you at least get the status update on the bottom right corner?
The behavior of that changed since the last version. Previously, the entire app would freeze while it sends out the datafile. But with screenshots, the datafile is much larger. So depending on your internet connection, it may freeze for a long time. So I changed the implementation so it wouldn't freeze but would at least display an indicator that it's still sending.
Submission worked here: http://hwbot.org/submission/3189693_massman_y_cruncher_pi_25m_core_i7_4500u_12sec_927ms
I'm super impressed with the integration, @Mysticial, brilliant approach to the integration problem. If all benchmarks would have a submit function like this, part-taking in OC would be so much easier!
That's a pretty low bar to be "impressed" by.
If I tried to present this to the UI folks back when I was at Google, they would've blown me off and told me to go back to programming.
In all seriousness. I'm also a user of the app. So if something doesn't feel right, or inconvenient, I'll tweak it until it does. After a few iterations of that, this is what it converged to (at least for my workflow).
The screenshot one was particularly tricky since I didn't want to over-complicate the UI. Screenshots are very invasive and can capture sensitive stuff. So I wanted the user to know exactly what he/she is about to send before actually sending it. And then there was the horrific side-effect of making the datafile really big and taking 20 seconds to send and freezing the app for the entire time. (My 2560 x 1440p monitor screenshots to a 2MB png.)
submition working whit out the screenhot.
but if i click take a screenshot furst and than submit. noting happens ,no message : sending datafile, pleas wait
So you do not see the message, "Sending datafile. Please wait..."?
It may take a few seconds before the message pops up. But if you have a slow internet connection or a really large screenshot, then the actual sending may take a while. It takes 20+ seconds for me if I try to upload a 1440p image. And my connection isn't the worst one out there.
It sounds like I need to make the send progress more explicit. That way even when it's not working, it's clearer where it gets stuck. Unfortunately, I can't do a progress bar since the Java network API doesn't seem to have any way to relay that information back to the caller.
yes i don't see the message, "Sending datafile. Please wait..."
i am alredy waiting 8 minits sind the last try
Try this one and see how far it gets.
Version 0.9.3.95: http://www.numberworld.org/y-cruncher/HWBOT%20Submitter%20v0.9.3.95.jar
I did manage to hack in a progress counter for the submission.
When everything works properly, you should see things in this order:
That last one will refresh every second until the submission is complete. Then it disappears. If any errors occur, there should be an error-box that pops up. If something crashes, you probably won't see anything and it will hang.
Anyway. I need to get some sleep. So I probably won't be able to respond for quite a while.
whil try it.after the 10b run
edit: the message, "Sending datafile. Please wait..." apears for a second or 2 and then noting.
whil try it now on a other system
I was going through the submissions and I noticed a number of multi-socket systems that have seemingly terrible performance. (Especially that 4-socket Magny-Cours Opteron.)
I'll go ahead and explain why this is the case. It will probably be obvious to those of you who are familiar with the topic.
-----
Why does y-cruncher (sometimes) suck on multi-socket systems?
This is due to memory access. Specifically, Non-Uniform Memory Access (NUMA).
y-cruncher can only run efficiently when the following assumption is true:
This is true for all single-socket systems as well as some of the pre-Nehalem dual-socket servers. But not on modern multi-socket systems.
On multi-socket systems, each processor socket has its own set of memory banks. A processor has fast access to its own set of memory. But if it needs to access memory that's elsewhere (on a different socket), it needs to go over the interconnect to get it from the other processor. So it's a lot slower.
In other words, the assumption that is critical to y-cruncher's performance is no longer valid. Some memory is faster, and some memory is really slow - hence "Non-Uniform Memory Access". If you have two sockets, half the memory will be fast and the other half slow. If you have a lot of sockets, then the vast majority of the memory will be slow with respect to each individual processor.
If you think that's bad, get ready for more bad news.
Operating systems are aware of the NUMA. So they try to be smart about it. When a program runs, it biases the memory in favor of the core that asked for it. This maximizes locality so that memory access stay within the same NUMA node. While this sounds reasonable for most applications, it actually backfires for y-cruncher. Unlike most programs, y-cruncher wants to use the entire system.
Some of you might have noticed that y-cruncher's memory usage is static throughout the entire computation. What's happening is that y-cruncher allocates all the memory it needs upfront and reuses it through the computation. And that's where the problem is. That allocation is done by a single thread. So the OS will put all of it on one NUMA node.
During the computation, y-cruncher spawns threads that run on all the cores and all the sockets/NUMA nodes. Since all the memory is on one socket, all processors from all the sockets will hammer that one socket. Not only is it overloading the memory bandwidth in that node, it's also swamping the QPI going in and out of that socket. Meanwhile, all memory on the other nodes are idle. In other words, a massive traffic jam while everybody tries to park in one garage while there are 3 others that are empty.
This is why the performance sucks on those quad-Opteron servers. It also affects Intel machines as well, but to a lesser degree since they seem to have better interconnects.
What can you do about it?
The biggest problem is the traffic imbalance. If your BIOS has the option to disable the NUMA, then do it. This doesn't actually disable the NUMA since the NUMA is a physical thing, but it tricks the OS into thinking there's no NUMA so it randomizes the memory allocations across all the nodes.
In Linux, you have a bit more control. The numactl package lets you run a program with interleaved memory. This also spreads out the memory across the nodes.
These tweaks will help y-cruncher run faster. But it doesn't completely solve the NUMA problem. There's still the latency problem, and even when the interconnect traffic is balanced out, it will still be a bottleneck.
Solving the NUMA problem can only be done by redesigning the program. That's obviously beyond the scope of benchmarking.
That said, it doesn't mean you should avoid multi-socket systems. A high-end dual-socket machine that is properly configured will still beat out all the single-socket setups - LN2 or not.
What makes y-cruncher different from programs like wPrime?
y-cruncher actually needs to use memory - and a lot of it. (Not that I needed to say that.)
When you will have time after finalizing Y-Cruncher launcher, it would be appreciated if you could give your talents to rebuild Hwbot Prime and Unigine launchers.
Open a Kickstarter/GoFundMe, we'll take care of you.
(Joke aside, if it happens one day, i would totally fund it ! )
one nice feature to UI , adding swap mode, because for total novice this will be very difficult to do via command line..
but so far very good..
I've thought about that. But swap mode is part of the custom compute menu. And that custom compute menu is f:D:oking complicated. So it will take a lot of work to mirror that menu into the UI.
Aside from that, there are some technical roadblocks. The custom compute menu is an interactive menu. Unlike the benchmark menu, you can't just fire a command at it and expect it to always work gracefully.
The custom compute menu will show memory calculations and warnings. It will also hide/disable options that aren't applicable or are incompatible with existing settings. And it will automatically adjust things in response to user-input.
But in order to do that in the UI, y-cruncher needs to be able to send information back to the UI. Unfortunately, that's not possible right now. And I don't know how to do it that atm. That said, I can try to design something around this limitation, but no promises.
guys i have an i7 3820 at 4.9ghz and im getting horrible times anyone knows why??
i get 3.265sec on the 25m
and 225.801 on the 1B
there are core i3 better then this =/
Sandy Bridge processors don't have AVX2.
thought so , maybe via a batch file?
I don't see how a batch file would help anything. What did you have in mind?
Alright... Someone in China was kind enough to give me remote access to a dual Xeon E5-2696 v4 (Broadwell-EP)
It's got 44 cores/88 threads and 768 GB of ram. And after playing around with it for a few hours, I have some benchmarks for it.
Does anyone mind if I lay waste to the leaderboards?
I feel very guilty just thinking about it since I wrote the benchmark.
Btw, getting the program to run efficiently on these high-end boxes is actually quite difficult. At this level, the program is very sensitive to a lot of things. Combine that with a half-dozen knobs to turn within the program and it's a very large search space to play with.
Can't wait to see the results
Fine...
I hope I don't get yelled at.
says:
Hello ,
On Win 7 , R5E , 5960X , show launcher Error on create data file , unable to create ... what happens ??
How can i solved it ?
Thanks !!
PD : On Win 7 , 2600k , not problem.
Just to make sure I know which error message you're getting. The exact text reads, "Unable to create datefile." right?
Do you see a file named, "datafile.hwbot" in the path where you are running the app? This error message will show up if the submitter app is unable to write to the directory that it's running from.
says:
Yes , the data file is missing ....
I download the benchmark and unrar file at HDD USB after I copied the directory to another PC on R5E and 5960k.
Posible is blocked from permissions ? How can I solved it ?
The JAVA version its for x64 but not the last 9.JRE its 8.JRE.
While I'm not 100% sure this is the case, I do agree that it sounds like the submitter can't write to disk at all. Where did you put the y-cruncher folder? If you copied directly into "c:/" there's a chance that it won't have permissions to write there.
Try copying it somewhere else on that system. If the problem persists, then I'll have to push out a new version with some additional logging to pinpoint the error.
says:
Solved , copied at C:\ , and dismark only read at directory.
Excellent. I'll make the error-message more specific for the next version. You probably won't be the last one to hit this.
It just occurred to me that not being able to write to disk will also prevent the submitter from running y-cruncher at all since it uses scripts.
says:
Thanks for support , is a good one benchmark ....
The beta competition has been winding down for the past couple weeks and it officially ends in two days. So I thought I'd shed some light on what's potentially coming next.
y-cruncher v0.7.1 has been in feature freeze for the past 3 weeks and I've started handing out a release candidate to a handful of people. In other words, it's almost ready. If any staff members are interested, PM me or shoot me an email.
The new version has a bunch of changes. The ones that are relevant to HWBOT are:
This version adds a lot of environment detection mostly for validation and to streamline the submission process. At the very least, the motherboard and # of processors will now auto-fill themselves (if HWBOT recognizes it). On Windows, it will also detect the individual memory modules, but HWBOT currently doesn't take this information. (I couldn't figure out how to do this on Linux.)
OS and reference clock detection is obviously for validation and to lift the Windows 8 and later restriction. However, I found out today that this might not be 100% reliable. Apparently my laptop's platform clock is neither HPET nor ACPI. y-cruncher doesn't recognize it and the submitter blocks me from submitting anything run on my laptop. After a bunch of Googling I still couldn't figure out what the hell it is. In any case, this is something that can probably be fixed on the submitter side. So I'm fine with rolling out y-cruncher as is.
Last are the performance improvements. They aren't massive speedups. (Mostly around the 1 - 5% range.) Last time I checked, Sandy/Ivy Bridge with AVX had the biggest improvement. But still nowhere near enough to be competitive with Haswell/AVX2. Those with Broadwell and Skylake will be able to run the new "x64 ADX ~ Kurumi" binary that utilizes the ADX instruction set.
Unfortunately, this means that v0.7.1 will not be speed consistent with v0.6.9 and older versions. So if y-cruncher is here to stay on HWBOT, this is something we'll have to live with since y-cruncher probably continue to get incrementally faster with major release. With respect to that, I have AVX512 binaries lined up and good to go. So expect a potentially "unfair" advantage for Skylake-EP and Cannonlake whenever I can get my hands on them. (Knights Landing also has AVX512, but that's still uncertain since the architecture is so drastically different.)
----------------
In any case, there are some open questions:
There's a new version of the submitter that will be released simultaneously with y-cruncher v0.7.1. Right now, that submitter is set to block all v0.7.1 submissions on Windows 8 or later if they aren't using HPET or ACPI. Should I extend this to block earlier versions of y-cruncher that cannot detect the clock? This means that the current version of y-cruncher will no longer be usable for HWBOT.
What about Linux? I have no idea how easy it is to tamper with the clocks in Linux. (I've never tried.) But it's certainly possible since Linux is open-sourced. So a capable kernel hacker can modify it in a way to trick y-cruncher's timings.
When should I actually release v0.7.1? Should I do it immediately after the competition ends? Or should I wait around a bit. (I'm not sure what usually follows a beta competition.)
Will the new version support saving data file (for later manual submission via hwbot site) like some other hwbot-integrated benchmarks?
No, but you can (almost) already do that by saving the validation file. I can't allow manual submissions until I have a mechanism to block submissions into the wrong category. (i.e. submitting a 25m into the 1b.)
Hi, I keep getting this error on my Asus X99e-WS with my 5960X anyone know what it relates to?
Modular Redundancy Check Failed
Exception Encountered: generic Exception
Error Code: 2
Regards,
Are you AVX-stable? Can you run Prime95 Small FFTs (version 28.x) for any amount of time?
Hi Mysticial,
Well I could not at the 4.75 Ghz I was on right now but on standard bios setup non turbo clocked anything the Prime 95 will run smooth it uses about 87% of the CPU usage . But there is exactly the same error on Y-cruncher. For some reason I can neither run GPUPI CPU mode. Meanwhile Geekbench will run as a charm with up to 99% CPU usage.
Regards,
Matsglobetrotter
Just to clarify that I'm understanding your response correctly:
If this is the case, then I have no clue. It sounds like something is very wrong with the system. Stuff not working at 4.75 GHz is reasonable. A lot of Haswell-e overclocks at 4.75 GHz are not AVX-stable and will instantly fail on anything that uses AVX (such as y-cruncher or Prime95 28.x).
But if things are failing at stock, then something is messed up. The 87% CPU usage thing is usually an indication that you have background programs running.
All correctly understood. Yes on the 87% i had not necessarily closed down everything just checking that it would be stable. Even had realtemp on just to see if there was unusual heatpatterns but looks normal. Anyway too bad, have given up on those Benchmarks on this setup.
I'm struggling with 25M scaling with scaling to 24 threads getting sub 39.85 % multi-core efficiency; with 1B it's better but I'm still only getting 75.56 %. Is there something more to it?
Not really. The 25M is so small that the amount of overhead needed to manage that many threads is significant compared to the amount of computation itself.
In other words, you're not benchmarking how fast you can compute Pi, you're benchmarking how quickly you can create and synchronize threads. Some OS'es do better than others - namely Win8 and Win10 are better at this than Win7 or Linux.
Thank you for pointing me in the right direction, I had at some high thread-count scores and I see that Windows 10 sits at around 60+ % efficiency while Windows 7 is generally a touch under 40 %,
I was trying to get my first score... but it wasn't as easy as I thought. All I get is this error message after pressing the submit button. Other HWBOT-integrated benchmarks works fine on this PC, so internet connection is not the issue.
I didn't actually see this until Massman sent me an email. I thought I get email updates to my own threads...
Anyways. What happened was that the benchmark got renamed when it was promoted out of beta. The when program tries to submit using the old name, the server rejects it.
I just pushed out a quick update to fix this issue. But it looks like there's more work to be done on my side of things.
You both have mail.
I'm trying to set up a redirect for legacy URL support
Thank you, it is working now.
I'd like to bring back the topic of HPET platform timers and open up a discussion about virtual machines (VMs).
y-cruncher submissions to HWBOT currently require either HPET or ACPI for Windows 8 and 10. But I've come to realize that there are other platform clocks out there. When you use bcdedit to force on the platform clock, you're not guaranteed to get either HPET or ACPI.
My very own laptop seems to use an unknown platform clock which I've been unable to identify. Since neither y-cruncher nor the HWBOT submitter is aware of this clock, it blocks me from submitting benchmarks from my laptop.
The broad problem is that only the TSC clock is affected by the clock skew on Windows 8+10. Does anyone mind if I change clock enforcement to blacklist TSC rather than whitelist HPET and ACPI? This should allow all other platform clocks.
The downside of this is that it also allows VM clocks. Some of the Linux benchmarks that I've gotten from people use the "xen" VM clock. But that seems to be mostly a Linux thing. VMs in general are difficult to tackle since they (by design) attempt to trick the software into thinking it's running natively. So everything from clocks to instructions can (theoretically) be faked. y-cruncher v0.7.1 will attempt to detect if it's in a VM. But it currently doesn't block VM benchmarks because:
Speaking of Linux, the submitter currently does not block any Linux submissions regardless of the reference clock. So you are (technically) allowed to run benchmarks in Linux, take a screenshot there, then transfer the files to Windows and submit to HWBOT. But given that there's no CPUz for Linux, this probably won't work for the majority of the competitions.
I've researched this topic for some days back when I was developing GPUPI and in my opinion the only manageable option for me was to ban TSC from 8 and 10 and only allow HPET there. Using ACPI as a timer if available is possible but depends on how it's done. I would not advise to use Windows' QPC functions for example.
ACPI + QPC is vulnerable to clock skew? I admit that I have no way to test this since the only box I have that lets me disable HPET (to force ACPI) is my AMD box - which isn't affected IIRC.
y-cruncher uses more than just QPC. So even if QPC gets skewed, it theoretically should still be okay. I can PM you the details if you're interested. Either way, I have no way to test this.
QPC falls back in various ways depending on the hardware and OS version, it can be HPET, ACPI or even RTC. The timer resolution of QPC differs greatly as well so it's difficult to find out which timing method is currently in use.
I think we might be talking past each other. I'm fully aware that QPC varies depending on the clock. That's sorta the whole point of this discussion. It's easy (for me at least) to determine whether QPC is backed by HPET, ACPI, TSC, or none of those. But you implied in your previous post that QPC is vulnerable to skew when backed by ACPI.
Oops, I meant that I am avoiding QPC when HPET is not enabled. Sorry, I have currently a lot on my plate.
I can't remember if it's precisely ACPI that's vulnerable, but on Windows 7 - which is not affected by the RTC bug - QPC gets skewed if HPET is disabled. My best guess is, that it falls back to ACPI, otherwise the fallback to RTC would not produce skewed results. See my results here: https://www.overclockers.at/articles/gpupi-2-1 ... I should have displayed the timer frequencies as well, hrmpf.
Btw, I've recently received some benchmarks on a 68-core (272 thread) Knights Landing Xeon Phi:
As far as I can tell, these are world records for single-socket processors. And despite running at a stock 1.4 GHz, it edges out all the LN2 systems.
It still falls short of the really big Haswell-E and Broadwell-E dualies. But the program is completely untuned for the Xeon Phi. So there's probably a lot more performance that can be squeezed out from just software alone.
I won't submit these to HWBOT since they were done in Linux using the unreleased AVX512-CD binary. The rules currently require Windows with a screenshot. And I prefer to keep benchmarks of unreleased versions separate from the released stuff.
The validation files for these with all the details are available from my website if you want to take a closer look. Just don't try to steal them and submit them to HWBOT.
The submitter app can't distinguish a legit benchmark from one stolen from someone else. So it will actually allow such a submission.
Hello, I've playing little bit for the benchmark. And i have some question.
1. For HWBot submission, It's allowed to run the benchmark via standalone binaries instead of HWBot submitter launcher?
And It's allowed to use Screenshot created by another apps for making the datafile?
Since both of them is possible for now, right?
In GPUPI / HWBot x265 saving the datafile will capture the screen & only the screenshot from datafile is valid.
2. The datafile is created by an TXT validation file + screenshot (or any images file
).
And created when clicked 'Submit to HWBot', deleted by the apps after uploading.
It's possible to directly make / save (multiple) datafile (With Screenshot included) after benchmarking?
I mean, after the benchmark finished, there's no TXT validation file appear first. And the datafile created when I click 'submit to HWBot'. And screenshot is captured & included in the datafile directly when I click 'submit to HWBot'.
I'm aware, if you implement this idea the benchmark must run from the HWBot Submitter. And my first question above is no longer valid.
or
I'ts possible to make multiple validation file (TXT) created by the binaries, to make datafile, after the benchmark running?
Which is easier?
My concern it's, we are talking on OCed system, that anything can happen suddenly. Corrupted TXT Validation file, problems when saving, crash, etc.
3. Why you make the datafile deleted after submitting to HWBot?
I think it's better to keep those file. Make good datafile naming mechanism / scheme. Such as (preset - time.HWBOT). It's related to my idea in 2nd question.
I still can keep it by copying the datafile when it was made though. But it's useless for now since manual submitting doesn't possible.
4. Make manual submit possible.
Many integrated HWBot benchmark could do this, HWBot Prime, GPUPI, HWBot x265 or even XTU, etc. With no issue i think.
It's funny (at least for me) if you willing to submit from another computer you must copy the entire folder which contains the application, and run the apps on another system just for submit. In fact in the end we had to open the browser also to fill in complete submission data form on the HWBot web to finish the submission processes
Many thanks, regards.
Yes. The submitter already lets you do this, but you're also free to manually run them. You are also free to fully customize the settings from within the y-cruncher application itself. You aren't required to run the benchmark from the submitter.
And for that matter, you may need to do this anyway since the default settings (as chosen by the submitter) aren't always the best.
(I have a feeling that some people won't like this ability to over-customize the settings. Since it means it's not 100% about hardware anymore. Someone with a sufficient understanding of the program itself and how it works will be able to pick better settings for their hardware setup. But when the goal is to be as fast as possible, I consider that as part of the game.)
The official rules say you need to use the built-in screenshot. But that's impossible to enforce since the built-in screenshot is no different from any other screenshot. (The submitter doesn't watermark the screenshot.)
No it's not possible. The submitter is completely separate from the main y-cruncher application. This is by design and I have no intention to merge them.
The y-cruncher application is primarily used outside of HWBOT and it has its own validation system. The submitter is only responsible for integrating with HWBOT. I admit that the integration is less than ideal, but it's the best that can be done without being too intrusive to the main y-cruncher project and with the limited man-power that I have.
Can you rephrase that? I don't think I understand. The benchmark already makes a new validation file for every computation. And the datafile is made by the submitter - which is after the computation is done running.
How is that different from any other benchmark? And isn't that just part of the game? Realistically speaking, if the overclock can pass a y-cruncher benchmark, it'll have a pretty decent chance of handling a screenshot as well.
Forget about the HWBOT datafile. You aren't supposed to know or care about it. If you're seeing it at all, that probably means that a submission failed. And when that happens, it won't delete it so that you can send it to me as a bug report.
Instead of the HWBOT datafile, you should care about the validation .txt file. With that (and a screenshot), you can submit from anywhere.
Right now, manual submission is disabled because:
Hi Guys....
i can´t submit any scores with hwbot submitter on Win 7 because the log Window don´t show me scores. .txt data file is still there
So whats wrong ?
Can you clarify the problem? If I'm understanding you correctly, the validation file is there (i.e. "Pi - 20160916-194531.txt" ). But it doesn't show up in the submitter?
Exactly.
Is there another way to submit without Hwbot submitter?
Cheers...
No there isn't. The submitter is the only way right now.
Can you PM me the validation .txt file? I'll take a look.
PM ed you :-)
I found the problem. The submitter is bailing out when trying to read the memory configuration.
It's supposed to look something like this:
Java's String.split() doesn't behave the way I expected it to when the fields are empty. The submitter skips any file that it has trouble reading so that it won't error things which are not validation files. Since it had trouble reading this file, it thinks it's not a validation file. Hence why it doesn't show up.
I'll roll out a fix in a bit.
Fixed rolled out. (HWBOT Submitter v0.9.5.107.jar)
In addition to fixing the memory parsing thing. I've also relaxed the reference clock restrictions. Rather than explicitly whitelisting only HPET and ACPI, it now only blacklists TSC. This solves a problem on my laptop which uses some platform clock other than HPET and ACPI. Hopefully this won't cause any problems.
thanks for your work man, i will check it.....
i also found something... i flashed back from Broadwell E to Haswell E Bios and it Works....
How can i save data file for future submit ? drivers for lan doesnt exist under XP.
The program won't run at all on Windows XP. Support for Windows XP was dropped more than 4 years ago.
You will need a minimum of Windows Vista to run the program at all.
You need Windows 7 SP1 to be at all competitive since that's the earliest version of Windows that supports AVX.
You'll want Windows 8 or 10 to get the best performance since the schedulers are much better.
But to answer your question directly, you can transfer the validation .txt file and the screenshot to a different computer and submit from there.
Yup, thx for reply, my question was little bit faster than i cant research it myself
offcourse afert a few hours left by installing useless XP here.
i seem to have run into this issue,
checking the file it does detect the memory config.
file attached
Thanks for the report. I'll take a look when I get home tonight.
That validation file works fine for me. The only thing unusual about it is that it's using an unrecognized platform clock. The newest version of the submitter will allow it, but older versions won't.
Are you using version 0.9.5.107?
yes 0.9.5.107
Well. That validation file works fine for me. I'm able to actually submit your score. So I can't reproduce your problem. Can you elaborate on exactly what the issue is?
Just a heads up to any staff members who might want to know...
I'm planning a March 14th (Pi Day) release of version v0.7.2 of the program next year. And it will not be speed consistent with the current v0.7.1 release. Let me know if this will cause any problems with any ongoing competitions that will cover that date. I can't say what the speed difference is since that isn't set yet, but it will be more than enough to trip up an ongoing competition.
The date isn't set in stone and will depend on the stability of the program in the preceding month. But it is the date that I tend to aim for since it's Pi Day.
I did actually make it in time. Version 0.7.2 has been released. Happy Pi Day everyone!
This new version is faster than the previous version on pretty much all processors that I've tested on.
For internal reasons, you are now required to use the latest submitter version (v0.9.6).
Here are some submissions from the Ryzen build that I used to tune for Zen.
Has anyone ever seen a machine with 6TB of memory?
No client version given, required version: 0.9.6 ?!
Using 0.9.6.114, submit through the interface.
I'll investigate tonight. I pushed an update a few days ago, but it's certainly possible that I broke something.
I can't reproduce the issue. Submissions work fine for me. Can you give me more information? Such a screenshot of the error?
If anyone's wondering about AVX512. It's coming... No ETA yet since I've hit a number of unexpected snags.
If you're wondering why the chip is underclocked. There's a reason for that. And it's a long story.
Don't worry, it's much harder to fry your chip with AVX512 than I'm making it sound - unless you disable thermal protection...
And version 0.7.3 is out.
And here are the first few submissions on a Core i9 7900X @ 4.0 GHz (all-core AVX512) and memory @ 3200 MHz:
The AVX512 didn't bring as much of a speedup as I'd hoped. But it's still enough to beat all the Haswell and Broadwell HEDTs and come within arm's length of the dual-socket servers.
Just a heads up, submissions are currently broken.
The HWBOT API seems to have changed without any warning: http://forum.hwbot.org/showthread.php?t=175630
So until that gets resolved, no submissions will go through.
I will create an internal task for richbastard, we just implement a new server and hwbot revision, might be because of this
This is getting fixed on both sides. richba5tard says a server-side push tomorrow will change the default from XML back to JSON.
At the same time, I've updated the HWBOT Submitter to explicitly request JSON.
Here the first submission of the new y-cruncher version that I released yesterday (and re-released just now with the fixed HWBOT Submitter):
http://hwbot.org/submission/3766703_
HWBOT Submitter shows all time, no Benchmark found. Latest Version 0.7.8 Y Crunsher
How to fix it ?
@P5ych0I found a workaround, here:
1) try to run manual once ( choose 0 -1- 1 ),
2) once it completed open the hwbot submitter , your previous result should be in there
3) Either press Run Benchmark or go to Options - Run a Benchmark - choose the 1B one
4) Select the run you want to submit - Submit to HWBot
Thanks Albrecht, i couldt fix it. WinRAR was the problem.
Problem with submit 25M ( but with 1B OK). Everything is correctly determined, but error with information:
"Sorry, there are a few errors in the form. Please fix them and submit again.
You are not using a compatible version of y-cruncher - Pi-25m: 1.0.2. You can download the latest version here: http://bit.ly/2l5VC1V"
Try edit result and:
"No client version given, required version: 1.0.1
25m is no longer an active category on the bot.
@Leeghoofd This question arises again and again. It might be best to enable 25M but don't give any points.
It is in Beta ( so no points ) , but I'll contact you on discord Mat , I can manually submit the datafile , but not via the automatic uploader.
EDIT: Small update on the HWBot API and it works, thx Mat
Thanks. It is working.
But: the general problem is that the chipset detection is wrong. When submit 25M, the screenshot was taken incorrectly. Added the correct one manually.
Please post the wrong screenshot so we can look into it.
BenchMate doesn't show any chipset information. Where is that detection and what's wrong with it?
Please log in or register to comment.