The 12th Annual Folding@Home Holiday Season Race

[petrusbroder]

Here are the stats for the 8th day of the 12th Folding@Home Holiday Season Race
Stats are as of December 8, 2017, 21:00 UTC (approx).
The inter-team race stats are from the Folding@Home stats page.
The intra-team race stats are from EOC.com.

The inter-team race:

The intra-team race:

The intra-team race daily production:

The intra-team race individual daily production:

Today is the first day that the Cancer Terminators produced more points than the Cancer Eradicators! Congrats!

 

[StefanR5R]

So, folding@home credits are currently not showing in the stats frontend for many CPU clients, presumably due to a firewall mis-configuration at temple.edu.

Here is a little estimation of Brony@Home's real daily output, based on EOC stats, which assumes that only Brony's user Nekolepsy is running F@H on CPUs (there may be others but I do not think that those change the numbers by much), and that Nekolepsy is really producing 12 M per day (which is just a guess and may be very wrong).

            Brony@Home    Nekolepsy    Nekolepsy    Brony@Home
EOC day     visible       visible      estimated    estimated
--------------------------------------------------------------
Dec 1        36.8 M         7.9 M        12 M         41 M
Dec 2        34.4 M         3.6 M        12 M         43 M
Dec 3        37.3 M         3.7 M        12 M         46 M
Dec 4        36.7 M         4.6 M        12 M         44 M
Dec 5        39.0 M         3.1 M        12 M         48 M
Dec 6        38.7 M         2.1 M        12 M         49 M
Dec 7        39.8 M         1.4 M        12 M         50 M
Dec 8        40.1 M         1.6 M        12 M         51 M

Regarding the total sum of credits granted to Brony@Home during the race: If temple.edu's firewall problem gets fixed before February 1, all of Nekolepsy's missing credits (of which some are for work from before our race started) will get counted towards his and Brony@Home's total in the race.

It is hard to estimate how much of that credit will be from work done in November, because stanford.edu's stats outage during November distorted the daily stats. I am guessing it could be 10 or maybe 15 M, but this guess could be far off.

As for our own stats, I had been producing about 1.5...2 M PPD on CPUs during a few days, and received 0.5...0 M credits/day for it. (And I am shutting down CPUs as I will add GPUs.) So this stats problem does not change our PPD much.

So, where do we stand with Folding@Home's stats service:

• A multi day outage at the end of November made pre-race estimations difficult.
• On normal days, stats on stanford.edu's all-teams list are different from stanford.edu's per-team lists, on the tune of a million, or a few million.
• Work delivered to one of F@H's sites (temple.edu) is not given any credit since end of November, presumably because an IT department at temple.edu which is not controlled by the F@H consortium changed firewall configuration.

So, maybe it is fair to say that F@H's accounting of donors' credits is a mess. But at least it is not as bad as with certain BOINC projects (Climate Prediction who didn't give credit for months; YAFU who give credit on a regular basis but cannot get the Credit New algorithm to converge; ...). But considering the size of the F@H project, I wish they would do better than they are currently.

 

[crashtech]

So their nice chart that I've been looking at all this time is pretty much worthless as far as knowing where we'll be when the other shoe drops. It's a false sense of security.

 

[IEC]

So their nice chart that I've been looking at all this time is pretty much worthless as far as knowing where we'll be when the other shoe drops. It's a false sense of security.

Yes, we are probably quite a bit behind, unfortunately.

 

[TennesseeTony]

Ah Dang. Triple 1070 system is down. Gonna be a long night.

 

[petrusbroder]

Yesterday evening about 1 hour before stats collection time:
Wife: "Peter, are ready, we need to leave for the dinner party"
Peter: "Was that today? Ready in 5 ..."

Thus: no stats yesterday.

 

[StefanR5R]

Ah Dang. Triple 1070 system is down. Gonna be a long night.

Spoiler: Speaking of a long night...

Yesterday I rebuilt my triple GPU host, which involved swapping 2x 1080Ti + 1x 1080 from different makers by 3 new* 1080Ti of identical make+model, swapping the case from another PC rebuilt one week earlier), and swapping the previously used 420 mm + 360 mm radiators for a MO-RA 360 external radiator and an internal 280 mm radiator.

[ *) The 2 old 1080Ti will go into a new host RSN. I also pulled 4 of the 8 RAM modules of the 3-GPU host, as a organ donation for two other PCs.]

It was a pain to get the cabling of the 9 fans in the MO-RA to work properly. Also, while the mounting of waterblocks to the 3 new GPUs went reasonably quick because I am now used to this procedure and did it for 3 GPUs in one go, it was still tedious with all those little screws to unscrew and screw back, and all those thermal pads to place...

So I filled water into the loop, which went quicker than I thought considering the large radiator, put the host back into operation, and applied an overclock even. (The 3 new GPUs seem to overclock a little better than the older ones.) All good, went to bed at 1 AM. I had a light sleep though and woke up at 3 AM again, and noticed a suspicious absence of coil whine. Monitor didn't want to switch on, radiator was cool, but the PC could still be pinged. I pulled another monitor elsewhere, found spare cables and Displayport converter, still no signal. Reset the host and luckily the original monitor came back on.

But the horror: Only 2 of the 3 GPUs were present. I shut the PC down again, tried for badly seated cards or cables, and perhaps the power cables on the missing GPU had indeed bad contact. Booted the PC, and the 3rd GPU was back. And so I was folding again at 4 AM.

 

[VirtualLarry]

Yay, I've got my RX 470 4GB, RX 570 4GB, and my GTX1060 3GB, all Folding, (along with two cores of an i5-7400). Not sure what my PPD is, but should be around 700,000-800,000PPD?

Also have a pair of R5 1600 CPUs running WCG, 10 threads each.

 

[Pokey]

Yay, I've got my RX 470 4GB, RX 570 4GB, and my GTX1060 3GB, all Folding, (along with two cores of an i5-7400). Not sure what my PPD is, but should be around 700,000-800,000PPD?

Also have a pair of R5 1600 CPUs running WCG, 10 threads each.

Which OS are you running the AMD cards with?? Windows or Linux?
I had a hard time getting them to work with Linux drivers so I gave up.

 

[TennesseeTony]

Triple 1070 is back up, with a twist: Nvidia driver 384 didn't want to run Folding this time, so I tried 387 drivers and instead of 1.7M ppd, I'm seeing 1.9M ppd. Maybe that "hot-dropable" hard drive issue was a good thing.

 

[VirtualLarry]

Which OS are you running the AMD cards with?? Windows or Linux?
I had a hard time getting them to work with Linux drivers so I gave up.

Just Windows. They are dual-use boxes.

 

[petrusbroder]

Nobel stats
Here are the stats for the 10th day of the 12th Folding@Home Holiday Season Race
Stats are as of December 10, 2017, 20:00 UTC (approx).
The inter-team race stats are from the Folding@Home stats page
The intra-team race stats are from EOC.com

The inter-team race:

The intra-team race:

The intra-team race daily production:

The intra-team race individual daily production:

The total production of all racers:

Some more numbers ...
This has been a great first third of the race. We are, as the numbers are now, some 19M points ahead of the Brony@Home team. The stats here set the difference at 15M points. The difference can be explained by the source of the stats (the folding@home different stats do not quite match and by the difference in the time stamp. As I see it, it does not matter if the difference is 15M or 19M.

The total production during the race (for both teams) is some 713M points and that is awesome. The biggest winner is of course Folding@home science.

Lets look at some other numbers:

• According to the Teams' page at EOC the production of the past 7 days for
  • Brony@Home 19 501 723 points and
  • for the TeAm .. 22 961 780 points.
• The TeAm has produced 311 262 507 points in the first 10 days,. This could mean that the month's production will be in the magnitude of 1000M points or more. This prediction is probably wrong, because I see from the thread that upgrades are ongoing ... in 2016 the production was 1 280 349 534 points and considering this, I would guesstimate the 2017 production should be closer to 1 500 M points. I hope that I am underestimating ... but this is how the production curve looks like
  (x-axis: race #, y-axis points produced by the TeAm). race #11 was in 2016, thus the race #1 was in 2005. The introduction of powerful GPUs makes a lot of difference...
  
  
  
• In the 2017 race there are 25 racers. This is how the number of racers have developed during the years:
  
  
  One can see that the racers are much more efficient now ...

Ok, no more to see, let us continue folding ...
Oh, by the way, those numbers above are out of date, as the WUs and points accumulate.

 

[VirtualLarry]

Ugh, when I got up this morning, my primary Ryzen 5 1600 box was rebooted, and not Folding (BOINC is set to auto-start, however), and I went out tonight, after re-starting everything, and came back home, and AGAIN, sitting at the desktop, with none of my customary programs running. Getting really annoyed. Reliability Monitor on Windows 10, just said "Unknown Shutdown" or whatever it says for a hard reboot or power loss. Not apparently due to Windows Updates.

I had clocked the CPU back down, after the first reboot, so now the CPU is at stock, (3.4Ghz with all-core Turbo, gets to 77C or so on 120mm water), and the RX 570 4GB, even with -10% or -15% power target, is pushing 89C. I really wonder if the fan is running on the video card? It seems to be...?

I DID recently update the UEFI on both of my currently-running Ryzen boxes, but the one with the RX 470 (air-cooled, stock heatsink, CPU at stock clocks), hasn't rebooted since I removed the OC from that one too, and updated the UEFI on that one too to newest.

 

[TennesseeTony]

In 3 days, we should overtake Tekzilla in overall points. And in 3 weeks, we should overtake Tom's Hardware (again).

 

[StefanR5R]

@petrusbroder, that's price-worthy work. Thank you.

--------
When we look at Brony@Home's numbers, let's keep in mind that stanford.edu's stats server currently misses credits for a large part of completed work from CPU clients, which for all we know affects Brony@Home much more than it affects us. My estimation is that Brony@Home's true PPD on December 2 already exceeded 40 M, and by now exceeds 50 M.

As per the linked foldingforum thread, the Folding@Home staff is aware of the problem but cannot give a timeline for its correction yet, because they need cooperation of an outside IT department. The expectation so far is that the missing credits will be awarded in one big update once temple.edu campus IT accommodates Folding@Home's site-to-site traffic requirements.

 

[Markfw]

Ugh, when I got up this morning, my primary Ryzen 5 1600 box was rebooted, and not Folding (BOINC is set to auto-start, however), and I went out tonight, after re-starting everything, and came back home, and AGAIN, sitting at the desktop, with none of my customary programs running. Getting really annoyed. Reliability Monitor on Windows 10, just said "Unknown Shutdown" or whatever it says for a hard reboot or power loss. Not apparently due to Windows Updates.

I had clocked the CPU back down, after the first reboot, so now the CPU is at stock, (3.4Ghz with all-core Turbo, gets to 77C or so on 120mm water), and the RX 570 4GB, even with -10% or -15% power target, is pushing 89C. I really wonder if the fan is running on the video card? It seems to be...?

I DID recently update the UEFI on both of my currently-running Ryzen boxes, but the one with the RX 470 (air-cooled, stock heatsink, CPU at stock clocks), hasn't rebooted since I removed the OC from that one too, and updated the UEFI on that one too to newest.

I don't know what tools there are in AMD's arsenal, but I would find one and set the video card fan to at least 75%. Also, just in case it does, disable the windows update service. It has killed me more than one.

 

[VirtualLarry]

rebooted again, right when I was sitting at the PC, using it, on a Skype conversation. Something's not right here. Using an Antec Neo Eco 650 or 620 that was in storage for a while, but I think that it is one of the ones made by SeaSonic. So I have a hard time imagining that the PSU is bad. I do have another Rosewill "80Plus Gold" PSU, 600W or so, to throw in to test that theory. Anyways, there's only a single GPU in this box, it has a 65W nominal TDP CPU (R5 1600), at stock clock settings. Ugh. Maybe I have a buggy CPU?

 

[VirtualLarry]

Bumped the RAM up to 1.35V, from 1.20V, even though it's only 2400 RAM, and the XMP profile is 1.20V.

Edit: Discovered a "Power Supply Current (something)" option, seems like a way to make non-"Haswell Ready" PSUs workable with these newer boards?

I changed it to be compatible with "typical" rather than "low current". Maybe that will fix my reboots. Here's hoping!

And if anyone happens to know if an older Antec Neo Eco 620 is "Haswell Ready" or not, chime in. I think this one was purchased over five years ago.

https://techreport.com/review/24897/the-big-haswell-psu-compatibility-list#antec

Antec Neo ECO 620C is... I wonder if this is a "C" model.

Edit: Just slid my PC out to look, while it was still on... and my 4TB HDD I just put in there last week, still spinning, I guess.

Anyways, this one does appear to be a 620C, so perhaps that's not the issue after all. Oh well, seemed like a simple fix.

 

[Markfw]

@VirtualLarry, I gave up on Antec PSU's years ago. I use almost only Corsair, and I do have a couple of Seasonic, but all of them (Corsair and Seasonic) are all gold or platinum series.

 

[VirtualLarry]

Well, I don't have a stable of high-end PSUs, really, although I do have some Antec Edge 750W Gold PSUs, a pair of them, unopened from a few years back. Those were supposedly SeaSonic as well as this Neo Eco 620C.

 

[Markfw]

Well, I don't have a stable of high-end PSUs, really, although I do have some Antec Edge 750W Gold PSUs, a pair of them, unopened from a few years back. Those were supposedly SeaSonic as well as this Neo Eco 620C.

Well, that sounds good. I no longer have any spares, all used up with all my video card machines.

 

[Markfw]

BTW, now there are no CPU units available. So I put 60 more cores on WCG

16:48:58:WU02:FS00:Connecting to 171.67.108.45:8080
16:48:58:WARNING:WU02:FS00:Failed to get assignment from '171.67.108.45:8080': Empty work server assignment
16:48:58:WU02:FS00:Connecting to 171.64.65.35:80
16:48:58:WU02:FS00:Assigned to work server 155.247.166.219
16:48:59:WU02:FS00:Requesting new work unit for slot 00: READY cpu:31 from 155.247.166.219
16:48:59:WU02:FS00:Connecting to 155.247.166.219:8080
16:49:00:WARNING:WU02:FS00:WorkServer connection failed on port 8080 trying 80
16:49:00:WU02:FS00:Connecting to 155.247.166.219:80
16:49:21:ERROR:WU02:FS00:Exception: Failed to connect to 155.247.166.219:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

 

[StefanR5R]

http://fah-web.stanford.edu/pybeta/serverstat.html shows a single SMP work server to be accessible (155.247.166.220), but its table column for available WUs is empty.

BTW, a week ago I tried to set up BOINC to pause if there is x % non-BOINC work running, with the goal that BOINC crunches WCG everytime FAHclient doesn't receive work. Didn't work as intended, as BOINC still ran even if FAHClient had CPU work too. More research needed.

 

[crashtech]

BTW, a week ago I tried to set up BOINC to pause if there is x % non-BOINC work running, with the goal that BOINC crunches WCG everytime FAHclient doesn't receive work. Didn't work as intended, as BOINC still ran even if FAHClient had CPU work too. More research needed.

I've never been able to get that feature to work, even after playing with process priorities.

 

[petrusbroder]

Here are the stats for the 11th day of the 12th Folding@Home Holiday Season Race
Stats are as of December 11, 2017, 20:15 UTC (approx).
The inter-team race stats are from the Folding@Home stats page.
The intra-team race stats are from EOC.com.

The inter-team race:

The intra-team race:

The intra-team race daily production:

The intra-team race individual daily production:

Those are the Monday stats.
The difference between the two teams is just above 2%, i.e. less than 7.5M points, i.e. less than 19% of one day's production ...I like this ...