Asus Z10PE-D8 god box

[Jay V]

Et Ali

I used an Asus Z10PE-D8 WS as the backbone to a god box.

Asus has bios 1202 out which also fully support m.2 pcie 3.0 drives as well as Sata Express drives.


I was using the Z9PED8 in the prior incarnation, this is the new box based on the Z10

For video I use 4 titan x SC (quad SLi) so I had to accommodate the four cards on the board as well as the dual cpus, I had to really plan out the cable selection and cable management for a relatively small enclosure.

I run at 4096x2160 true 4k and I like my games to be full eye candy and I like to run a lot of VM for the various things I do for work and pleasure. Unfortunately the box also sees a lot of work

While at first glance one would not notice, but there were a lot of details worked out in this build.
Getting the Z10 and all the accouterments into a relatively tiny box was somewhat challenging.

Warning, I've been through 4 of the Asus Z10PE-D8 WS motherboards: the prior 3 were board revision 1.00, 1.02, 1.03 and now I'm on version 1.04. The last fixed some underlying issues so if you are contemplating a build, please try to get board version 1.04

Well here are my specs and some pictures, please enjoy

thx

J

--------->


Asus Z10PE-D8 WS dual mobo (version 1.04, bios 1202)
(2) E5-2699 v3 (36cores/72HT) (O.C. 2.6/4.0 Turbo) (sequential sr numbers)
256GB LR Samsung ram DDR4 ecc reg
(QUAD SLI) 4 Titan X SC
Corsair Aix1500 DIGITAL PSU
1 Samsung M.2 SM951 pcie 3.0 (os drive)
8 Samsung 850 Pro 1TB SSD RAID 0 (apps drive)
9 noctua fans (almost silent)
Corsair Aluminum Red LED Keyboard, red level 10 mouse
LG 31MU97z 4096x2160 true 4K at 60hz dp
CaseLabs Mercury
2 modified Scythe ninja 3 rev b heatsinks
Thermal Grizzly Kryonaut TIM
Windows server 2012 R2, Ubuntu 15, LinuxMint
1lb. of flesh and a pint of blood

OC rock stable up to 109mhz using bios profiles, etc etc

anyhow, here are the final build pics




















continued:

 

[Jay V]

continued:









I took care of a couple of hot spots on the board by affixing some small heatsinks to the area. I also as you can see from the picture, added some heatsinks to the samsung m951 m.2 drive. Those heatsinks took the samsung from 126F down to 101F during game play and heavy use.



The case design allows the motherboard to be vertical, sad arrays and power supply are on the lower level with the motherboard on top

There are 9 fans

6 are in the upper shelf blowing air down.

there are NO fans on the sides, front, back or bottom.
Just the top

Fan speed is controlled by motherboard only

3 fans are on the heatsinks to the cpus
Cpu0 has only 1 fan on front of heatsink
Cpu1 has two fans, 1 front blowing, back sucking
the middle fan acts like a step fan keeping airflow concatenated

air is continually fed down from top bathing whole cube, pushing air out the venting and pushing air into 4 gpu cards, each blowing air out the back of the case.

I have no exhaust fans, top of case creates constant pressure head

all fan rpm's are from 490-580rpm at idle
Load may cause cpu fans to go to 620-950rpm (so far)

GPUs cause the more moderate noise, but it’s not a fan vibrating noise, it’s more like a whoosh sound.

So switching from CPU water cooling to air has not added any additional sound


Power supply has a fan in it but that seldom ever comes on

All fans are Noctua 2000rpm ip97 PWM


TIM is Thermal Grizzly Kryonaut (the best other than liquid metal) I have found this to an important step

Heatsinks are modified Scythe Ninja rev b for lag 2011 - and I tweaked them
Heatsinks are the biggest most massive that I could fit in this board. Also they are the only heatsinks that are 116mm across and have double the cooling pipes of the nearest competitor. See prior post pictures for up close detail



System is not audible in normal conversation, it is audible when no one is up, traffic is not going on , and barely audible when house is silent.

So the answer is "barely" audible.


All temps measured with AIDA64

I included the aida64 reading after 20 min of moderate load. heatsinks were lukewarm to the touch

room is was 75F +/- 1.5F

continued:

 

[Jay V]

continued:










One of the major improvements for me was the move from a vertical/pedestal environment to the cube/horizontal environment

Air management is easier on cooling board components and cpus



Synthetic BENCHMARKS (starting to)

I've run a few benchmarks that make more sense to me but I'll run some of the more generic ones so people can have relevant comparisons

example here is Cinebench 15, however(1) it should be noted that cinebench may see all the cores but only utilize them 1-12% so it’s not a relevant example other than a synthetic benchmark. (2) cinebench does not invoke the turbo mode out of the intel Xeons, so the score is running at the base 2.3ghz

Cinebench cpu scores on windows 764 were:

4788
4801
4721
4696

so it also lacks consistency, but here is a pick if the last one from the prior 764 build:





On win srv 2012 r2 and the scores are actually higher, so I don't really see the synthetic benches as accurate or consistent

Win2012 r2 cinebench 15

4861
4804
4799
4830

Hence my lack of faith in Cinebench 15. It is however handy for some quick checks


I'm currently benchmarking the system using AIDA64 and I'll post those scores as I wrap up.


I'll run some gaming benchmarks in srv 2012 r2, however those also are not as relevant. "Heaven" for example runs a high score, median score, low low high. Every time, it’s a reproducible pattern, so I'm not sure why the software mimics that pattern but it does, and it’s not load or temperature dependent.

The problem with video benchmarks is that not everyone benchmarks the same parameters for an apples to apples comparisons (exception is locked parameters such as cinebench)

Example, I run at real 4096x2160 with max eye candy, occlusion, parallax, ssao, AF16. FSAA etc. The gaming benchmarks will be skewed at the very least by the fact it’s at real 4K, but I'll run those shortly. I also edit and modify games to show all it can and I modify the cfg and ini etc to use more threads in both physics and cpu. I also script it to run cpu affinity for real cores (not HT) and all cores (real cores and the HT threads)

every says they have 4k, but what they are referring to is 3840x2160, my monitor and set up is 4096x2160.

The benchmarks I like are the built in Metro Last Light, the Painkiller Black edition built in bench, the doom 3 built in bench, the FEAR built in bench etc


SLI profiles are custom with max visuals and tweaked, in most cases I run AFR2 4GPU SLI mod profiles with the multi-threaded optimization

continued:

 

[Jay V]

Continued:

Game BENCHMARKS, all at 4096x2160 maxed visuals, maxed threads, maxed physics (again, parameters are well beyond the game settings menu)


Doom 3, modified, cfg maxed, using full suite of higher textures, higher lighting, rag doll, liquids, ssao, occlusion, adv physics mods: 97FPS



Painkiller Black cfg maxed 424FPS

Cryostasis, full physics, threading, full eye candy, textures, etc 291FPS

Wolfenstein 2009, cfg maxed, havok physics threading, game threading, max eye candy, veil fully rendered: 147FPS

FEAR cfg maxed, ssao, physics, occlusion, true 4k, liquids, lighting, destructive physics, particle motion etc etc Min 146FPS, Max 281FPS, Av 188FPS




I'm currently tweaking "Outlast", and I'm tweaking the Unreal 4 engine "Kholat" game. The new engine can really take advantage of the hardware if configured correctly.


At the end of the benchmarking the video card temp high was 69-71C (158F) the cpus highest was 41C (105F) ram temp was 44C (111F)

there are other types of benchmarks, that are more important to me

On Games:
none of my games are from steam or EA or Ubiplaly, or Blizzard. I don't like to play games that require me to connect to a service or require their client running. I bought all my games, and enjoy them well. I've been using GOG recently since the titles run clean, launch instantly and don't require the steam/ea/blizzard/ubiplay stuff. Except for few titles I've been able to get everything I want that way.

In addition I end up modifying ini, cfg, etc files so that game can run at 4096x2160, use as many cores as possible, thread physics as much as possible, add eye candy, enhance textures, change rag doll physics, lighting, ssao, occlusion etc etc.

I've no issue running any game on server, occasionally I have to edit the msi with stuff like orca, to gat it installed. OR I install it in a win 7 64 vm, and then copy the game directory, registry settings over to real OS.

DATA read write Throughput -->

example, using MACRIUM Reflect Server version, I can compress and back up 1.9TB full system and data in 6 min (6:18) to a full self-restoring image.



and I can restore full windows, apps, and mail data blank to fully loaded in about 1 1/2 minutes (1:23)




So I really can't complain about the speed from the m.2 drive and the built in sata raid features.

The Asus Z10PE-D8 WS has a jumper for LSI raid or for INTEL raid. I benchmarked both and looked at their features. The LSI raid requires install drivers as well as the fact that it 1/2 the throughput of the INTEL raid. Also, the intel raid supports TRIM down to the individual sad drives but the LSI raid DOES NOT. So beware of that issue.


In addition I have moved to Win SRV 2012 R2 from Win 764 ultimate.

Took a few registry hacks to get the classic desktop and
about the same privacy as 7-64
I used various built in tools it's a workstation and so far I'm content
I've got a few days of tweaking ahead of me.

All games installed ok, a few fps of improvement across the board
Also was able to do GPT and M.2 boot without workarounds

Verified file transfer speed, sustained, 2.3 GB/Sec on the 2012R2
Back up transfer speed says 13.5 Gb/Sec but I'll have to test, not sure if it can be trusted yet.

Compared to 764 ein 2012r2 offered a few improvements:
a. I have some hardware in my build that benefitted from the 2012r2, mainly the m.2 drives, and the new sata express. I was using several macrium tricks to re-install from a back up to do the gpt and m.2 as the C drive for boot as well.

b. There are a few settings in 2012r2 that benefitted from the e5-2699 v3 and the z10 when using "cache on Die" in the memory handler. Do not use early or late snoop.

c. and there are some benefits to using Hyper V in a very controlled fashion.


Do these benefits rank high enough to outweigh the win 764 build? - probably not...

I have my win 764 build saved off and I can in 5.33 min have it back on the system.

For now I'm tweaking and exploring the 2012r2 and tweaking Ubuntu 15, and my known addiction to DRM free shooters in high rez to incorporate into the fun


I'm getting two additional perks now that I've had it running for a week:

1. Specific games that have multicore and multi gpu malleability benefit from running on 2012r2. This might be how 2012r2 makes the environment available to an app. DX11 games in combination with NVIDIA drivers really saw a bump. Example Metro Last Light Redux built in benchmark had a performance increase in Max frames by 34 f/s, increase of 17 frames in Av f/s and raised the min f/s by 28.
Open GL games perked evenly by several f/s.

2. Load and precaching time on game levels has dropped noticeably. I have not benched this aspect since its hard to quantify with consistent results.

3. Compared to 8.1, the tile streaming, and other user burdens, were automatically not installed. In server manager you add and subtract features very easily, much easier than win 8.1 and more control than 8.1. In fact, having server 2012 r2 as a workstation was cake.

4. Boot and shutdown time - this is relative since I have a longer power to loader moment due to the nature of the power cycle of the z10ped8ws. However, once the video card displays the first part of the bios/uefi splash, the os boot time is about 7 seconds to login. That is considerably faster than 764 which was more than 30 seconds. The shutdown process is about 1-3 seconds to all power off.

I started easily with a site to guide my initial efforts. Thank you to several forum member for suggesting http://www.win2012workstation.com as it did make it much easier.

Pain in the arse issues that have been solved:
1
Office protection platform service kept running after you open outlook. Fixed by registry entry
Key: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\OfficeSoftwa reProtectionPlatform
ValueName: InactivityShutdownDelay
ValueType: DWORD
ValueData: The number in seconds, that the Office Software Protection Platform will stay running for, after the most recent activity.

I set mine to 30sec

2 Windows update service keeps running unless I set it to disable. usually I have it set to manual and when I manually go and do updates and select then it will start, after a reboot its gone no prob. However, now with 2012r2, I have update set to never and service set to manual (so I can do it on my terms) but the update service on reboot is running. In task scheduler I removed the triggers but they magically reappear in task scheduler after a reboot. So now I have the service disabled until I figure out how to keep it on manual and keep the pesky triggers out of task scheduler.


Continued:

 

[Jay V]

Continued:

In comparison to the Asus Z10PED8 WS, there have been significant changes in motherboards, but not many if you consider the amount of time that has passed since I first got into this around the mid 90's.

I've been building my own personal workstation since 1998. In 1999 and 2000 was water-cooling a dual cpu rig with raid. I even water-cooled the video card back then. I had to use dual power supplies at that time. I used to run an obsidian x-24 card then moved up to 3d wildcat dual GPU/CPU video card etc. I did one of the first sli rigs on a test mule than mobo. Then moved to an 8 core dual 2 motherboard rig from tyan. Lots of nvidia developer systems etc, and

This just happens to be the 7th incarnation of the original build. Great for games and when I have to do work from home.


Anyhow, here are some of my first PCs from 15+ years ago
















earlier this year water-cooling





The last one was my water-cooling attempts on the Z10PED8 WS build. While it worked well, the results were not any better than air cooling and not with the maintenance and complexity of the liquid cooling. So not the box is completely air cooled.

I hope that helps on initial benches and perception, and forgive my long winded explanation, the devil is in the details. The Asus Z10 is the best motherboard I've used so far and seemed fitting in my attempt at a god box for both heavy work and some gaming.

So has anyone else done a Z10PED8 or Z10PED16 build or even a Z9PED8 build, what challenges did you run into?



Ciao


J

 

[Donanza]

Those gpu are winking for the camera I believe.