Discussion Rudi_Float_Bench v0.02a

[igor_kavinski]

Now if we can talk @Det0x to do it on his 9950x !

No way his CPU is beating yours. I would be seriously surprised if he can

I mean it's basically 32 threads at roughly 5.7 GHz. There is a slight chance but I haven't checked how this benchmark scales with SMT so that's an unknown. All I can think of is that he has 16 real cores vs. 64 of your cores. That's a crazy comparison.

 

[Markfw]

No way his CPU is beating yours. I would be seriously surprised if he can

I mean it's basically 32 threads at roughly 5.7 GHz. There is a slight chance but I haven't checked how this benchmark scales with SMT so that's an unknown. All I can think of is that he has 16 real cores vs. 64 of your cores. That's a crazy comparison.

SMT OFF for the last run of 2308. See the picture.

 

[igor_kavinski]

SMT OFF for the last run of 2308. See the picture.

My guess is that on Zen 2 and Zen 5 server CPUs, the FPU units get clobbered with too much context switching (pausing the real thread, switching to virtual thread and so on, for who knows how many times per millisecond) which decreases their throughput with SMT on.

Need to investigate the SMT/HT impact of other x86 CPU models/architectures to see if there is some design that manages a net benefit with SMT on.

So I'm betting that Det0x will also get his best score with SMT off.

 

[igor_kavinski]

@Micrornd , hoping to see your HT on and off scores

 

[igor_kavinski]

Overclock.net user Veii's 285K result:



Fun fact: Those asterisks and pipe characters? Those are the number of real cores in the CPU. Unfortunately, no plan to display HT/SMT virtual cores. That's not gonna be a trivial task. Apparently, all threads ARE recognized by the benchmark, including SMT/HT threads.

User SizzlinChips' 285K result:


Seems Arrow Lake is tricky even with microbenchmarks to get consistent performance out of.

 

[Det0x]

Now if we can talk @Det0x to do it on his 9950x !

There is a slight chance but I haven't checked how this benchmark scales with SMT so that's an unknown. All I can think of is that he has 16 real cores vs. 64 of your cores. That's a crazy comparison.

My guess is that on Zen 2 and Zen 5 server CPUs, the FPU units get clobbered with too much context switching (pausing the real thread, switching to virtual thread and so on, for who knows how many times per millisecond) which decreases their throughput with SMT on.

Need to investigate the SMT/HT impact of other x86 CPU models/architectures to see if there is some design that manages a net benefit with SMT on.

So I'm betting that Det0x will also get his best score with SMT off.

Kinda strange benchmark, i'm seeing almost 100% thread scaling on 16 core Zen5 with SMT enabled/disabled

16/16 = 1031 points


16/32 = 1991

 

[Markfw]

Kinda strange benchmark, i'm seeing almost 100% thread scaling on 16 core Zen5 with SMT enabled/disabled

16/16 = 1031 points
View attachment 119285

16/32 = 1991
View attachment 119286

It works better with SMT !! I think there is a problem handling more than 16/32 cores.

 

[LightningZ71]

What's the memory footprint of this bench? Usually, when I see a task get almost no benefit from SMT, it's because the cache footprint of two instances of the thread is too big for the same cache level as only one instancefor example, if one thread has a footprint of 192KB on a processor with a 256KB L2, one thread will run completely out of L2. When the second thread starts on the core with SMT enabled, now there's 384KB of data trying to fit in 256KB of cache. It's going to constantly be swapping data in and out of L2. When you see it really impact older cores with smaller L2, but fly on newer ones with larger L2, that's often verification. The same effect can be seen on tasks that are small enough to live in L1 alone, but spill with two copies.

 

[Schmide]

Visual Studio 2022 is apparently putting in an alternate AVX2 codepath. Seems they wisened up and did the right thing, for once.

The actual hot loop code:

View attachment 119235

An issue here? The loop is polling the high resolution clock which generally has a period of std::ratio<1,1000000000>. So at best it can operate at 1ghz and your loop can queue operations at a higher rate. Moreover its operation is most likely a fenced memory operation meaning it runs and reads from outside the cache.

Maybe have a subloop that iterates at a higher rate than the clock check.

i.e.

volatile float va = 1.0f; // should prevent optimizing the loop 
__m512 a = _mm512_set1_ps(va);
__m512 b = _mm512_set1_ps(0.0f);
int subLoop = 100;
do {
   b = _mm512_add_ps(b, a);
} while (--subLoop);
va = *reinterpret_cast<float *>(&b);

// then check clock

 

[igor_kavinski]

What's the memory footprint of this bench?

 

[igor_kavinski]

Maybe have a subloop that iterates at a higher rate than the clock check.

Thanks! Will run some tests to see the difference.

Do you think there is some issue in the following code that is preventing the benchmark thread from getting loaded onto cores >32?

 for (DWORD i = 0; i < sysInfo.dwNumberOfProcessors; ++i) {
     threads.emplace_back(BenchmarkThread, std::ref(totalOps));
 }

 

[Nothingness]

Do you think there is some issue in the following code that is preventing the benchmark thread from getting loaded onto cores >32?

 for (DWORD i = 0; i < sysInfo.dwNumberOfProcessors; ++i) {
     threads.emplace_back(BenchmarkThread, std::ref(totalOps));
 }

Random thought from someone who has no Windows programming knowledge: does that sysinfo structure work for systems with multiple sockets?

 

[Schmide]

Thanks! Will run some tests to see the difference.

Do you think there is some issue in the following code that is preventing the benchmark thread from getting loaded onto cores >32?

 for (DWORD i = 0; i < sysInfo.dwNumberOfProcessors; ++i) {
     threads.emplace_back(BenchmarkThread, std::ref(totalOps));
 }

If you're threading and counting operations you need to use an atomic_ref. (or int)

Good example https://en.cppreference.com/w/cpp/atomic/atomic

Edit: note. Using atomics will often flush certain areas and affect total throughput. So in general do a fair amount of work, then atomic sum.

 

[igor_kavinski]

Random thought from someone who has no Windows programming knowledge: does that sysinfo structure work for systems with multiple sockets?

Yup. Works on a dual socket Ivy Bridge-E server and runs on all 24 threads ( 6C12T + 6C12T ).

 

[igor_kavinski]

Good example https://en.cppreference.com/w/cpp/atomic/atomic

atomic is being used: "void BenchmarkThread(std::atomic<uint64_t>& totalOps)"

What I don't understand is, why is there an invisible limit that causes the benchmark thread to refuse to go above 64 threads?

See below:




EDIT: typo. Wrote 32 instead of 64 threads. The limit seems to be 64 threads.

 

[igor_kavinski]

Maybe have a subloop that iterates at a higher rate than the clock check.

Sadly, the score dropped to below 300 on the 48 thread Xeon

Looks like gotta leave v0.01 as it is and try to change the approach for the next version (maybe direct comparison between AVX2 and AVX512 to see how much speedup AVX512 gives).

 

[Schmide]

What version of c++ are you using ?

 

[igor_kavinski]

What version of c++ are you using ?

Gonna try different C++ version builds later tonight to see if any version fixes this anomaly.

 

[Schmide]

Are you compiling to a 64bit binary ???

 

[igor_kavinski]

Are you compiling to a 64bit binary ???

Yes.

 

[igor_kavinski]

Maybe have a subloop that iterates at a higher rate than the clock check.

You were right!

The mistake I made was failing to increment the ops in the subloop.

Performance increased four times!

Just going to run a few more tests before dropping the AVX-512-only binary.

 

[MS_AT]

An issue here? The loop is polling the high resolution clock which generally has a period of std::ratio<1,1000000000>. So at best it can operate at 1ghz and your loop can queue operations at a higher rate. Moreover its operation is most likely a fenced memory operation meaning it runs and reads from outside the cache.

Maybe have a subloop that iterates at a higher rate than the clock check.

i.e.

volatile float va = 1.0f; // should prevent optimizing the loop 
__m512 a = _mm512_set1_ps(va);
__m512 b = _mm512_set1_ps(0.0f);
int subLoop = 100;
do {
   b = _mm512_add_ps(b, a);
} while (--subLoop);
va = *reinterpret_cast<float *>(&b);

// then check clock

I would double check with the diassembly. You can use https://godbolt.org/ for that, if diasebling the actual binary would prove problematic.

I mean since the loop does not touch the volatile variable compiler should be able to replace the loop with the constant unless the reinterpret_cast is confusing it.

And if the loop is not optimized away the loop carried dependency will not allow hitting optimal performance, since every iteration depends on results of previous iteration. Since most cpus are able to execute 2 fadds per cycle, second unit will not be used in parallel. Introducing another accumulator could help with this. Then you could try to account for add latency but since I guess the benchmark is trying to be generic, that would be going too far.

The simplest way to ensure the compiler will not optimize the loop away is to store accumulators to memory, ideally outside the loop. Load the initial value also outside of the loop from memory so compiler won't be able to assume anything about the data.

 

[Schmide]

I would double check with the diassembly. You can use https://godbolt.org/ for that, if diasebling the actual binary would prove problematic.

I mean since the loop does not touch the volatile variable compiler should be able to replace the loop with the constant unless the reinterpret_cast is confusing it.

And if the loop is not optimized away the compiler won't be able to unroll it due to carried loop dependency. Since most cpus are able to execute 2 fadds per cycle, you are leaving performance on the table. Introducing another accumulator could help with this. Then you could try to account for add latency but since I guess the benchmark is trying to be generic, that would be going too far.

The simplest way to ensure the compiler will not optimize the loop away is to store accumulators to memory, ideally outside the loop. Load the initial value also outside of the loop from memory so compiler won't be able to assume anything about the data.

The loop is dependent on the volatile so it will be unable to reduce it without treating it as a constant.

Compilers generally will not reduce intrinsics as they are explicit instructions.

https://godbolt.org/z/YzsE7zhhM

gcc unrolls it twice. clang unrolls it 10 times.

The loop is a chain. Each instruction is dependent on the previous one, so there is no way to eek out more performance.

It seems godbolt doesn't execute avx512

 

[MS_AT]

The loop is dependent on the volatile so it will be unable to reduce it without treating it as a constant.

Compilers generally will not reduce intrinsics as they are explicit instructions.

https://godbolt.org/z/YzsE7zhhM

gcc unrolls it twice. clang unrolls it 10 times.

The loop is a chain. Each instruction is dependent on the previous one, so there is no way to eek out more performance.

It seems godbolt doesn't execute avx512

Yup, my bad, writing after hiking is apparently bad idea. What I wanted to say by introducing another accumulator you could eek out more performance since right now your code read and write to zmm0 every cycle. Since this is read after write OoO engine cannot do anything about it. If you lowered the loop count to 50, and introduced c variable (being handled exactly as b), then it will use more registers. And since usually you have more than one add unit it will do that in parallel.
I have edited my post for clarity. I hope I have not messed anything again.

 

[igor_kavinski]

v0.02 with an additional AVX-512 specific binary (it will either crash or exit unexpectedly if the CPU is lacking the necessary ISA extensions): https://drive.google.com/file/d/12RuZsWdhNueu7th2HCuzblBA8CUGFu9u/view?usp=sharing




Please do test and let's compare some scores