Flaws on Oxide

[jj109]

You can enable it in the .ini files.

I tried. It just keeps crashing on load . What are your guy's windows versions?

 

[sontin]

I tried. It just keeps crashing on load . What are your guy's windows versions?

Windows 7 64bit, GTX780TI.

 

[BrightCandle]

Window 8 64 bit

 

[Imouto]

With

Average Batch Count:        14166
Maximum Batch Count:        79900

Without

Average Batch Count:        28048
Maximum Batch Count:        151054

It's not DC which is broken.

Please explain.

 

[sontin]

Ask the star swarm benchmark guys.

 

[BrightCandle]

Deferred context enabled and motion blur disabled, otherwise extreme:

Test Duration:			360 Seconds
Total Frames:			36967

Average FPS:			102.69
Average Unit Count:		4818
Maximum Unit Count:		5990
Average Batches/MS:		275.47
Maximum Batches/MS:		1897.88
Average Batch Count:		3113
Maximum Batch Count:		26889

Deferred context disabled, motion disabled, otherwise extreme:

Test Duration:			360 Seconds
Total Frames:			13583

Average FPS:			37.72
Average Unit Count:		4057
Maximum Unit Count:		5383
Average Batches/MS:		608.67
Maximum Batches/MS:		1814.12
Average Batch Count:		17666
Maximum Batch Count:		88308

I could clearly see the second one was struggling badly from the very beginning. The second has minimums around 6, the first more like 30 and the maximum FPS on the second scenario was more like 120fps whereas it was regularly 190 with deferred context.

The problem of course, the flaw really, is that the speed of the benchmark depends on the FPS, so the benchmark without deferred context gets less far, has less units but seems to have significantly more batch count. This suggests that deferred context also allows them to use a smaller batch count to render, for whatever reason. Because the first benchmark with deferred has more units it seems reasonable to come to the conclusion that the batch count is affected by deferred context.

One other note is that the impact of deferred context is enormous, its thoroughly unacceptable at this point for AMD to not have implemented this in their drivers. Its performance improvement is significant in this benchmark and if AMD would get it working all DX games would potentially benefit enormously (we have 300% performance improvements there).

 

[Gloomy]

Deferred context enabled and motion blur disabled, otherwise extreme:

Test Duration:            360 Seconds
Total Frames:            36967

Average FPS:            102.69
Average Unit Count:        4818
Maximum Unit Count:        5990
Average Batches/MS:        275.47
Maximum Batches/MS:        1897.88
Average Batch Count:        3113
Maximum Batch Count:        26889

Thanks. Very nice FPS. Can you do one without DC and without blur? Just for comparison's sake.

 

[BrightCandle]

Thanks. Very nice FPS. Can you do one without DC and without blur? Just for comparison's sake.

Beat you too it, see above edit.

 

[sontin]

In the RTS mode DC is sometimes more than twice as fast.
I find it hilarious that such a quick and dirty implementation is able to boost the frames in this way and at the same time a person calls DC "fundamentally broken".

/edit:

WO DC

== Results ================================================
Test Duration:            120 Seconds
Total Frames:            1607

Average FPS:            13.38
Average Unit Count:        3213
Maximum Unit Count:        5284
Average Batches/MS:        1148.15
Maximum Batches/MS:        1564.52
Average Batch Count:        85549
Maximum Batch Count:        161617
===========================================================

W DC

== Results ================================================
Test Duration:            120 Seconds
Total Frames:            2128

Average FPS:            17.72
Average Unit Count:        3649
Maximum Unit Count:        5508
Average Batches/MS:        1760.18
Maximum Batches/MS:        2602.41
Average Batch Count:        96509
Maximum Batch Count:        172994
===========================================================

 

[Imouto]

I find it hilarious that such a quick and dirty implementation is able to boost the frames in this way and at the same time a person calls DC "fundamentally broken".

Johan Andersson - Technical Director on Frostbite at Electronic Arts

 

[jj109]

There's something up with Oxide's DC implementation. Probably because they spent all their time optimizing immediate context...

Metro LL:

Star Swarm RTS:

SharpDX 65536 draw call benchmark for comparison:

Johan Andersson - Technical Director on Frostbite at Electronic Arts

Please do elaborate on how one of DX11's three key features is "fundamentally broken." Preferably not from someone who's trying to sell Mantle on twitter.

 

[sontin]

Johan Andersson - Technical Director on Frostbite at Electronic Arts

Yeah, and?
It sounds better than " a person who has an agenda to bash a feature of a universal API which would help everyone instead of only their sponsor".

 

[BrightCandle]

I thought I would do some frametime analysis as well and I ended up with a 83 fps run for DC=off and MB=off. So as much as I would like to draw some actual numbers (like its 50-100% faster) the variation between runs is so high that a 50% increase is within the margin of error. You would probably need to run this hundreds of times and do statistical analysis for comparison and use the mode for frametime analysis and I frankly don't want to waste any more time running a benchmark that is clearly fatally flawed.

What I don't understand is what Oxide gets from it, because they must have the same problem with it - the variance is between runs is so massive that its impossible to see if a change they made shows any improvement or not. This is not a good showcase for anything other than how not to write a performance demo. Period.

In all cases I have seen Deferred context improves performance, and its anywhere from 10% to 300%. Getting that down to a more useful and realistic number with this benchmark isn't really possible.

 

[Noctifer616]

I thought I would do some frametime analysis as well and I ended up with a 83 fps run for DC=off and MB=off. So as much as I would like to draw some actual numbers (like its 50-100% faster) the variation between runs is so high that a 50% increase is within the margin of error. You would probably need to run this hundreds of times and do statistical analysis for comparison and use the mode for frametime analysis and I frankly don't want to waste any more time running a benchmark that is clearly fatally flawed.

What I don't understand is what Oxide gets from it, because they must have the same problem with it - the variance is between runs is so massive that its impossible to see if a change they made shows any improvement or not. This is not a good showcase for anything other than how not to write a performance demo. Period.

In all cases I have seen Deferred context improves performance, and its anywhere from 10% to 300%. Getting that down to a more useful and realistic number with this benchmark isn't really possible.

http://forums.overclockers.co.uk/showthread.php?t=18576886

They talk about custom runs for the demo, trying to make it as accurate of a benchmark as possible, check it out.

Also, Oxide said in their FAQ that they will release details on how to correctly use the demo as a benchmark.

 

[ShintaiDK]

So the demo is heavily rigged to boost mantle. Why do these cheats not surprise me

 

[richaron]

So the demo is heavily rigged to boost mantle. Why do these cheats not surprise me

Yep this demo shows just what the real world performance is.

Most games are coded badly and Mantle improvement is awesome!!

 

[Imouto]

Yeah, and?
It sounds better than " a person who has an agenda to bash a feature of a universal API which would help everyone instead of only their sponsor".

You're pulling a dangerous gambit here. Next week you may see even better results with Mantle but keep this stance, it's perfectly fine.

 

[Shivansps]

The RTS mode is just hilarius, i mean im GPU limited the whole test and all im seeing are dots, awful 2D explosions, a static background and eventually a few fighters will fly close.

This is no the way to do things.

 

[sontin]

You're pulling a dangerous gambit here. Next week you may see even better results with Mantle but keep this stance, it's perfectly fine.

You're sure?
Okay, here is a Mantle result from Star Swarm:

And here is my result with DC from the Follow-test:

== Results ================================================
Test Duration:            360 Seconds
Total Frames:            17999

Average FPS:            49.98
Average Unit Count:        4346
Maximum Unit Count:        5553
Average Batches/MS:        1007.96
Maximum Batches/MS:        3578.56
Average Batch Count:        20882
Maximum Batch Count:        114586
===========================================================

So, Mantle leads with 16,6%. Yeah...

 

[Imouto]

You're pulling a dangerous gambit here. Next week you may see even better results with Mantle but keep this stance, it's perfectly fine.

Well, that escalated quickly.


+99%
+109%
+124%
+209%

 

[guskline]

Oxide Games
Star Swarm Stress Test - ©2013
D:\Documents\Star Swarm\Output_14_02_01_0712.txt
Version 1.00
02/01/2014 07:12
===========================================================
== Hardware Configuration =================================
GPU: NVIDIA GeForce GTX 780
CPU: GenuineIntel
Intel(R) Core(TM) i7-3770K CPU @ 3.50GHz
Physical Cores: 4
Logical Cores: 8
Physical Memory: 17145409536
Allocatable Memory: 140737488224256
===========================================================

== Configuration ==========================================
API: DirectX
Scenario: ScenarioFollow.csv
User Input: Disabled
Resolution: 1920x1080
Fullscreen: True
GameCore Update: 16.6 ms
Bloom Quality: High
PointLight Quality: High
ToneCurve Quality: High
Glare Overdraw: 16
Shading Samples: 64
Shade Quality: Mid
Deferred Contexts: Disabled
Temporal AA Duration: 16
Temporal AA Time Slice: 2
Detailed Frame Info: Off
===========================================================

== Results ================================================
Test Duration: 360 Seconds
Total Frames: 12992
Average FPS: 36.09
Average Unit Count: 4098
Maximum Unit Count: 5251
Average Batches/MS: 768.23
Maximum Batches/MS: 2103.57
Average Batch Count: 21859
Maximum Batch Count: 144948
===========================================================
My results with rig2

 

[Imouto]

And here is my result with DC from the Follow-test:

I'll tell you the same as I told Carfax83. "Excuse me but your testing methodology, credibility and equity are nowhere to be found."

Infraction issued for callout.
-- stahlhart

 

[cytg111]

You're pulling a dangerous gambit here. Next week you may see even better results with Mantle but keep this stance, it's perfectly fine.

+1 for style.

Flaws on Flaws on Oxide : Have laymen dictate their expert opinion on stuff they know nothing about. Yea lets do that again (not really).

 

[sontin]

I'll tell you the same as I told Carfax83. "Excuse me but your testing methodology, credibility and equity are nowhere to be found."

It's not my fault that you dont believe the Mantle-hype-benchmark-tool.

Or do you mean that i lie?

 

[guskline]

Oxide Games
Star Swarm Stress Test - ©2013
E:\Users\My Documents\Star Swarm\Output_14_02_01_0808.txt
Version 1.00
02/01/2014 08:08
===========================================================
== Hardware Configuration =================================
GPU: NVIDIA GeForce GTX 670
CPU: GenuineIntel
Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz
Physical Cores: 6
Logical Cores: 12
Physical Memory: 17117147136
Allocatable Memory: 140737488224256
===========================================================

== Configuration ==========================================
API: DirectX
Scenario: ScenarioFollow.csv
User Input: Disabled
Resolution: 1920x1080
Fullscreen: True
GameCore Update: 16.6 ms
Bloom Quality: High
PointLight Quality: High
ToneCurve Quality: High
Glare Overdraw: 16
Shading Samples: 64
Shade Quality: Mid
Deferred Contexts: Disabled
Temporal AA Duration: 16
Temporal AA Time Slice: 2
Detailed Frame Info: Off
===========================================================

== Results ================================================
Test Duration: 360 Seconds
Total Frames: 17075
Average FPS: 47.41
Average Unit Count: 4450
Maximum Unit Count: 5614
Average Batches/MS: 611.27
Maximum Batches/MS: 1575.29
Average Batch Count: 14086
Maximum Batch Count: 115697
==========================================================
Results for rig 1 below.

Note that results for both rig 1 & 2 were run on extreme setting.