Awful AMD VCE performance

[Shivansps]

I use the card encoder to record gameplay whiout performance hit, that i later edit and upload to youtube, after 2 years of using a GTX750TI and ShadowPlay whiout mayor problems, im now moved to a RX480 and at this point already want ShadowPlay (and NVENC) back.

First i attempted to use Plays.TV, that used to be bundled with AMD drivers, i attemped to use the same setting i used with shadowplay, 1080P@35Mbps, that yielded very good quality for its size, first problem i found with Play.TV is that it uses variable bitrate and framerate, and the problem with that is that even as i set 35mbps, the overall birate of the videos, specially if they are long ones is 10mbps, and they end up with a TERRIBLE quality, maybe on short 2 minutes videos it may get close to the 30mbps, maybe, but quality is still terrible, yet i can notice when the bitrate drops, even if its for a second.

Then later tryied more software that supports AMD VCE encoder, like Bendicam, Action!, etc, they all had a similar problem to Play.TV, on some, like Bendicam, it allowed me to play more with the settings, like setting in CBR, but to no avail, sometimes it gets really strange results like videos with 1mbps bitrates and/or with lag/shuttering.

Finally i ended with OBS Studio+Plugin, the exact same issues, some settings end up with really weird results, with videos with 1-4mbps, some other with like 10% of cpu usage, what is kinda high, Finally i managed to test both constant and variable framerates with a constant bitrate of 35mbps, still quality is really bad.

Here i uploaded some samples
https://www.dropbox.com/s/jk9g59mexvn6baw/Muestras.rar?dl=0

The first video is recorded in OBS with a constant framerate of 60 fps and constant 35mbps, quality is bad, you can notice it specially on the skybox.

The 2nd is a short video recorded with play.tv at 35mbps setting, ended with overall bitrate of 28mbps, still quality os bad.

The 3rd video is a part split of a long video recorded with play tv at the same settings than the 2nd video... overall bitrate end up as 10mbps and quality is terrible.


So, my question will be, what is going on here? this is VCE bug? Software is bad, or the VCE block on AMD IS REALLY this bad? because i cant work with this. I dont have shadowplay original videos anymore to compare it, but the videos that i edited and re-encoded at 22mbps had much more quality than this, and the variable bitrate worked fine on them, both original and re-edited.

 

[Bacon1]

Make sure you are using the latest version of the plugin for OBS:

https://github.com/Xaymar/OBS-AMD-Advanced-Media-Framework/releases

Unless obs has a newer one.

You'll need to use 16.10 drivers since they've changed from VCE to AMF (Amd Media Framework?)

The author posts here: https://www.reddit.com/user/Xaymar


Q: My recorded file only works up until 20:50!

This Issue has been fixed completely in 1.3.2.3, please follow the Troubleshooting Guide properly.

https://github.com/Xaymar/OBS-AMD-A...s#q-my-recorded-file-only-works-up-until-2050

https://obsproject.com/forum/resour...lity-recording-and-multiple-audio-tracks.221/

 

[chummy]

AMD not put many effort in their VCE for years, only last month they started messing their ass for it slowly. Shadowplay record with VBR too so i dont understand your complain here. But seeing your footage yes there is a lot of blockiness in the OBS version too which keep 30Mbps and that a bit strange, i never noticed it with high bitrate in my VCE 2.0. Probably AMD not improved the settings for Polaris yet in their drivers and some glitches there. Like other said be sure to use latest drivers since AMD only started improving VCE for AMF in 16.9.x+ mostly.

AMD is not good for features support, AMD is a case for the average Joe which only use card to plug and play games, if you want some extra features AMD will destroy your expectations. In that case you made a bad move going to AMD if you like recording. But if you have Intel Haswell or higher mainstream CPU with Quicksync it's better than VCE because there is so many effort from Intel put on it than advanced options will not be a problem with Quicksync and quality from Haswell+ is fine.

The problem with AMD features is not only hardware which VCE hardware is inferior to QuickSync and NVENC but the lack of persistent AMD software support to keep improving these.

Here some image comparison from 1080p@30fps/4Mbps from NVENC 1.0, VCE 2.0, Quicksync 3.0. VCE 2.0 was release after NVENC 1.0 and Quicksync 3.0 but was the weakest hardware and software of the three.

Source x264 150Mbit vs x264 1080p@4Mbit Veryfast: https://cdn.knightlab.com/libs/juxt...html?uid=d239b672-8f67-11e6-aace-0edaf8f81e27

NVENC 1.0 vs VCE 2.0: https://cdn.knightlab.com/libs/juxt...html?uid=f594192a-9adc-11e6-9008-0edaf8f81e27

The quality is bad for both but you can see than VCE produce lot of blockiness there.

 

[Shivansps]

Yes, i know that shadowplay uses VBR as well, or i think it does since i just tested and bitrate seems to be constant, ill report back shortly with samples. The problem is that, with Shadowplay, if i say 35mbps, it never dropped to 10mbps providing awful visual quiality like in those samples, shadowplay handles VRB a lot better because visual quality is impressive, the VBR with VCE is not working as it should, first i trought it may be a Play.TV issue, but i tested with other programs, like bendicam, playclaw5, mirillis, all have the exact same issue.

For some reason OBS was the only program who really allowed me to do constant bitrate(Bandicam should allowed me as well but it didnt work), with a really BAD result, i dont know of another program that could work to test.

It really CANT be THIS bad, i think the 1st gen QS on my 2500K performs better than this.

 

[Bacon1]

Yes, i know that shadowplay uses VBR as well, or i think it does since i just tested and bitrate seems to be constant, ill report back shortly with samples. The problem is that, with Shadowplay, if i say 35mbps, it never dropped to 10mbps providing awful visual quiality like in those samples, shadowplay handles VRB a lot better because visual quality is impressive, the VBR with VCE is not working as it should, first i trought it may be a Play.TV issue, but i tested with other programs, like bendicam, playclaw5, mirillis, all have the exact same issue.

For some reason OBS was the only program who really allowed me to do constant bitrate(Bandicam should allowed me as well but it didnt work), with a really BAD result, i dont know of another program that could work to test.

It really CANT be THIS bad, i think the 1st gen QS on my 2500K performs better than this.

Are you using the plugin I linked to or the super old one?

Which version of OBS? Studio or Original?

 

[Elixer]

I have seen some of AMD's twitch streams, and they were using OBS with the new plugin, and it looked crisp, @1080p.

Also, get http://bluesky23.yukishigure.com/en/AsVideoConv.html (you may have to request a newer build, I don't recall if the one up there now works OK with the 480 or not) and do some tests, does that one output better?

 

[Shivansps]

Are you using the plugin I linked to or the super old one?

Which version of OBS? Studio or Original?

I was already using it, but im gona re-test later today.

 

[Bacon1]

I was already using it, but im gona re-test later today.

Well there are two amd ones iirc in OBS, one is the very old plugin, and the newer one looks like this:

 

[Shivansps]

Ok i tried with the preset settings for "recording" and ends up with a very high bitrate video of overall 41mbps and still bad quality.

Here that sample
https://www.dropbox.com/s/apcwge5s596kdzm/Muestra 4_OBS.mkv?dl=0
Looks very similar quality to shadowplay 20mbps...

For comparison, ive also uploaded some shadowplay samples as well.
https://www.dropbox.com/s/0zw937bvaflrq0g/muestras_shadowplay.rar?dl=0

This is the best the AMD enconder can do?! i hope its some kind of bug or bad settings on my side here, its totally worthless as it is.

 

[chummy]

What game quality settings was you really using in each encoder? Let's start with basics. There is something called complexity in video compression. The more detail more complex to compress, adding fast movements make it even harden to compress. In VCE video has all that shimmering, sharpness and in the NVENC one you keep the quality almost blurred with far lower image complexity for compression.

Somethings which are very hard for video compression and demand higher bitrate in gameplay is detailed vegetation, adding all that sharpness and putting high motion things go really bad then and your better friend in this case is higher bitrate. In your shadowplay recording it's notorious than you disabled sharp and it make things much easier for the encoder and amount of bitrate used.

If you going to compare both encoders you need to keep same level of detail at least.

One real problem AMD has is with dark areas where it will give far worse quality and apparently NVENC dont suffer that much.

 

[Shivansps]

What game quality settings was you really using in each encoder? Let's start with basics. There is something called complexity in video compression. The more detail more complex to compress, adding fast movements make it even harden to compress. In VCE video has all that shimmering, sharpness and in the NVENC one you keep the quality almost blurred with far lower image complexity for compression.

Somethings which are very hard for video compression and demand higher bitrate in gameplay is detailed vegetation, adding all that sharpness and putting high motion things go really bad then and your better friend in this case is higher bitrate. In your shadowplay recording it's notorious than you disabled sharp and it make things much easier for the encoder and amount of bitrate used.

If you going to compare both encoders you need to keep same level of detail at least.

One real problem AMD has is with dark areas where it will give far worse quality and apparently NVENC dont suffer that much.

shadowplay offer no avanced settings, all of these are defaults to everyone that record games using it. There is no such things involved here, using VBR, the bitrate goes to the floor for no aparent reason, im setting 35mbps on Play.TV, wth is doing recording at 10mbps with a bad quality? You can notice it most by looking at the skybox even at still frames, what image complexity?
And even at very very high bitrates the result is still BAD.

 

[Shivansps]

record with dxtory(i tried with shadowplay also)
I worked with:2560x1440
import to adobe premiere
export to h264 codec
upload to utube

topic: http://www.portvapes.co.uk/?id=Latest-exam-1Z0-876-Dumps&exid=threads/capturing-playback-posting-4k.2482269/

DXtory, i wanted to try that one, but i finally gived up, it just crash on app launch, no error, no log file data of anything wrong, no window is draw either.

 

[nickmania]

I have seen some of AMD's twitch streams, and they were using OBS with the new plugin, and it looked crisp, @1080p.

Also, get http://bluesky23.yukishigure.com/en/AsVideoConv.html (you may have to request a newer build, I don't recall if the one up there now works OK with the 480 or not) and do some tests, does that one output better?

Polaris (460, 470, 480) is still unsupported in this converter in the version that is available for download.

 

[chummy]

VCE2 has lower quality than Quicksync3 in H264. VCE2 is worse than x264 Superfast while Quicksync is draw to x264 veryfast. Just tested with Witcher 3 and at same bitrate 36Mbit/s Quicksync is better than VCE, so increased VCE bitrate to 47Mbit/s but it yet lost by a few to Quicksync at 36Mbit/s 1080p@60fps.

Quicksync Quality at 36Mbit/s:
https://www.dropbox.com/s/hnpneybl9sg00rl/59.94fps QS-0B-G60 35Mb.mp4?dl=0

VCE Quality preset at 47Mbit/s:
https://www.dropbox.com/s/viimjixkytf6mji/59.94fps VCE-0B-G60 35Mbv2.mp4?dl=0

VCE Quality preset at 36Mbit:
https://www.dropbox.com/s/w0l9mym67c204kt/59.94fps VCE-0B-G60 46Mb.mp4?dl=0

When you want to compare both encoders make it at same scene and quality game. Or just like me record a clean lossless footage then use VCE/Quicksync/NVENC software encoding to compare quality.

Edit: VCE was always known to be slower and worse quality in all generations against Quicksync and NVENC. NVIDIA apparently added new H.264 quality improvements for Pascal NVENC while AMD make nothing new in quality for VCE4/H.264 compared to old versions, only increased a bit speed.

 

[Azix]

Ok i tried with the preset settings for "recording" and ends up with a very high bitrate video of overall 41mbps and still bad quality.

Here that sample
https://www.dropbox.com/s/apcwge5s596kdzm/Muestra 4_OBS.mkv?dl=0
Looks very similar quality to shadowplay 20mbps...

For comparison, ive also uploaded some shadowplay samples as well.
https://www.dropbox.com/s/0zw937bvaflrq0g/muestras_shadowplay.rar?dl=0

This is the best the AMD enconder can do?! i hope its some kind of bug or bad settings on my side here, its totally worthless as it is.

post a picture of your settings. and make sure you uninstalled OBS studio and the AMD media framework plugin, if you have that installed separately. Reinstall OBS without browser plugin, reinstall latest AMF plugin.

OBS-Studio-0.16.2-Small-Installer.exe https://github.com/jp9000/obs-studio/releases
1.3.3.1 - https://github.com/Xaymar/OBS-AMD-Advanced-Media-Framework/releases

You would do well to get it working. Definitely not as bad as you are experiencing. On my Fury I can do 1080p 60 fps with good quality, little to no fps loss and <2GB for 23 minutes.

VCE2 has lower quality than Quicksync3 in H264. VCE2 is worse than x264 Superfast while Quicksync is draw to x264 veryfast. Just tested with Witcher 3 and at same bitrate 36Mbit/s Quicksync is better than VCE, so increased VCE bitrate to 47Mbit/s but it yet lost by a few to Quicksync at 36Mbit/s 1080p@60fps.

Quicksync Quality at 36Mbit/s:
https://www.dropbox.com/s/hnpneybl9sg00rl/59.94fps QS-0B-G60 35Mb.mp4?dl=0

VCE Quality preset at 47Mbit/s:
https://www.dropbox.com/s/viimjixkytf6mji/59.94fps VCE-0B-G60 35Mbv2.mp4?dl=0

VCE Quality preset at 36Mbit:
https://www.dropbox.com/s/w0l9mym67c204kt/59.94fps VCE-0B-G60 46Mb.mp4?dl=0

When you want to compare both encoders make it at same scene and quality game. Or just like me record a clean lossless footage then use VCE/Quicksync/NVENC software encoding to compare quality.

Edit: VCE was always known to be slower and worse quality in all generations against Quicksync and NVENC. NVIDIA apparently added new H.264 quality improvements for Pascal NVENC while AMD make nothing new in quality for VCE4/H.264 compared to old versions, only increased a bit speed.

AMD not put many effort in their VCE for years, only last month they started messing their ass for it slowly. Shadowplay record with VBR too so i dont understand your complain here. But seeing your footage yes there is a lot of blockiness in the OBS version too which keep 30Mbps and that a bit strange, i never noticed it with high bitrate in my VCE 2.0. Probably AMD not improved the settings for Polaris yet in their drivers and some glitches there. Like other said be sure to use latest drivers since AMD only started improving VCE for AMF in 16.9.x+ mostly.

AMD is not good for features support, AMD is a case for the average Joe which only use card to plug and play games, if you want some extra features AMD will destroy your expectations. In that case you made a bad move going to AMD if you like recording. But if you have Intel Haswell or higher mainstream CPU with Quicksync it's better than VCE because there is so many effort from Intel put on it than advanced options will not be a problem with Quicksync and quality from Haswell+ is fine.

The problem with AMD features is not only hardware which VCE hardware is inferior to QuickSync and NVENC but the lack of persistent AMD software support to keep improving these.

Here some image comparison from 1080p@30fps/4Mbps from NVENC 1.0, VCE 2.0, Quicksync 3.0. VCE 2.0 was release after NVENC 1.0 and Quicksync 3.0 but was the weakest hardware and software of the three.

Source x264 150Mbit vs x264 1080p@4Mbit Veryfast: https://cdn.knightlab.com/libs/juxt...html?uid=d239b672-8f67-11e6-aace-0edaf8f81e27

NVENC 1.0 vs VCE 2.0: https://cdn.knightlab.com/libs/juxt...html?uid=f594192a-9adc-11e6-9008-0edaf8f81e27

The quality is bad for both but you can see than VCE produce lot of blockiness there.

I tried quicksync. It was not so great. You should update yourself on the state of VCE. I mean, it worked fine before but was causing performance loss more than necessary. Now its working really great thanks to the new AMF sdk from AMD and the plugin.

I doubt VCE hardware is inferior. I know at least one thing it does that is said those others do not do. Allow changing encoding settings on the fly. The issue was mostly software, but its working well now.

 

[Shivansps]

I tryied QS of my 2500K and quiality is slightly worse than AMD VCE, this is not a suprise to me as this is 1st gen QS.

As for settings, i managed to get good quality, at very HIGH bitrates, but now i have problems with the videos not being fluid, at 30 fps is aceptable, but still choppy at times(not sure if this is the right word for want i mean to say), at 60 fps is just not ok, with game locked at 60 fps and video recording at 60 fps constant, it should be fluid, but its not, its very very choppy, specially with movement, fast and no so fast movements.

As for settings, this is what im using right now.

Size is crazy, 4GB for 10m, but at least quality is good.

Im waiting for framework 1.4.0.0 to be released to test again.

 

[chummy]

Just to clarify, each generation of ASIC can be added quality and speed improvements, most of time they only is adding speed improvements. Quicksync1/2 SB/Ivy has no good quality like Haswell+. For VCE apparently VCE1 is the worse then VCE2,3,4 has around same H.264 quality but maybe there is some quality enhancements which need to be verified. For NVENC NVIDIA stated in their docs than Maxwell get some quality improvements over Kepler then Pascal get even more quality improvements.
NVENC1 was already close to Haswell3 quality, so i'm a bit curious to see Pascal capabilities since NVIDIA says there has multiple(2x or 3x) encoders engines in Maxwell/Pascal which can do multiple encodings at once. I dont know if Broadwell/Skylake received H264 quality improvements over Haswell but i find nothing about it.

Shivansps use advanced encoder settings. Use VBR Peak constrained and not Latency since latency is meant for streaming. Keep GOP size at default value since your keyframe interval is deciding your GOP based in your framerate.

Follow this with except you can change some of them like:
Minimum/Maxium QP,
IPB Frame QP,
Keyframe interval(for 60fps try 30/60/120 values to find what fit your best),
B-pictures you can try with 0 or 3 to see if it improve or decrease quality,
B-Picutures Reference only works if there has 1 or more B-picutres added,
Preset quality,
Motion Estimation,
CABAC.
The other settings try avoiding changing

http://i.imgur.com/jRUXcrk.png

http://i.imgur.com/WYXOera.png

 

[chummy]

I tried quicksync. It was not so great. You should update yourself on the state of VCE. I mean, it worked fine before but was causing performance loss more than necessary. Now its working really great thanks to the new AMF sdk from AMD and the plugin.

I doubt VCE hardware is inferior. I know at least one thing it does that is said those others do not do. Allow changing encoding settings on the fly. The issue was mostly software, but its working well now.

Maybe VCE3 has some quality improvements over old one. But VCE4 from Polaris apparently loose some H264 settings like B-frames which were added only to VCE3. What Quicksync version and quality you used to compare to VCE3?

Besides that VCE worked for me with old Media foundation, Xaymar plugin/AMF make nothing new for beside giving some advanced settings which make no many differecens for VCE2 at all.

The major problems with VCE encoding stucking in old times were why AMD has major issues in their DX11 drivers which were fixed for encoding in recently releases. It was not the AMFSDK/Xaymar plugin which fixed the glitches because new AMF SDK introduced 0 features. Now look NVIDIA new H264 features for Pascal release:

• Asynchronous H.264 Motion-Estimation-(ME)-only mode
• Look-ahead
• Improved H.264 spatial adaptive quantization
• H.264 temporal adaptive quantization
• Rate control and quality improvements

* These features require Pascal generation GPUs.
Release Date: July 18, 2016


AMD VCE2 which was bundle in 2015 390x cards is only capable to do 1080p@45fps at quality while GTX970 NVENC3 released at 2014 can do 4k@60fps encoding. For those who use NVENC/VCE for encoding outside gaming NVIDIA is far superior in that matter. Pascal has 2x 4k@60fps encode engine which can encode 1080p@350fps, while VCE4 is 1080p@120fps.

Old NVENC1 dated from early 2012 which is 18 months older than VCE2 can do 1440p@60fps while VCE 2 is stucked at 1080p@60fps being released 18 months later. GTX650 which cannot run 1440p games can record 1440p@60fps while 390x cannot record at 1440p because like you said VCE hardware is not inferior. AMD needed 3 years to achieve NVENC1 2012 capability with VCE3.

 

[Shivansps]

Ill try, but man i whould have love to know all that before getting the RX480, i whould have picked the 1060 for sure.

 

[el etro]

Just to report, i made a recent recording with plays tv, but also had a last recording made with Raptr. Both are around the same size(quality) settings, but this last one was pretty bad. Eight months(and a lot of drivers) passed between the first recording and the second.
But i also remember that my Nvenc/VCE-auxilied transcodings on Expresso had good quality.

Card today is Radeon HD 7950 Boost.

 

[Bacon1]

FYI new drivers today and some mentions specifically aimed at AMF fixes. Also there is a new build (4.0?) for the OBS plugin.

 

[el etro]

AMD should offer more control for VCE behavior.

From what i said, quality was good but not great, yet time decreases significantly. At the time Kepler exhibited better image quality than GCN1 from what i recall.

 

[Shivansps]

There was new set of driver, OBS version and AMF plugin released, there is a bunch of new bugs, even hard crash BSODs at some settings, and no fixes, VBR recording used by most programs like Plays.TV is still broken and causes the bitrate to drop way below the target with important quality drops.
And OBS recording is worse than ever, new bug makes impossible to record at 60 fps because AMF overrides it, probably be fixed soon, and recorded videos still shutter and lag, even using 30fps@speed profile quality, the reason seems to be the encoder is unable to keep up and drop frames.

This is exactly what happens when you make everything "open source" and take no responsability for it, the entire game recording and streaming on AMD now depends on 1 guy that is not related to AMD in any way. he doing a good job, but he not the one that should be doing this.

Or things may work on VCE 3.0 and below, but VCE 3.4/4.0 of Polaris is bugged as hell on drivers(i hope).

Also for some strange reason, VCE 3.4/4.0 is inferior to 3.0 as far as H264 recording is concerned, because it lacks BFrames.

 

[chummy]

B-frames are a bit useless in Hardware encoders. In NVEnc & Quicksync they only help making frames totally blurry when too hard to keep quality to avoid artifacting instead, remember something from HEVC/VP9. Using only P-frames make a better average frame quality for QS and NVEnc. The quality difference between encoder brands and generations is in other hardware capabilities not B-frames. To discover if VCE4.0 has better image quality than VCE3.0 need to make a direct comparison. Even Nvenc1 supporting B-frames NVIDIA shadowplay only use P-frames.

About your current problem, with VCE2 target bitrate is respected at CBR and VBR too. So should be a driver bug for new VCE. There is yet the old problem you related before latest update with recording frameskipping where the encoder is reencoding same frame multiple times and make video playback choppy. This something happens in some games even with VCE2 and i guess is AMD driver problem.

 

[Abwx]

And OBS recording is worse than ever.

Rather than doing random statements you should check if the issue wasnt adressed..

http://www.hardware.fr/news/14842/maj-radeon-software-16-11-2-call-of-duty.html

AMD annonce également avoir optimisé son support d'OBS Screen Capture qui pouvait créer des ralentissement lorsque l'on capturait des vidéos tout en regardant du contenu vidéo dans un navigateur web.

So there s some fixes in the latest drivers that should adress your demands...