ASRock Z97 Extreme9 Ultra M.2 x4 and PCIe lanes

[quadfire]

Hi all,

I am the proud owner of an ASRock Extreme9 board since today but there is something I wonder about:

Like the Extreme6 that Ian Cutress reviewed for Anandtech here: http://www.anandtech.com/show/8045/asrock-z97-extreme6-review-ultra-m2-x4-tested-with-xp941/11 the Extreme9 has an Ultra M.2 x4 slot.

Now for the Extreme6 Ian wrote:
"The Haswell CPUs for the Z97 platform have 16 lanes of PCIe 3.0 which are split into two lots of eight. One of these eight can be split into two lots of four, and it is typical to see on a motherboard a multi-GPU arrangement of x8/x4/x4 from the CPU. It is these last four lanes that ASRock has adapted for its M.2 slot, which means that when an M.2 drive is placed into the x4 slot, it will reduce the bandwidth of the first two PCIe slots down to x8/x4. This also disables SLI, due to NVIDIA’s requirement for x8 PCIe lane allocation (either PCIe 2.0 or 3.0) for each graphics card."

So if youi install the M.2 SSD your PCIe lanes go down form 8x/8x to 8x/4x for 2 cards in the Extreme6.

Now I want to run 4x R9 290s in the Extreme9 and the board should do that at 8x/8x/8x/8x using the PLX PEX 8747 chip.

However this chip also "only" has 48 PCIe lanes, 16 upstream to the CPU and 4 x 8 = 32 to the GPUs. In total this is 48 lanes.

Now I wonder what does happen when installing a M.2 SSD in this 4x PCIe3 M.2 slot?

Will it take some PCIe lanes from the upstream between PLX and CPU, so this will only have 12 lanes - if this is even possible?

Or will it reduce the lanes from the PLX chip to one of the GPUs so the end result would be 8x/8x/8x/4x for the 4 GPUs, similar to the result for the Extreme6 for 2 GPUs?
As the PLX chip specs say it has 5 ports, which are already used by one to the CPU and 4 to the GPUs, this also seems unlikely, as where would the 6th port for the M.2 suddenly come from?

My last theory is that it does not really use PCIe3 lanes from the CPU as advertised but instead PCIe2 lanes from the Z97. But would ASRock advertise this incorrectly? And if this is possible why didn't they also do it on the Extreme6?

I read around in the manual but couldn't find any useful information about the distribution of PCI lanes for the Ultra M.2 about it in there.

Also I asked the ASRock support about this and got a response, that "Z97 Extreme9 using PLX 8747 chip so do not have the limitation as Z97 Extreme6 (let PCIe x 16 slot working at x8).".
But somehow I can't believe that as those 4 PCIe3 lanes have to come from somewhere, don't they!?

I'd appreciate any input on this as my decision on whether to get an XP941 and install it in there largely depends on that.

 

[Galatian]

Hey there,

I have the same board since Friday and I kept wondering about the same question. The manual does not state anything. I'm planning on Tri-Fire so I guess I have a little less to worry. But yeah: I would like that question answered as well...

 

[weevilone]

The PLX chip lets you view the PCI lanes as bandwidth, and it apparently can then share the bandwidth between devices that might otherwise nail up the resource but not entirely utilize it.

The question is legit though, what is the performance penalty and how does it manifest when you use a 4 lane M.2 with that many GPUs?

It seems like perhaps X99 might be the way to go for this (soon).

 

[Galatian]

On the other hand, the Anandtech Review of the Extreme6 revealed that GPUs rarely need all the bandwidth they are rated for and it is such a tiny decrease in performance.

The way I understood it the data sent to the GPU is always the same, so it makes no difference if you have one, two, three or four GPUs. The PLEX chip will "multiply" the data somehow. To me this sounds like there shouldn't be a bandwidth problem at all, but maybe someone more knowledgeable can chime in.

X99 is a no-go for me, first because the performance/€ is way out off line and second I do enjoy Intel Quick Sync which requires an socket 1550 CPU.

 

[Dahak]

Well there is Two M.2 slots on the board

The top one that uses x4 Gen3 - is a pci-e drive only, not sata

From Manual - 1 x Ultra M.2 Socket (M2_1), supports M.2 PCI Express
module up to Gen3 x4 (32 Gb/s)

The bottom one uses x2 from the chipset and is shared with the SATA Express connector. So this one is like the one that is on the Extreme6 that was reviewed

From Manual - 1 x M.2_SSD (NGFF) Socket 3 (M2_2), supports M.2 SATA3
6.0 Gb/s module and M.2 PCI Express module up to Gen2 x2
(10 Gb/s)

Although, I am also confuses as to where the x4 PCIe Gen 3 comes from as well as the math does not add up, there is 16 from the CPU which you assume all 16 goes to the PLX chip, the PLX chip has 32 and if doing quad video is x8/x8/x8/x8.

Unless they split the x16 from the CPU to x8 to plx(which it can do, each port can be either x8 or x16) and then the other x8 for other items for example x4 for the top m.2 slot.

Without a block diagram, it would be hard to say

 

[Galatian]

Well there is Two M.2 slots on the board

The top one that uses x4 Gen3 - is a pci-e drive only, not sata


The bottom one uses x2 from the chipset and is shared with the SATA Express connector. So this one is like the one that is on the Extreme6 that was reviewed

No, the Extreme6 also has the Ultra M.2 slot.

Although, I am also confuses as to where the x4 PCIe Gen 3 comes from as well as the math does not add up, there is 16 from the CPU which you assume all 16 goes to the PLX chip, the PLX chip has 32 and if doing quad video is x8/x8/x8/x8.

Unless they split the x16 from the CPU to x8 to plx(which it can do, each port can be either x8 or x16) and then the other x8 for other items for example x4 for the top m.2 slot.

Without a block diagram, it would be hard to say

I think you are right on your second paragraph. The AnandTech Review describes this best, and you obviously can use only 8 PCIe 3.0 lanes to power the PEX 8747 chip! That's some serious magic done right there

 

[Spin5000]

I know this answer won't satisfy you, but if you're running multiple GPU's, especially 3-4, then go with X79 LGA 2011, discussion over.

 

[BrightCandle]

The manual doesn't say where the 4x PCI-E slots come from to support the module. I couldn't find anything in there to suggest it would reduce the width of the GPU connections, but then since it doesn't define it that is a bit unclear how its doing.

But I have to concur with the others, these really is one of the cases the X99 systems is for. The mainstream is great up to 2x GPUs but past that it becomes less and less worthwhile and your adding a lot of PCI-E usage to a system that really doesn't have many lanes coming out of the CPU.

Edit: The only thing I have that suggests 4x GPU and a M.2 drive is ok is a picture on Asrocks advertisement page:

Its not conclusive proof that all is well, but Asrock isn't saying you can't or that it will cause problems, and based on it being 4x PCI 2.0 I suspect its separate lanes to the chipset. Its still all got to go into the CPU via 20 lanes total anyway, its not like you are actually getting mores lanes out of the PLX chip or Asrock's board, the CPU is still the bottleneck. But I think you can have 4x 8x GPU and have your 4x PCI-E 2.0 on your M.2. If you use both at once your going to get performance problems of course but separately it should work out OK. I wouldn't do it myself, 3 and 4 GPUs is really more suitable for more real lanes.

 

[Galatian]

I know this answer won't satisfy you, but if you're running multiple GPU's, especially 3-4, then go with X79 LGA 2011, discussion over.

That's just wrong. The PLEX chips adds latency, fair enough, but it's negible from the reviews

Edit: discussion over ;-)

 

[quadfire]

Well the specs on ASRock's site specifically say it's 4 "Gen3" PCIe lanes for the Ultra M.2 slot. Also in my virtual block diagram in my head the Ultra M.2 slot can't really share lanes from the PLX side unless the slot is incompatible with a quad GPU setup as all 5 ports of the PLX chip are already used in this scenario. So the only thing I can think of is that it will reduce the upstream from the PLX chip to the CPU from 16x to 12x (if possible at all) or to 8x.

I have asked a follow up question to the email I got from ASRock support as well as created a new ticket asking for specifics but unfortunately got no response from them

 

[quadfire]

I finally got feedback from ASRock support today, claiming that it will work fine using 4 lanes from the CPU for Ultra M2 while also being able to run 8x 8x 8x 8x with 4 GPUs.

They attached this image, I guess/hope it's fine to share this here:

So there is some kind of PCIe switch in between the CPU and the PLX PEX 8747.

Sill not 100% sure what this means for the lanes available to the PLX chip from the CPU. ASRock support also claims in their message that the SSD can get a 1xxx/9xx Critical diskmark score in this configuration.

 

[BrightCandle]

The PEX 8747 is a PCI-E switch. So presumably the PCI-E switch before it is just another cheaper device that can share 4 lanes of the 16 lanes going out to the GPUs. They do much less capable 16 lane switches as well as the 48 lane one used for the GPU so presumably its not any different.

Still its a bit frightening how much latency all that switching is likely costing combined with the lanes being oversubscribed even more than they were before. You have 16 lanes on the CPU servicing 36 lanes total.

Basically if you use your SSD heavily at the same time as the GPUs are transferring data heavily you are going to see a drop in frame rate. How much and how badly really depends.

 

[Galatian]

The PEX 8747 is a PCI-E switch. So presumably the PCI-E switch before it is just another cheaper device that can share 4 lanes of the 16 lanes going out to the GPUs. They do much less capable 16 lane switches as well as the 48 lane one used for the GPU so presumably its not any different.

Still its a bit frightening how much latency all that switching is likely costing combined with the lanes being oversubscribed even more than they were before. You have 16 lanes on the CPU servicing 36 lanes total.

Basically if you use your SSD heavily at the same time as the GPUs are transferring data heavily you are going to see a drop in frame rate. How much and how badly really depends.

Well from all the tests the latency amounts to something like 1 - 2%...not a big deal if you ask me. According to Anandtech all of the dual GPU cards actually have a pcie multiplexer included as well and nobody ever complained about their latency.

 

[Insert_Nickname]

I finally got feedback from ASRock support today, claiming that it will work fine using 4 lanes from the CPU for Ultra M2 while also being able to run 8x 8x 8x 8x with 4 GPUs.

They attached this image, I guess/hope it's fine to share this here:

So there is some kind of PCIe switch in between the CPU and the PLX PEX 8747.

Sill not 100% sure what this means for the lanes available to the PLX chip from the CPU. ASRock support also claims in their message that the SSD can get a 1xxx/9xx Critical diskmark score in this configuration.

Interesting configuration. I'd assume that if you're using the M.2 you'd have 12 lanes available for the PLX 8747. That's still ~12GB/s, so I don't think the bottleneck will be -that- bad actually. Inter-GPU communications doesn't have to go through it after all. Theoretically you could potentially have slightly better throughoutput then a dual GPU solution (8x/8x)...

I still think quad SLI/Crossfire belongs on LGA-2011(-3) platforms though. X99 would have been a prime candidate here.

 

[quadfire]

I still think quad SLI/Crossfire belongs on LGA-2011(-3) platforms though. X99 would have been a prime candidate here.

Due to some older tool I use I also need maximum single thread performance and the upcoming Haswell-E seem to be no match to the 4790k there, at least at stock clock rates.

 

[craige4u]

I have a Similar question: Buying Asus Z97 Deluxe

1] Now, I will always be using Single GPU... and say if I add upcoming Asus M.2 x4 Adapter on PCIE slot - In this case will my GPU run in x16 OR x8 mode?

2] If I add M.2 x4 adapter + Lan Card, In this case will my GPU run in x16 OR x8 mode?

 

[Insert_Nickname]

I have a Similar question: Buying Asus Z97 Deluxe

1] Now, I will always be using Single GPU... and say if I add upcoming Asus M.2 x4 Adapter on PCIE slot - In this case will my GPU run in x16 OR x8 mode?

2] If I add M.2 x4 adapter + Lan Card, In this case will my GPU run in x16 OR x8 mode?

1) If you add it in one of the slots coming from the CPU, then yes, your GPU will be "limited" to x8

2) Again, if you use the secondary PCIe slot coming of the CPU, the GPU will be "limited" to x8.

If you don't need the additional features of the Z97 Deluxe, look at the Asrock Z97 Extreme6. If you have a desire to add a PCIe M.2 drive, it will allow you to do that, but you have an additional 3.0 x4 slot from the CPU to use as you see fit. You can even add two M.2 drives, as it has two M.2 slots, though the other one is from the PCH, limiting bandwidth to x2.

You could also wait for Skylake, rumour has it that the PCH is upgraded to PCIe 3.0...

 

[abaez]

Can someone let me know if UEFI has been updated so that you can use M.2 drives as OS drives with the Asrock boards?

At the bottom of this page I see that you couldn't before, but will eventually: http://www.anandtech.com/show/8045/asrock-z97-extreme6-review-ultra-m2-x4-tested-with-xp941/11 (this was 2 months ago).

If I buy one of these: http://www.newegg.com/Product/Produc...9SIA0ZX1PA5339 and an Extreme6, will I be able to install the OS on the M.2 today?

Thanks

 

[Insert_Nickname]

Can someone let me know if UEFI has been updated so that you can use M.2 drives as OS drives with the Asrock boards?

At the bottom of this page I see that you couldn't before, but will eventually: http://www.anandtech.com/show/8045/asrock-z97-extreme6-review-ultra-m2-x4-tested-with-xp941/11 (this was 2 months ago).

If I buy one of these: http://www.newegg.com/Product/Produc...9SIA0ZX1PA5339 and an Extreme6, will I be able to install the OS on the M.2 today?

Thanks

You can. Though it'll not work in the Ultra M.2 slot, that is PCIe M.2 drives only. The 530 is a SATA drive in M.2 format, so the board and OS will see it as a regular SATA drive. With no restrictions on booting.

The difficulty is if you have a PCIe M.2 SSD, like the XP941. Those require UEFI and OS (Windows 8(.1)/Linux) support to boot...

(M.2 can be slightly confusing, since its only a form factor that supports multiple standards with the same connector... :|)

 

[abaez]

Really? Why make an M.2 if it's just the same as an SATA?

So if I get an XP941 is it available as a boot drive yet or no?

 

[Galatian]

If you use Win 8.1, yes

 

[Insert_Nickname]

Really? Why make an M.2 if it's just the same as an SATA?

One word: Mobile...

So if I get an XP941 is it available as a boot drive yet or no?

Yes.

http://www.asrock.com/mb/Intel/Z97 Extreme6/?cat=uM2SSD

These are what is currently officially tested in the Ultra M.2 slot, but I've heard the smaller capacity XP941's work just fine too. (though you'll want the 512GB for maximum performance)

http://www.asrock.com/mb/Intel/Z97 Extreme6/?cat=M2SSD

These are what are currently officially certified to work in the "standard" M.2 slot. I see no reason why other drives shouldn't work.

 

[craige4u]

1) If you add it in one of the slots coming from the CPU, then yes, your GPU will be "limited" to x8

Well all PCI slots comes from the CPU only, Right !?

 

[Insert_Nickname]

Well all PCI slots comes from the CPU only, Right !?

No, the CPU only has 16 lines available, and 4 lines for the DMI 2.0 to the PCH (really just PCIe with a fancy name). Depending on chipset these can be used as either a single 16x slot, or switched to 8x/8x or 8x/4x/4x. The last two options require a Z-series chipset with appropriate connectors.

The rest of the PCIe lines available is provided from the PCH. There are two catches with those slots. First is that they're only PCIe 2.0 compliant and second all communication with the CPU has to go through the DMI link (PCIe 2.0 x4) and worse shared with everything else that's connected to the PCH (SSD/HDDs, NIC, HD Audio etc.). Which is bad for performance, though its not much of a problem unless you're running a GPU from them. Its still kind of hard to saturate the DMI link (2GB/s, around 1600MB/s practical), but its getting easier...

 

[craige4u]

No, the CPU only has 16 lines available, and 4 lines for the DMI 2.0 to the PCH (really just PCIe with a fancy name). Depending on chipset these can be used as either a single 16x slot, or switched to 8x/8x or 8x/4x/4x. The last two options require a Z-series chipset with appropriate connectors.

My understanding is quite limited to pci lanes which I find a lil confusing...
So with Z97 deluxe and with both the scenario I have given above, no matter what configuration of the two I do, my GPU will slow down to 8x Right ?

If thts the case then I guess I am stuck with M.2 slot (x2) with max transfer of 1,000MB/s and wont be able to take advantage of near futures ultra fast M.2 drives