Question 10gbit adapter running out of pci-e lanes?

blade8079 · Nov 13, 2024

Hi All,

I have a 2018 built i9-7920 (44 lanes) on x299 gigabyte aorus gaming mobo with 3 m.2 slots and 5 pci-e 3 slots. All slots are min x4. I have 3 m.2 ssd on mobo, one additional ssd on pci-e and 4x sata expansion card in 3rd pci-e slot for connecting 4 hard drives + one intel 550T2 in 2nd pci-e slot and geforce 1080ti in 1st pci (4 pci-e slots are used and one is free)
Running win 10 with latest intel network drivers

My speeds over 10gbit QNAP switch to NAS are about 180-200mb/s where it should be 1000mb/s over 10gbit connection (12 disk raid-6 on nas). I have several other nas @2.5 gbit and I get same speeds with them as well.

I definitely suspect it's my network adapter on PC as direct copy to usb from NAS gives way higher speeds. The only way to check would be to get another model adapter , say 710, plug it in and test, but I'm curious if a hardware limitation of my cpu/mobo can be identified some other way?

Everything is showing 10gb connection and it shows it's connected via pcie3x4 to the mobo, but is it true or not or did I just run out of pci-e lanes on my system and that's limiting the network speeds?

even a pci3 x1 connection would be more than 2gbit speed...

Any ideas?

Thanks!

thecoolnessrune · Nov 14, 2024

You haven't mentioned much about your NAS I don't think, just that you have a QNAP Switch. RAID-6 is heavy and depending on your layout, may not give the performance you're expecting. It depends on the underlying architecture.

Your NAS giving high performance locally, and less performance over LAN only tells you that when the network stack is skipped, your speeds are higher. That could be from something on your Desktop, but it could also be the limitations of your NAS. You have not mentioned what protocol you're using to transfer these files. If it's SMB, which version? Depending on the NAS capabilities, and version being negotiated, SMB Encryption / Signing can impose a substantial burden on smaller NAS units. Can you see CPU metrics on your NAS while you're performing a file transfer?

Shmee · Nov 14, 2024

What hardware are these NAS systems using? And what drive are you using in your desktop for testing? Depending on what drive you are copying to/from, that could also be the issue. If using a SATA SSD, for instance, you are going to max around 550MB/s r/w. If using a HDD, it will be much less.

Hail The Brain Slug · Nov 14, 2024

What drive are the files stored on in your PC that you're copying to the NAS from?

I think a pcie lane issue seems unlikely. I agree they bottleneck is either network transfer protocol related (SMB overhead, not using SMB multichannel), link speed related (are you confirming all links in the network chain are at 10gbit?) or local drive related (copying from a hard drive?)

Micrornd · Nov 16, 2024

I can assure you it should not be because of a PCI-e 3 x4 slot, although it may be the network adapter OR how it is set up.
You will not be able to use that last slot you mentioned, as you are out of lanes and this assumes you are not trying to use the slot that becomes unusable when you populate all the M2s slots (see your manual or the Gigabyte website).

So here's something else to consider.

I run a direct connection (no switch in-between) from my workstation to server using Nicgiga 10g adapters (in PCI-e 3 x4 slots on both machines), server and workstation are both Raid6 and max out the 10GbE connection (1.16-1.19 GB/s) transferring large files, smaller files transfer in the 650-900 MB/s range depending on how "small".
This works in either direction as the R6's on the server and workstation are identical in configuration.

As has been mentioned, both your transmitter and target must be capable of those transfer speeds or you will be limited to the speed of the slowest, have you confirmed that they are?
File size also makes a difference.
CrystalDiskMark will give you an idea of what you can reasonably expect, but it will normally give a slightly higher speed than real world usage, as most benchmarks do.

You also don't mention if you also have the MB's onboard network adapter connected to your network or not, and this does make a difference.
If the metrics are not set properly and left at the default of "auto", the default will always be to use the MB's network adapter, rather than an add-in network adapter, when using Windows.

blade8079 · Nov 16, 2024

Thank you all for your comments!

I copy to NAS from m.2 drives and the speed between m.2 drives inside my pc is almost 2 GBytes /s (not gbit) - still limited by pci-e, but way above what I need at this point.
I do NOT have SMB multichannel enabled on my qnap... should I enable SMB multichannel on qnap?

No, I do not use my built in 1GBit motherboard port on PC anymore. I used to before I got 550T.

On my qnap i have intel 710 based qnap 10gbit card and 10gbit qnap switch where it's plugged along with my PC. Switch is managed and shows 10gbit link speed on these connections. It is ts-1655 card with 64gig ram and QuTS hero. System share is on 2 m.2 drives and main storage is 12 drive wd gold raid6.

I am borrowing intel710 card to test to at least see if there is the same issue there.

Hail The Brain Slug · Nov 16, 2024

blade8079 said:
Thank you all for your comments!

I copy to NAS from m.2 drives and the speed between m.2 drives inside my pc is almost 2 GBytes /s (not gbit) - still limited by pci-e, but way above what I need at this point.
I do NOT have SMB multichannel enabled on my qnap... should I enable SMB multichannel on qnap?

No, I do not use my built in 1GBit motherboard port on PC anymore. I used to before I got 550T.

On my qnap i have intel 710 based qnap 10gbit card and 10gbit qnap switch where it's plugged along with my PC. Switch is managed and shows 10gbit link speed on these connections. It is ts-1655 card with 64gig ram and QuTS hero. System share is on 2 m.2 drives and main storage is 12 drive wd gold raid6.

I am borrowing intel710 card to test to at least see if there is the same issue there.

You definitely can't get anywhere near 10gbit speeds without SMB multichannel. You should enable it, and check to make sure it's enabled and operational on your desktop and try again.

blade8079 · Nov 16, 2024

Thanks! I was under impression that SMB multichannel was required only when you have several ethernet ports hooked up on NAS so it can at least double throughput ? I guess I am totally wrong and should enable it regardless?

How would I enable it in Windows 10 or is it on by default?

Thanks!

Hail The Brain Slug · Nov 16, 2024

blade8079 said:
Thanks! I was under impression that SMB multichannel was required only when you have several ethernet ports hooked up on NAS so it can at least double throughput ? I guess I am totally wrong and should enable it regardless?

How would I enable it in Windows 10 or is it on by default?

Thanks!

SMB is inherently single-threaded/single stream. In order to overcome its limitations due to protocol overhead you have to use SMB multichannel. It will use RSS in order to use multiple streams and multiple threads. It can utilize multiple like network links to aggregate the bandwidth but it still benefits a single link.

There are powershell commands you can run to check the status of it, and enable it if it's not already enabled.

Deploy SMB Multichannel

learn.microsoft.com

You can use the Get-SmbClientConfiguration and Get-SmbServerConfiguration commands to get the current state, which will show if SMB Multichannel is enabled. The commands to enable it, if it is not enabled already, are in the article.

For reference, your PC is both a client and a server depending on which direction the transfer is going - It's not safe to assume the PC is always a client to the NAS.

blade8079 · Nov 16, 2024

Thank you! Will test tomorrow and have it enabled on both qnap and PC.

dlerious · Nov 17, 2024

blade8079 said:
even a pci3 x1 connection would be more than 2gbit speed...

pcie3.0 is 985 MB/s per lane.

blade8079 · Nov 18, 2024

So I did enable SMB multi with RSS and also swapped 710NIC to try. Right now I am able to saturate 2.5GBPS on one of my 2.5gbps qnaps, but my 10gbit one approximately doubled from 180-190 to 360mb/s network transfer speed so I think it's something weird with some windows setting. Technically I can hook up a second network cable to another 10gbit port on qnap and put it into a switch and see if I get double the 360x2 bandwidth with that, but I still believe I should be able to achieve 1GB/s with just one cable)

Question 10gbit adapter running out of pci-e lanes?

blade8079

Member

thecoolnessrune

Diamond Member

Shmee

Memory & Storage, Graphics Cards Mod Elite Member

Hail The Brain Slug

Diamond Member

Micrornd

Golden Member

blade8079

Member

Hail The Brain Slug

Diamond Member

blade8079

Member

Hail The Brain Slug

Diamond Member

Deploy SMB Multichannel

blade8079

Member

dlerious

Golden Member

blade8079

Member

TRENDING THREADS