Re-educate me on dual-GigE NAS units, and the differences between LAGG, LACP, and SMB3.0 multi-channel?

[VirtualLarry]

So, my understanding is that, with SMB 3.0 multi-channel, and two GigE links, I can get two "streams" of SMB going over each link, thus, for multiple file transfers, I can get a speed-up.

(Edit: I may be misunderstanding this, maybe not one file == one stream, but rather, for one file, there can be multiple TCP connection streams, one for each link?)

My understanding of LAGG and LACP are a bit more foggy.

People have told me that they work basically the same way, but with one IP address for both ports, and that any one particular "stream" cannot exceed 1Gbit of b/w.

So, I guess my ultimate question is, if I have two switches, connected via a 10GbE-T link between them, and on one switch, I have a bunch of 2.5GbE-T links, connected to individual PCs (single NIC @ 2.5Gbit bi-dir), and then the other switch, I have 2.5GbE-T ports, but connected in pairs, using either LACP or LAGG, to a pair of 1GbE ports on a NAS that support LAGG and/or LACP, can I get a SINGLE stream, to transfer at 2x 1GbE, or 2Gbit, in one direction, if I'm connecting to my local switch at 2.5Gbit from my PC? Or will I still be limited to 1Gbit per single stream, but capable of doing 2x streams? Or not even that, because I only have a single NIC in the PC?

Apparently, you can combine LACP/LAG with SMBMC? O_O

https://forums.servethehome.com/index.php?threads/smb-lacp-nic-teaming-you.6782/

One connection can only go through one switchport. Thats why SMB multichannel sets up multiple connections for a single datatransfer and thus can scale through multiple nics. LACP however in the most defalt setting won't loadbalance the connections coming from the same IP and going to the same IP. You shlould check your switch manual how to configure the switches LACP connections to be correctly loadbalanced for SMB multichannel.

https://forums.servethehome.com/index.php?threads/smb-3-0-and-bandwidth-aggregation.1046/page-2

brendan_kearney
Ars Tribunus Militum
Registered: Nov 27, 2004
Posts: 1626
Posted: Mon Feb 09, 2015 8:37 am
you trunk VLANs
you bond or port-channel interfaces

unless you nas has more that one interface on more than one network you likely wont be doing trunking.

all of the bonding/port-channeling you accomplish with 802.3ad and LACP or balance-XOR help you have more conversations at full bandwidth, rather than give you more bandwidth on one conversation. if you want a real bandwidth increase on a per-conversation level, you need to move up to 10G.

(emphasis added by me)
https://arstechnica.com/civis/viewtopic.php?t=1263753

First of all two recommondations for a SMB 3 Multichannel:
- Normal TEAM will not support a SMB 3 Multichannel
- RSS must be enabled on the TEAM

https://server720.blogspot.com/2012/07/smb-30-multipath-over-nic-team-in.html

 

[SamirD]

Really great questions. I'm waiting to read equally great answers because I don't have them.

 

[VirtualLarry]

Well, not having much luck.

I've got three NAS units, two QNAP, one Asus, one ARM CPU, two Intel CPUs, all three with dual 1GbE jacks, both connected to the same Netgear GM110MX switch, which is an 8x 1GbE-T / 2x 10GbE-T switch. It is wired to another unmanaged 8-port GigE Netgear switch, and then wired to a WD switch (which has hard-wired port-based priorities on the ports, which is unhelpful to my application, but it would be a pain to swap out at this time). I have a cable going from the WD switch, to an Intel 1GbE-T NIC on my main PC, and I have an Asus 10GbE-T card that I just installed, newest Asus drivers for it, connected also via Cat6 25' cable to one of the 10GbE-T ports on the GM110MX.

Whichever NIC(s) I have enabled on the PC, nor which NAS, I cannot seem to get transfer rates over 105MB/sec to any of my NAS units.

I have not tried setting up Teaming or LAG on the QNAP NAS units, which have menu options for it, I don't believe that the Asus NAS does.
The GM110MX is unmanaged. I should probably have purchased the GM110EMX, which is Managed.

Anyways, I have a D-Link 8-port 2.5GbE-T with 2x 10GbE SFP+ ports (with some copper transcievers on the way), Managed, waiting in the wings to be deployed.

Thus far, I haven't been able to see any benefit to SMB 3.0 multi-channel.

My questions: Do you NEED to set up a LAG group on a switch, to see the benefits of SMB3-MC? Or do both NICs on the client PC have to be DIRECTLY connected to the switch, that the NAS units are connected to, and does that switch need to be managed, or otherwise support "Passive LACP"?

There is a new 5GbE-T USB3.0 NIC by QNAP, that's around $100 on Amazon, both of my QNAP NAS units can utilize those NICs, and are supported as of the last couple of firmware updates. I suppose I could buy two of those, and connect them to each NAS, and connect those to the 2.5GbE-T D-Link switch, and connect that switch via the 10GbE SFP+ copper uplink to one of my GM110MX 10GbE-T uplink ports, and then to my main PC, and try it that way, and see if I can get 2.5Gbit/sec out of it (250MB/sec transfers). I doubt that the RAID will hold up to that speed.

Unrelated, I tried copying a bunch of Linux ISOs, maybe 10-15GB worth, from my main PC (with 2x 1TB NVMe SSDs in RAID-0), over the 10GbE-T, and/or the 1GbE-T NIC, and I was only getting 30-40MB/sec write speeds to the ARM-based QNAP NAS. I know that NAS in the past has supported 100MB/sec+ read and write speeds to a RAID-5, so I'm not sure what's up here. The newest firmware seems to chew up 95% CPU time on the NAS writing to it, for some bizarre reason.

 

[mxnerd]

SMB 3.0 is a complex thing, it doesn't always work.

LACP or LAG not required.


SMB MULTICHANNEL: HOW IT WORKS & TROUBLESHOOTING GUIDE

https://level1techs.com/video/smb-multichannel-how-it-works-troubleshooting-guide

====
Requirements

SMB Multichannel requires the following:

• At least two computers running Windows Server 2012 or Windows 8.

At least one of the configurations below:​

• Multiple network adapters
• One or more network adapters that support RSS (Receive Side Scaling)
• One of more network adapters configured with NIC Teaming
• One or more network adapters that support RDMA (Remote Direct Memory Access)

https://blogs.technet.microsoft.com...a-feature-of-windows-server-2012-and-smb-3-0/

 

[mxnerd]

Use smbclientnetworkinterface powershell cmdlet to check NIC's capability.

http://www.portvapes.co.uk/?id=Latest-exam-1Z0-876-Dumps&exid=threads/windows-8-1-multichannel-smb-3-0-no-switch.2409070/

 

[mxnerd]

SMB 3.0 Multichannel Auto Configuration In Action With RDMA / SMB Direct

https://blog.workinghardinit.work/2...configuration-in-action-with-rdma-smb-direct/

 

[VirtualLarry]

SMB 3.0 Multichannel Auto Configuration In Action With RDMA / SMB Direct

https://blog.workinghardinit.work/2...configuration-in-action-with-rdma-smb-direct/

Auto configuration looks at NIC type/speed => Same NICs are used for RDMA/Multichannel (doesn’t mix 10Gbps/1Gbps, RDMA/non-RDMA)

Emphasis mine. That's what I was trying to do. Or get 10GbE-T on client, direct to switch, and from switch, 2x 1GbE-T to dual 1GbE-T NICs on NAS.

 

[mxnerd]

Not related to SMB 3.0.

ASUS Aquantia 10G vs INTEL 10G NICs performance.

ASUS Aquantia 10G probably thermal throttled during file transfer.

 

[VirtualLarry]

Interesting video, thanks. He "recommends" the Asus XG-C100C 10GbE-T PCI-E 3.0 x4 card.

I picked up both of mine for ~$41 ea. after tax and shipping, and a pair of 8-port 2.5GbE-T / 2-port 10GbE SFP+ D-Link switches for ~$135 (before tax) as well.

 

[mxnerd]

Interesting video, thanks. He "recommends" the Asus XG-C100C 10GbE-T PCI-E 3.0 x4 card.

I picked up both of mine for ~$41 ea. after tax and shipping, and a pair of 8-port 2.5GbE-T / 2-port 10GbE SFP+ D-Link switches for ~$135 (before tax) as well.

Excellent deals!

 

[SamirD]

Interesting video, thanks. He "recommends" the Asus XG-C100C 10GbE-T PCI-E 3.0 x4 card.

I picked up both of mine for ~$41 ea. after tax and shipping, and a pair of 8-port 2.5GbE-T / 2-port 10GbE SFP+ D-Link switches for ~$135 (before tax) as well.

So if the NICs were open box deals, there is a chance it could be the nics, but honestly with everything else you've tried, it seems to be something else. The joys of the bleeding edge...

 

[aigomorla]

The joys of the bleeding edge...

10GBe isn't even bleeding edge... lol...

Also i hear nothing but rants and complaints about rj45 10gbe, hence why i went the proven and most reliable route of SFP+ with DAC cables.
If i need the length, i'll swap over to fiber and optical transceivers, but lucky for me i am only looking at several meters at most from the systems which require the 10gbe.

 

[SamirD]

10GBe isn't even bleeding edge... lol...

Also i hear nothing but rants and complaints about rj45 10gbe, hence why i went the proven and most reliable route of SFP+ with DAC cables.
If i need the length, i'll swap over to fiber and optical transceivers, but lucky for me i am only looking at several meters at most from the systems which require the 10gbe.

At home it is.

Interesting that you're hearing a lot of issues with rj45 10Gb. I know most go the route of sfps and dacs, but most do this because of cost.

 

[aigomorla]

That's what I was trying to do. Or get 10GbE-T on client, direct to switch, and from switch, 2x 1GbE-T to dual 1GbE-T NICs on NAS.

larry this makes no sense, why on heavens would you want client to have 10gbe and not the NAS.
Unless you intend to do a lot of transfers to client, you wont even need that 10gbe.

Also having 2 streams of 1GBe does not = 2GBe, its 2 streams of 1GBe meaning simultaneous connections of 1gbe when used.
Simple analogy is you are not getting a 4 lane highway, you are getting 2 x 2 lane highways, per packet.
Which means when you transfer 1 file or operation, you will be getting the same bandwidth as having a single 1gbe, and not 2gbe which i think your hoping for.

 

[VirtualLarry]

Which means when you transfer 1 file or operation, you will be getting the same bandwidth as having a single 1gbe, and not 2gbe which i think your hoping for.

I was kind of hoping, that somehow, if I transfer multiple files at once, they might go at 2X the speed, with 2X the GbE links on the NAS, compared to just 1X. I don't have 2x 1GbE on the client, but I added a 10GbE-T, so I was hopeful I could push 2x 1GbE streams through the 10GbE-T NIC, and still utilize the 2x NICs on the NAS. Maybe not really doable.

 

[aigomorla]

I was kind of hoping, that somehow, if I transfer multiple files at once, they might go at 2X the speed, with 2X the GbE links on the NAS, compared to just 1X. I don't have 2x 1GbE on the client, but I added a 10GbE-T, so I was hopeful I could push 2x 1GbE streams through the 10GbE-T NIC, and still utilize the 2x NICs on the NAS. Maybe not really doable.

I dont think you can do that, or i have never heard of anyone even attempt that, because you always ran into the bottle neck of one of the pc's having 1gbe.

typically when you use LACP with multi nics, its so you have multi clients accessing that server and not getting bottlenecked through a single 1gbe.
I have not seen a case where a single server could hand out all nics on LACP load balancing, to a single terminal.

 

[mxnerd]

QNAP SMB multichannel still in experiment ? and you have to enable it using putty client.

 

[Rifter]

10GBe isn't even bleeding edge... lol...

Also i hear nothing but rants and complaints about rj45 10gbe, hence why i went the proven and most reliable route of SFP+ with DAC cables.
If i need the length, i'll swap over to fiber and optical transceivers, but lucky for me i am only looking at several meters at most from the systems which require the 10gbe.

Im using 2 intel nic's for RJ45 10GBe and it works fine, just dont cheap out on the cable.

 

[Muadib]

Im using 2 intel nic's for RJ45 10GBe and it works fine, just dont cheap out on the cable.

Cat 6?

 

[VirtualLarry]

So, bottom line is, my QNAP NAS units, at least, when they were on 4.3.6.x firmware, DO NOT support SMB multi-channel, even though they support SMB 3.0. (Someone commented that you can enable it, in the version of SAMBA that they use, by editing a config file, but that it may be "unstable".)

Newer 4.4.1.x firmware for my TS-451 mentioned a newer version of SAMBA, but I still don't think that SMB-MC is enabled by default.

Hopefully, I'll have my 2.5GbE-T Asustor (RealTek chipset) USB 3.x adapters here soon enough to test out my TS-451 with the QNAP 5GbE-T adapter, all plugged into a D-Link 8-port 2.5GbE-T switch (with 2x 10GbE-T SFP+ uplinks).

Btw, my first Asus 10GbE-T PCI-E 3.0 x4 card, seems to have croaked, nothing in device manager, nada. Just gone. Not sure what happened, it was an open-box special.

 

[VirtualLarry]

After discovered the dual-port 2.5GbE-T RealTek 8125-based cards for $40 on ebay, now I'm thinking of having a segregated storage LAN and internet LAN.

Would it be better to handle that with VLANs? On the NIC port, or on my managed 8-port 2.5GbE-T D-Link switches?

Would I define a VLAN for storage, and one for internet, and then set those VLAN IDs on the client NIC port(s), and then set some sort of "trunk port" with those VLAN IDs on the managed switch, and then connect the trunk uplink port, to another switch, possibly managed, with the NAS units on it, and one trunk uplink port, to my router, for internet? How do I strip the VLANs off, before passing them over the trunk port, or does it do that automatically?

I'm not all that well-versed in VLANs and trunking. I get the basic idea, I think. VLAN IDs create "virtual" networks, but I'm shaky on whether they should be defined by the port(s) on the client NIC, or on the switch that they are connected to, or both?

 

[abufrejoval]

Interesting video, thanks. He "recommends" the Asus XG-C100C 10GbE-T PCI-E 3.0 x4 card.

I picked up both of mine for ~$41 ea. after tax and shipping, and a pair of 8-port 2.5GbE-T / 2-port 10GbE SFP+ D-Link switches for ~$135 (before tax) as well.

The Asus card is based on the AQC-107 which is a PCIe 2.0 x4 chip: Four lanes, will work on PCIe 3 and 4, but still only at PCIe 2.0 line rates.

The AQC-111 chip (used for the QNAP) seems to be PCIe 3.0 x1: So one PCIe 3.0 lane only, but 5Gbit PCIe signal rate, if the host supports it: Perhaps a better fit in some cases.

In both cases the +1 chips (108/112) only support up to 2.5 Gbit Nbase-T.

 

[mxnerd]

The Asus card is based on the AQC-107 which is a PCIe 2.0 x4 chip: Four lanes, will work on PCIe 3 and 4, but still only at PCIe 2.0 line rates.

the +1 chips (108/112) only support up to 2.5 Gbit Nbase-T.

You probably got info mixed up.

Aquantia Launch AQtion 5G/2.5G/1G Multi-Gigabit Ethernet Cards (NICs) for PCIe

www.anandtech.com

 

[mxnerd]

Network Expansion Card (Aquantia) - Features

QNAP offers various cost-effective network expansion cards for businesses and organizations to upgrade the bandwidth of their QNAP NAS to accommodate intensive data transfer and virtualization applications.

www.qnap.com

The weird thing is its single port QXG-10G1T (AQC107) requires PCIe 3.0 x4 ?

===

QNAP USB 3.0 ethernet adapter QNA-UC5G1T is based on AQC111U

 

[abufrejoval]

Network Expansion Card (Aquantia) - Features

QNAP offers various cost-effective network expansion cards for businesses and organizations to upgrade the bandwidth of their QNAP NAS to accommodate intensive data transfer and virtualization applications.

www.qnap.com

The weird thing is its single port QXG-10G1T (AQC107) requires PCIe 3.0 x4 ?

===

QNAP 3.0 ethernet adapter QNA-UC5G1T is based on AQC111

Aquantia doesn't seem big on publishing technical detail to the public.
Marvell (the new owners) hasn't been that much better...

This is what I have gathered so far:
ACQ107/108 are pure PCIe 2.0 designs and 12x14mm chips. Lithography isn't mentioned today, but I believe some years back they mentioned the fact that using a smaller-than-Intel lithography for the built-in PHY and Energy-Efficient-Ethernet was the prime reason motivation to launch the company, as it allowed to reach or undercut 3Watts/port, when the competition was at 10Watts/port or worse at 10Gbase-T.

I think original 10GBase-T at Intel might have started on 90 or 65nm so Aquantia 107 may actually be on 45nm or so.

Hard numbers have been scarce or non-existent since...

So ACQ107/108 will work on PCIe v1 at reduced speeds, they'll also work with PCI v3/v4, because those are backward compatible: But to obtain 10Gbit speeds with the 107, you'll have to sacrifice 4 lanes of PCIe 2.0 or better, it cannot take advantage of v3/v4 PCIe, trading lanes for lane-speed.

ACQ111/112 is a new generation, 9x9mm physical size, perhaps again a different lithography (yet perhaps not, because PHY is much more difficult to shrink than logic) and it exist in two variants: PCIe and USB3.

So the ACQ111U in the QNAP is the USB3 variant, there is also the ACQ111C, which is PCIe v3x1.

Both are limited to 5GBit on either end, but because of the USB3 encoding overhead, ACQ111U will only ever deliver 400MB/s (440MB/s with 9k Jumbo packets) of net throughput, which isn't terribly much higher than what you can get from a 2.5GBit/s RealTek RTL8156 adapter (300MB/s).

The ACQ111C (PCIe), which I haven't seen in the wild yet, might offer slightly better throughput, but won't achieve 10Gbit Ethernet bandwidth, unless they create a PCIe v4 variant. Two lanes of PCIe v3 would do the same trick in terms of bandwidth, but require a completely different chip: Don't think that's going to happen, dual 10Gbit at 4x PCIe v3 means going after an enterprise market, where Ethernet isn't factoring on BoM and many SoCs seem to have embedded (yet unused) 10Gbit IP blocks on their South-Bridges already.

Since USB 3.2 doesn't loose as much in encoding as USB 3, perhaps a USB 3.2 10Gbit variant of the ACQ111U would be the better option going forward (ignoring form-factor issues).

I hope that clarifies things a bit...