"The BIG Networking Upgrade" - my home LAN, Strategies? Ideas? (Purchased 8-port 2.5GbE-T switches, cards)

[VirtualLarry]

Let me give you a run-down on my current LAN setup. I have a Gigabit FIOS connection, and a gigabit wired LAN. I have a FIOS G1100 router, and a secondary LAN router that's an Asus, that handles my Wifi. I generally run 3rd-party firmware. I may want to upgrade that router, especially if FIOS deploys a faster than 1Gbit internet connection speed. Exploring my options there, there doesn't appear to be many consumer routers, with an emphasis on forward-looking wired ports, as well as "Wifi 6" (AX). The Netgear AX12 router, is $400, and has a single 5GbE-T port, which I would like to have two, for both WAN and LAN. It does support teaming a couple of included 1GbE-T ports too.

Secondarily, I have several NAS units, none with faster than 1GbE ports on them (yet), but some of them have multiple 1GbE ports for teaming or LAG. I only use unmanaged switches currently, so I can't do LAG groups (yet).

I have some Ryzen AM4-based rigs, none with faster than 1GbE ports on them, either.

I would like to upgrade the home LAN to either 2.5GbE, or 5GbE. Both my client PCs, as well as my NAS units, and eventually, the WAN connection, once FIOS upgrades their offerings.

Currently, I can purchase, either now, or in the very near future:

ASRock TaiChi Ultimate X470 board, with a 10GbE-T AQ chipset onboard, like almost $300.

There are a number of X570 boards, now showing up on mfg's web sites, like the ASUS ROG STRIX X570-E and -F. The -E has 2.5GbE-T (RealTek) onboard + 1GbE Intel, as well as 2x2 AX Wifi 6 (Intel). Looks like a real option, if I want to upgrade my mobos, but that would cost me to purchase new CPUs, as my 1st-Gen Ryzen CPUs aren't supported on X570, according to a certain slide.

Alternatively, and cheaper, would be to get either:
TrendNet PCI-E 3.0 x1 2.5GbE card (RealTek) for $50 (Avail. starting in June)
TrendNet USB3.2 Gen 1 Type-C 2.5GbE (RealTek) for $50 (Avail. now)
QNAP QNA-QC5G1T (IIRC), which is a USB3 AQ chipset that does 5/2.5/1GbE-T. Unknown price, only one third-party seller on Amazon for $212, should be $70-80 once QNAP gets them in their US Store site. (*This will work with my two 4-bay QNAP NAS units, to upgrade them to 5GbE-T capability.)
AQ 5GbE (5/2.5/1GbE) PCI-E 3.0 x1 for $80 on ebay, or $70 on Amazon.
AQ 10GbE PCI-E 3.0 x4 for $100-110 on Amazon
Club3D 2.5GbE (RealTek) USB3 Type-C or Type-A (two different models), the Type-A model is on ProV for $37 right now.

Those appear to be my client options. There's also an Intel and a Killer AX Wifi 6 mPCI-E card for roughly $30 from third-party sellers on Newegg. (Though, I prefer PCI-E for wired NICs, and USB 3.x for Wifi.)

For switches, I splurged one night when someone on here sent me an Amazon GC anonymously for a few bucks. (Thanks, whomever you are!!!), and bought a GM110X or something like that. It's a Netgear switch, with 8x 1GbE ports, and 2x 10GbE-T (multi-Gig, I hope). I should have spent more, and gotten the managed one,. to take advantage of the LAG groups on the managed one. It's possible that this one may still support "Passive LAG / LACP" through my NAS configurations, with my NAS units.

There's also some 10GbE-T 8-port with 2x SFP+ uplink ports, a TrendNet for $550, and a 2.5GbE-T 8-port with the same for $330 right now on sale from BeachCamera on Amazon, and from BuyDig on ebay.

I'm not sure, if I get those QNAP USB3 Type-C 5GbE dongles with the AQ chipset, and then go through with getting AQ-chipset PCI-E 3.0 x1 cards for my client PCs, if it would even be worth spending money on the 8-port TrendNet 2.5GbE switches, if that would cut my performance down to 2.5GbE after spending all of this money, or if I should spend slightly MORE money, and get 8-port 10GbE-T (multi-gig) switches for $550.

Part of the problem is, due my physical layout of my apt., I may need TWO 8-port multi-gig switches. If I go with the TrendNet 2.5GbE ones, they are "Web managed", so I could run TWO ethernet CAT6 cables between them, and get sort-of 5Gbit/sec throughput, IF I had two clients on the side of the switch hitting the NAS units connected to the other switch.

So, it looks like it will be prohibitive to go full 10GbE-T (w/multi-Gig) on the client side, due to needing either expensive ($300+) mobo replacements, or PCI-E x4 slots for the cards, which I don't have free in all of my PCs, since I current have dual GPUs for mining in them. If I were to stop mining, and pull one GPU and keep one GPU in each box, I could go full 10GbE-T on PCI-E x4 cards. In that case, I would get the Intel dual 10GbE-T cards, for reliability, and maybe LAG to the switch on each of my machines. That seems like crazy overkill right now for me, though.

Really, my goals, other than to say that I've done it, and eventually down the road, provide for a 2.5Gbit/sec or faster internet connection, is to speed up things like bulk ISO transfers, bulk transfers between NAS units, and backups and restores from the NAS units.

I would be happy with an increase from 1GbE-T to 2.5GbE-T, but I wonder, if I'm spending that much on a switch, and the 5GbE-T client NICs, both PCI-E 3.0 x1 and USB3.x, aren't much more expensive, and I'll have to buy the QNAP 5GbE-T AQ-chipset NICs anyways to upgrade their capabilities (because they are supported in their NAS OS), then I could conceivably buy the two 10GbE-T 8-port switches (w/multi-gig), and then have a 5GbE-T LAN throughout, and then just when FIOS deploys something faster, upgrade the primary LAN router then. (May not be for a year or so more.)

Comments? I know that this was a bit long.

 

[mxnerd]

TLTR.

Why not setup a free ESXi environment with either one or two cheap 10GbE network cards? (You probably need 2 to get connected status in VM)

https://www.amazon.com/Mellanox-Connectx-2-PCI-Epress-Interface-MNPA19-XTR/dp/B016OYD0D4

https://www.reddit.com/r/homelab/comments/6dukrk/mellanox_connectx2_and_esxi_65

https://imgur.com/a/Ngvv6

or even 40GbE infiniband? https://www.bussink.ch/?p=1572

You also get a lot faster bandwidth between VMs.

You add GbE adapter on ESXi server to communicate with outside world.

In ESXi VM, even though the virtual ethernet adapter only show 1GbE, it actually will run whatever the adapter is capable of.

There is no need to stick to 10/5/2.5/1GbE BaseT RJ45 wiring/technology.

Run about everything on ESXi server and remote into it when necessary from your PC.

Also get a cheap used Dell RAID card on eBay.

****
Don't expect these new 10/5/2.5GbE RJ45 adapters be recognized by ESXi itself unless you pass through these adapters to VM (in which case you can install drivers for them) if you decide to take this route. ESXi does not support Realtek adapters unless you pass through the adapter to a VM, for exampe.

****
The best part is you can have multiple virtual ethernet cards bind to different real network cards.

 

[dave_the_nerd]

I know I'm not going to like the answer, but what's the cost/benefit here?

 

[VirtualLarry]

Well, primarily, such that my backups (full, especially) take less time on my main desktop.

I have:
SATA SSD (OS)
SATA SSD (secondary OS install)
SATA SSD (unpartitioned)
4TB spinner (HDD) (used for download archive, ISO archive backup (also on NAS), and BTC blockchain info)

I ordered a pair of 256GB Patriot Scorch M.2 NVMe PCI-E x2 SSDs, and I have a pair of 6TB X300 7200RPM Toshiba HDDs. Was planning on perhaps reformatting, and setting up StoreMI with a 256GB "fast drive", and the 6TB 7200RPM as the primary drive. Just to try out StoreMI and see how acceptable it would be day-to-day. If that seems sluggish to me (doubtful, I'm not a heavy user), then I have a pair of 1TB 660p Intel QLC NVMe x4 SSDs that I can pull out of another rig.

During a backup, the bottle-neck appears to be the 1GbE LAN. Disk, RAM, and CPU aren't pinned at all. Although, Macrium Reflect Free edition may be single-threaded for backup operations.

 

[mxnerd]

The Failure Rate: Downsides of a JBOD

StoreMI suffers from the same lack of protection from drive failures. There is no mechanism by which the data is protected if one drive in the tier fails

https://www.anandtech.com/show/12625/amd-second-generation-ryzen-7-2700x-2700-ryzen-5-2600x-2600/7

I really don't see any benefit using StoreMI except for experimental purpose.

=

And why full backup? Why not just backup what's essential? ISO can always be re-downloaded from internet.

 

[VirtualLarry]

Well, to address both points:
1) Yes, I want to experiment and experience StoreMI for day-to-day operations, and observe the performance I get with those drives. I want to know if it's worth-while building PCs with that config for clients.
2) I use Macrium Reflect Free edition, which allows you to schedule backups, that use VSS to allow full image backups, while still in Windows. So far, it seems to work great. It does a differential backup every day, and a full backup every few weeks. The full backup is what takes the most time (hours, currently), so that's what I want to optimize with a network upgrade, the worst-case time-usage scenario for backup.

Plus, the networking upgrade(s) will allow moving my ISO collections to and from my desktop PC to my NAS unit(s) a lot faster.

Edit: And OH, my point #2 addresses your first statement, that StoreMI offers no redundancy should a drive have an error or fail. That's why I do image backups, rather than file backups.

I do move bigger files to the NAS unit. Eventually, I want to get into actual NAS replication services, which I believe my two QNAP units can do, somehow. I do it manually now.

A 5GbE-T link on both NAS units, with a 10GbE-T/multi-gig switch in-between would help immensely with that, I think.

 

[mxnerd]

What's your QNAP models and HDD drives? If the theoretical RAID throughput on the NAS models could not exceed 1GBe or just over a bit, adding 5GBe link won't help.

===

BTW, if your QNAP supports SMB 3.0 and have 2 Gbe interfaces and all of your PCs run Windows 8.1 or 10, then you can double your bandwidth and don't need switches that support LAG.

https://www.qnap.com/en-au/how-to/tutorial/article/how-to-use-smb-3-0-in-qts-4-2/

 

[VirtualLarry]

Yes, all of my PCs are Windows 10, except for my HTPC, which is still Win7.

That's a good point about the 5GbE-T USB3 adapter. I was kind of wondering about that, if the NAS barely has enough CPU time to pump 110MB/sec through the 1GbE port, how much CPU time is it going to suck away, to tend to a 5GbE-T over USB3? (Most routers with USB3 ports don't perform very well compared to x64 PCs, due to CPU overhead and the relatively "weak" ARM CPUs in them... or even MIPS.)

I have a TS-431 and a TS-451 (no Plus, no P, just the original models).

I just updated the firmware on both of them today, and added the support for the QNAP 5GbE-T adapter, but the TS-431 with a dual-core ARM 1.7Ghz (I think?), was only doing 60-65MB/sec write speeds to a 4-drive RAID-5 array with 4x WD Red 8TB 5400RPM HDDs. (I could have sworn that both of my NAS units could push 100MB/sec both read and write, at least, the TS-451 does, or used to, I think that the HDDs in that one are getting a little older. I have 4x Seagate DM 5TB 5400RPM HDDs in that one. The ones that I shucked myself, that were all over ebay as "new pull" (shucked), and reviews said were horrible for NAS usage. *Knock on wood*, I guess, because mine have held up a few years at least. I shucked them myself, but they do seem to be slowing down... read speeds for the RAID-5 as low as 30MB/sec... but they pass a RAID scrub and SMART test every month, so I don't know. Never had a bad sector thrown from them either.

 

[mxnerd]

According to QNAP website, your TS-431 & TS-451 can reach 111MB/s and 224MB/s separately, using 7200RPM drives with RAID-5 configuration.
https://www.qnap.com/en-us/product/ts-431
https://www.qnap.com/en-us/product/ts-451 (it claims that the test uses only one interface, but I think it should be 2).

Since your read/write performance are under the test results by QNAP, upgrading your gears does not make any sense at this time.

Install 2 GbE NICs for your PC, connect both to an unmanaged switch, and also enable SMB 3.0 on TS-451, connect to the same switch with 2 built-in NICs should satisfy your needs at the moment.

 

[zeroidea]

Just FYI, Netgear's 10-port multi-gig (variety of 1G/2.5G/5G/10G ports) MS510TX switch is down to $210 on Amazon. And Aquantia's 10G NIC is just under $70 (after a 25% discount at checkout).

 

[VirtualLarry]

I think, that I've mostly decided, to either move to an all-2.5GbE-T (using $30 Syba / I/O Crest PCI-E x1 cards with the RealTek chipset onboard, TrendNet equivalent is $50, that's too close to the Aquantia 5GbE-T PCI-E card, at $70, which I've also considered, so that I only have to upgrade the desktops once, more or less.)

My basically only option for my two QNAP NAS units is their Aquantia-based 5GbE-T USB3.x adapter, for $80 on Amazon.

So that leaves the switches. I could either go with 1-2 TrendNet 2.5GbE-T 8-port switches ($330 ea.), or 2 Netgear Multi-Gig switches, the ones with the 4x 1GbE, 2x 2.5GbE, 2x 5/2.5GbE, and 1x 10GbE for back-haul.

I feel like the Netgear switches might be the better option. I think that the TrendNet switches have a pair of SFP+ 10GbE ports on them as well, for uplink/downlink, but I don't want to be stringing fiber cables over doorways. (Plus, I have no experience with consumer fiber for switches. Prefer Thinnet.)

If only TrendNet would come out with another one of those 8-port 2.5GbE-T switches, with two 10GbE-T (not SFP+) uplink/downlink ports, then I would get that one in a heartbeat (funds permitting), if it was under $400.

 

[SamirD]

I think since you're only trying to improve bandwidth to the NAS unit that mxnerd's suggestions are dead-on, especially since your NAS isn't running right. To give you some comparison, my intel ss4200-e transfers at those speeds and it's a decade old design.

 

[abufrejoval]

I think, that I've mostly decided, to either move to an all-2.5GbE-T (using $30 Syba / I/O Crest PCI-E x1 cards with the RealTek chipset onboard, TrendNet equivalent is $50, that's too close to the Aquantia 5GbE-T PCI-E card, at $70, which I've also considered, so that I only have to upgrade the desktops once, more or less.)

My basically only option for my two QNAP NAS units is their Aquantia-based 5GbE-T USB3.x adapter, for $80 on Amazon.

So that leaves the switches. I could either go with 1-2 TrendNet 2.5GbE-T 8-port switches ($330 ea.), or 2 Netgear Multi-Gig switches, the ones with the 4x 1GbE, 2x 2.5GbE, 2x 5/2.5GbE, and 1x 10GbE for back-haul.

I feel like the Netgear switches might be the better option. I think that the TrendNet switches have a pair of SFP+ 10GbE ports on them as well, for uplink/downlink, but I don't want to be stringing fiber cables over doorways. (Plus, I have no experience with consumer fiber for switches. Prefer Thinnet.)

If only TrendNet would come out with another one of those 8-port 2.5GbE-T switches, with two 10GbE-T (not SFP+) uplink/downlink ports, then I would get that one in a heartbeat (funds permitting), if it was under $400.

I got a Buffalo 12-Port NBase-T switch, that will go to 10Gbit on every port, yet support 1/2.5 and 5Gbit as well.
It was about $50/port, quite a welcome change from the old days. It seems to use three interconnected Aquantia AQR407 dies internally while the 8-port variant uses two four-port dies: So compatibility with Aquantia based AQC 107 NICs is a given, but I also didn't find any issues with Intel and Broadcom 10GBase-T NICs (those, however, like more cooling than my noise optimized desktops provide), the 2.5 NetGear USB3 NIC or any Gbit NIC.

The only issue I had with the switch was fan noise: That is unfortunately an issue most multi-gig switches share, as they put tiny fans into tiny chassis and then spin them at high speeds to dissipate ~40 Watts of power. I hacked the chassis with Noctua fans at static speeds, and it's been working fine without overheating ever since.

These days I'm totally pissed off that you cannot get PCIe x1 NBase-T NICs designed for PCIe v3 or v4: They all require 4 lanes, which on an x570 board are almost enough for 100GBit/s. The 10Gbit USB 3 ports which are allocated an single v4 lane there prove the point, but unfortunately you can't actually buy USB 3 to 10Gbit NBase-T adapter either: The highest I have seen was the 2.5Gbit from Netgear, which unfortunately only comes with USB-C (so I had to buy an extra powered USB-A to USB-C adapter to make it hit USB 3 speeds and 300MBit/s effective iperf3 performance). Still it's an option for high-end notebooks and other systems without available x4 slots. Btw. the Netgear 2.5Gbit chip also sells as an x1 PCIe card, but I haven't seen anything 5Gbit with x1 or USB, even if announcements even from Aquantia have been around for years.

 

[VirtualLarry]

There's a USB3.x to 5GbE-T from QNAP, it's $79.99 from Amazon. It uses an Aquantia chipset. (*Apparently, they make a USB variant. I don't think that they're using a USB3.x to PCI-E x1 bridge chip, and then another PCI-E x1 5GbE-T chip... but that could be a possibility.)

Edit: Also, Amazon has an Aquantia 5GbE-T PCI-E x1 card, too, it's $69.99 I think. The Aquantia 5GbE Pro or something like that. Has a heatsink.

https://www.amazon.com/Type-C-5GbE-Adapter-QNA-UC5G1T-QNAP/dp/B07RKLQPLP/

https://www.amazon.com/Aquantia-NIC-4-speed-Ethernet-Network/dp/B07BZ28BCW/

Edit: I haven't seen the Netgear 2.5GbE offerings, but I've seen ones from TrendNet (maybe you meant TrendNet?) for USB3.x to 2.5GbE-T and PCI-E to 2.5GbE-T, both RealTek chipsets, I believe. Both are $49.99 @ Amazon. Kind of pricey.

I've seen PCI-E x1 cards from Syba and I/O Crest on ebay and Amazon, with the RealTek PCI-E x1 chipset for 2.5GbE-T, for $29.99, or discount for qty on the ebay offer. Been thinking of going with those. RealTek generally has decent software support for "generic" NICs with their chipsets on them. I've used Syba and I/O Crest cards before from Newegg, mostly dual- and quad-port SATA6G PCI-E cards, they worked OK, had one DOA, possibly my fault (static?). Other than QC issues, the cards are usually OK. (Those were with an ASMedia SATA chipset, and Marvell on the quad-port, I think.)

 

[abufrejoval]

There's a USB3.x to 5GbE-T from QNAP, it's $79.99 from Amazon. It uses an Aquantia chipset. (*Apparently, they make a USB variant. I don't think that they're using a USB3.x to PCI-E x1 bridge chip, and then another PCI-E x1 5GbE-T chip... but that could be a possibility.)

Edit: Also, Amazon has an Aquantia 5GbE-T PCI-E x1 card, too, it's $69.99 I think. The Aquantia 5GbE Pro or something like that. Has a heatsink.

https://www.amazon.com/Type-C-5GbE-Adapter-QNA-UC5G1T-QNAP/dp/B07RKLQPLP/

https://www.amazon.com/Aquantia-NIC-4-speed-Ethernet-Network/dp/B07BZ28BCW/

Edit: I haven't seen the Netgear 2.5GbE offerings, but I've seen ones from TrendNet (maybe you meant TrendNet?) for USB3.x to 2.5GbE-T and PCI-E to 2.5GbE-T, both RealTek chipsets, I believe. Both are $49.99 @ Amazon. Kind of pricey.

I've seen PCI-E x1 cards from Syba and I/O Crest on ebay and Amazon, with the RealTek PCI-E x1 chipset for 2.5GbE-T, for $29.99, or discount for qty on the ebay offer. Been thinking of going with those. RealTek generally has decent software support for "generic" NICs with their chipsets on them. I've used Syba and I/O Crest cards before from Newegg, mostly dual- and quad-port SATA6G PCI-E cards, they worked OK, had one DOA, possibly my fault (static?). Other than QC issues, the cards are usually OK. (Those were with an ASMedia SATA chipset, and Marvell on the quad-port, I think.)

Yeah, I keep 'seeing' those 5Gbit USB3 announcements from Aquantia, too... for years, but no product to actually buy.

For the QNAP amazon.com won't deliver to .de and amazon.de only has them as 'unavailable' since months...

And that one does seem a little odd, like you described: While the original Aquantia USB3 NBase-T chip was a straight-forward USB3 to Ethernet design, the QNAP adapter is split into a USB3 to PCIe x1 and then a PCIe x1 to Ethernet solution involving 2 distinct chips: There is a story there, but nobody is talking!

The NetGear 2.5 Gbit USB variants have a USB-C connector and unfortunately many of my PCs have only 5GBit USB-A ports. A simple physical plug adapter doesn't work, because the NetGear needs more than USB 2 type power and tries to negotiate for more but fails on a passive form factor adapter, falling back on a USB 2 data rate or ~40MB/s, not the 300MB/s I see on USB-C ports.

It requires an adapter that supports power delivery mangement, which is almost as expensive as the 2.5Gbit adapter on its own. Still, I got one from DeLock, just to make sure that 2.5GBit on a 'standard' USB3 type A port with 5Gbit maximum data rate *would* work: It does, and so should 5Gbit Ethernet then, once NetGear (or whoever) decides to launch that variant, again announced, but not available (and it still fits the power envelope at 5Gbit: 10Gbit used to be 5 Watts only for the PHY when it launched).

BTW: Try finding the 2.5 Gbit or 5 Gbit chips that NetGear uses on the RealTek site... nothing there, or in Chinese, only.

I've also seen the x1 cards for 2.5Gbit. My problem there is that for the smaller J5005 based PCs I am using Mini-ITX chassis, which don't have any space at the rear, next to the back plate, even if the Mini-ITX boards have a free x1 PCIe v2 slot. They also have an M.2 E type slot with another PCIe x1 lane dedicated to it, good for almost 5Gbit of bandwidth.

Funnily enough there are actually 1Gbit RealTek Ethernet M2. E-type boards with internal Ethernet ports available from DeLock, using one of the 6 potential PCIe v2 lanes on Atoms, but they only support Gbit, not 2.5 or 5 Gbit.

Again, I think every points at PHY power consumption at these higher data rates being the issue: It seems linear or higher, never good in silicon.

 

[extide]

If you haven't already bought your switch I would say this one is a no brainer: https://www.fs.com/products/72944.html
24x 1G ethernet ports and 4x 10G SFP+. $280 Fully managed. You can get SFP modules from fs.com for super cheap too.

The 10G base T SFP+ modules that they sell also do NBaseT as well, so you get compatibility with the 2.5/5g stuff. You could use these in the SFP+ ports if you don't want to mess with fiber, but if you have SFP+ ports on your pc/server/nas it's much cheaper to go fiber or even DAC (Direct Attach Copper).

I just deployed a bunch of fs switches at work and have been pretty impressed with them for the price and am seriously considering that one for home.

 

[VirtualLarry]

The 10G base T SFP+ modules that they sell also do NBaseT as well, so you get compatibility with the 2.5/5g stuff.

Do those ONLY work for "FS" switches, or will they work on that TrendNet 8-port 2.5GbE-T w/2x SFP+ up/downlink 10Gb ports? That would be a perfect solution for me.

https://www.amazon.com/TRENDnet-2-5GBASE-T-Lifetime-Protection-TEG-30102WS/dp/B079Z2SHCX/

 

[extide]

Check out this: https://www.fs.com/products/66617.html the "Customized" option -- they can make all of their SFP/SFP+ modules work for basically ANY brand.

 

[abufrejoval]

If you haven't already bought your switch I would say this one is a no brainer: https://www.fs.com/products/72944.html
24x 1G ethernet ports and 4x 10G SFP+. $280 Fully managed. You can get SFP modules from fs.com for super cheap too.

The 10G base T SFP+ modules that they sell also do NBaseT as well, so you get compatibility with the 2.5/5g stuff. You could use these in the SFP+ ports if you don't want to mess with fiber, but if you have SFP+ ports on your pc/server/nas it's much cheaper to go fiber or even DAC (Direct Attach Copper).

I just deployed a bunch of fs switches at work and have been pretty impressed with them for the price and am seriously considering that one for home.

Not bad, not bad, but for the home: I don't think the price is right.

They sell the NBase-T JBICs at ~€50, which is pretty much the per-port cost on a Buffallo NBase-T switch: You just got the PHY, but you're still missing the switch.

I just picked up 4 Aquantia AQN-107 10Git/NBase-T NICs at €75/piece (including VAT), while FS sells Intel 82599EN based SFP+ NICs at €97 (not sure if that includes VAT): Those are x8 lane PCIe v2 designs launched in 2011 on a 65nm process. Software support should be excellent, they include SR-IOV, iSCSI, perhaps even FCoE but those things never really took off. They are ancient, outdated, power hungry even without 10GBase-T PHYs and designed with server air flow in mind: I have had their 10GBase-T cousins die in desktops optimized for low-noise not air-flow.

The fanless switch sounds nice, but you really only get 4 NBase-T ports (if you purchase 4x €52 SPFP+/NBase-T modules). 24 Gbit ports cost close to nothing these days, but of course not with a 10Gbit uplink or management.

Direct connect cables are only really cheap, if you have friends in the network team. If you need to buy them regular, RJ45 is hard to beat.

I'd stick with the 12-Port Buffalo NBase-T switch I bought at €50/port based on three Aquantia 407 chips for time being: Stupid (unmanaged), fast, inaudible (after some hacking) and affordable.

 

[extide]

That's why I mentioned it's much cheaper to NOT use 10GbaseT. The 10G fiber SFP+'s are $18 and fiber cables are like $5 for 10 ft. A passive direct attach copper cable is $14 for 10ft, hardly expensive.

I mean if you want >4 10G ports then yeah the buffalo switch seems like a reasonable option, but if you don't need that many ports this thing is a great price. Plus its an enterprise grade switch with all the L2+/L3 lite features.

 

[abufrejoval]

That's why I mentioned it's much cheaper to NOT use 10GbaseT. The 10G fiber SFP+'s are $18 and fiber cables are like $5 for 10 ft. A passive direct attach copper cable is $14 for 10ft, hardly expensive.

I mean if you want >4 10G ports then yeah the buffalo switch seems like a reasonable option, but if you don't need that many ports this thing is a great price. Plus its an enterprise grade switch with all the L2+/L3 lite features.

Good that you can get direct connect cables that cheap these days: I still have prices much higher in my mind, from when these things launched.

The main attraction of direct connect used to be power, but these ancient Intel NICs eat any potential benefit. There are actually Aquantia (Marvell now...) based NICs with SFP+ connectors appearing these days, that may lower the both price and energy consumption below NBase-T levels.

I still find it hard to beat the ease of working with just plain RJ45, mixing anything from 100 to 10000MBit seamlessly.

I only wish they'd offer 10Gbit on a single PCIe lane, instead of wasting 4-8 lanes, as they do currently.

I am using QSFP+ direct connect cables on Mellanox ConnectX-5 VPI 100Gbit adapters (16x PCIe v3) at work with host chaining, but that's another class of hardware and speed.

 

[extide]

Yeah hopefully with PCIe 4.0 we will see new chipsets that are 1 lane for 10Gb. You could almost do it on PCIe 3.0, but not quite. A PCIe 3.0 x2 one would be nice, though.

 

[aigomorla]

I know I'm not going to like the answer, but what's the cost/benefit here?

depends on your hardware...
I went with 2 melanox 10gb SFP+ cards and a mikrotik switch which supported SFP+ on 10GB.

The cost was about 20 for each cards (Ebay), 15 for both cables(DAC Direct Attached Copper SFP+ cables), and about 180 for the switch.
https://www.amazon.com/Mikrotik-CRS...ik+switch&qid=1565018332&s=electronics&sr=1-7
^ that guy using switch OS...

What is the benefit you ask?
If you do a lot of transfers from PC to PC...
My server backups a lot of stuff, and sometimes its a TB.
So dumping a TB of data to a NAS connection over 10Gb.. is well..

You decide.. and no that is not internal transfer from SSD -> to SSD... its a straight up over network transfer from RAID5 -> RaidZ2 on 2 different PC's both using magnetic spinners.

If you need more 10gbe... there is also this guy:
https://www.amazon.com/MikroTik-CRS...gb+switch&qid=1565018965&s=electronics&sr=1-3

you would need to config it to swOS tho, which is switchOS and not routerOS.

 

[SamirD]

Great to see some great numbers without SSDs! How many drives are in each raid to get that type of throughput?

 

[aigomorla]

Great to see some great numbers without SSDs! How many drives are in each raid to get that type of throughput?

6 x HGST He08's on the server with an Adaptec 6805, and 12 HGST He10's on the NAS running off 2 x LSI 9211's in IT mode on FreeNAS.