Switches with 10GbE SFP+

[destrekor]

Quite a few switches have two or four SFP+ ports for 10GbE connectivity. Usually these are utilized for stacking switches, but my question is simple:
can you utilize those same ports as if they are just another switched port?

I'm looking at building an ESXi host with the Xeon D-1500 platform at the heart of it, and most packages typically include two GbE RJ45 connections, and two 10GbE connections. Some motherboards have 10Gbase-T, but a majority seem to have SFP+ modules.

Ideally, I'd use the two 1000base-T connections for WAN/LAN connections within pfSense, and the 10GbE connections for FreeNAS and/or other VMs so that the internal network could potentially have multiple clients with a saturated connection.

That comes down to needing the switch to be capable of treating the 10GbE connections as simply additional switched ports.

Forgive me, as it's been a long time since I was neck-deep in my networking studies, and I never had a chance to work with stacked switches and, at the time, GBIC connections, so I never saw how they were treated in the switch's OS.

 

[RadiclDreamer]

Depends on the model of course but typically yes, I know this is the case for every cisco device ive used.

 

[Genx87]

Wouldn't it be cheaper to use 10GBase T with Cat6a? We are building out a 10Gbe solution right now with SFP+. The trancievers can be expensive.

btw I suggest looking into the free version of Sophos in place of pfsense.

 

[mv2devnull]

We are building out a 10Gbe solution right now with SFP+. The trancievers can be expensive.

There are cheap DAC's for short range (up to 10m?). DAC is a copper cable with SFP+ module attached to both ends. If the switch and server are near each other, that is enough.

Yes, a port is a port. I've seen servers hooked to switch by both fiber and DAC. Even fiber SFP+ is relatively cheap for short connection.

 

[destrekor]

Wouldn't it be cheaper to use 10GBase T with Cat6a? We are building out a 10Gbe solution right now with SFP+. The trancievers can be expensive.

btw I suggest looking into the free version of Sophos in place of pfsense.

Software decisions haven't been made, I'm just strongly leaning towards pfsense and FreeNAS for different reasons. Is the free Sophos software capable of being a traditional router and firewall? Never knew they had a free OS package.


As for 10Gbase-T and Cat6, well, yes, and while a few motherboards offer that from the SoC, most are utilizing SFP+. I wouldn't bother with transceivers, rather, I'd just use Twinax DAC cables between the system and switch. More expensive than Cat6, sure, but still inexpensive. And I've read that 10Gbase-T draws more power and adds latency versus the low power draw of SFP+, even twinax, and effectively no latency.

For the system, I'm trying to keep it ultra low power, not just for energy cost savings, but also for reduced noise and heat. Twinax may just help meet that goal, but I'm not letting the flavor of 10GbE connector be the deciding factor, I'm just trying to find out what options I have and if I can make do with either.

A 10Gbase-T RJ45 connection is nice because it can drop in immediately to my network and function at 1000base-T, whereas SFP+ will require a switch is bought at the same time. And I'd rather just buy the switch than rely on a PCIe NIC expansion card with 2 or 4 LAN ports, because I also haven't decided if I am going mITX or mATX, and whether the motherboard I get has an integrated LSI HBA or if I need to add an HBA card myself. If I go mITX and need to get an HBA card, then I can't add NICs.

I'm still in my "saving up for a project" phase, but this is helping me narrow down what is necessary.

 

[Genx87]

The home version of Sophos is the same as the paid version but with client limitations. You are limited to under 50 clients with the home version. But the AV, IPS, App control, firewall, VPN, and router functionality works. I havent tried to see if HA works. I used a bare metal install on an old PC with two nics. But there is a VM version you can download for VMWare.

I also run FreeNAS at home. Nice free software. Runs on an ancient E6600 with 8GB of ram and 7x750GB drives with a 74GB raptor to run the OS. I use it as an iSCSI target for my Win12-R2 box. But forcing it through a 1Gbps switch. Not doing any VM work yet. If I get back into playing with Hypervisors at home again. I would consider a 10Gbe solution as well.

 

[destrekor]

The home version of Sophos is the same as the paid version but with client limitations. You are limited to under 50 clients with the home version. But the AV, IPS, App control, firewall, VPN, and router functionality works. I havent tried to see if HA works. I used a bare metal install on an old PC with two nics. But there is a VM version you can download for VMWare.

I was just looking into that - looks like it is a terrific UTM for home use, and might serve well as an edge router.

 

[arch113]

The home version of Sophos is the same as the paid version but with client limitations. You are limited to under 50 clients with the home version. But the AV, IPS, App control, firewall, VPN, and router functionality works. I havent tried to see if HA works. I used a bare metal install on an old PC with two nics. But there is a VM version you can download for VMWare.

Sophos may be moving away from the UTM in favor of Sophos XG Firewall. Also free for Home Use, but its not limited by 50 clients, its limits what hardware you can use. Max Single CPU with max 4 cores and 6 gig of ram.

 

[Genx87]

Sophos may be moving away from the UTM in favor of Sophos XG Firewall. Also free for Home Use, but its not limited by 50 clients, its limits what hardware you can use. Max Single CPU with max 4 cores and 8 gig ram.

I havent looked into XG much due to it not being feature complete and the tools to migrate being a year out as of Dec(when I installed UTM 9 at home). And it is my understanding they will set an end date to drop UTM 9 once all of the tools to migrate to XG are finished near the end of 2016. At some point I will migrate to it as well. If they have limitations it will depend on what they are for me to re-evaluate them. A single socket 4 core with 8 GB is plenty for home use. I am running mine on an E5700 with 8GB + 60GB Samsung 830 SSD. Barely get above 5% CPU use with everything turned on. At work we use the SG210s in Active\Passive HA configuration which I think is some 4 core Intel i3 or i5 with 8GB of ram. Our CPU useage rarely goes above 25%. Unless we have an infected machine that floods the firewall trying to call home.

 

[destrekor]

I havent looked into XG much due to it not being feature complete and the tools to migrate being a year out as of Dec(when I installed UTM 9 at home). And it is my understanding they will set an end date to drop UTM 9 once all of the tools to migrate to XG are finished near the end of 2016. At some point I will migrate to it as well. If they have limitations it will depend on what they are for me to re-evaluate them. A single socket 4 core with 8 GB is plenty for home use. I am running mine on an E5700 with 8GB + 60GB Samsung 830 SSD. Barely get above 5% CPU use with everything turned on. At work we use the SG210s in Active\Passive HA configuration which I think is some 4 core Intel i3 or i5 with 8GB of ram. Our CPU useage rarely goes above 25%. Unless we have an infected machine that floods the firewall trying to call home.

Think that limitation applies to physical count only, or to what is made available through a hypervisor?

I'd be running this under ESXi so I wasn't planning on giving it any more than a couple cores and whatever amount of RAM made sense after researching it (I figure 4-8GB).

Ah, here's the requirements from their Home XG page:

What you need

Intel compatible computer with dual network interfaces. (Any previous OS or files on the computer will be overwritten when installing the XG Firewall Home Edition)
Home Edition is limited to 4 cores and 6 GB of RAM. The computer can have more than this, but XG Firewall Home Edition will not be able to utilize it.

 

[XavierMace]

I switched from pfSense to Sophos about a year ago and never looked back. WAY easier to manage and more feature complete out of the box.

Both ran as VM's on my ESXi hosts. 4 vCPU, 8Gb of RAM. vSphere shows in the last month, it's highest CPU usage was 9.68%, memory hovers around 15% used.

Firewall, IPS, Endpoint Protection, and SSL VPN turned on.

Edit: Keep in mind unless you're tossing a lot of RAM and some SSD's at your FreeNAS box, it's not going to have enough speed to make 10GbE really even matter.

 

[destrekor]

I switched from pfSense to Sophos about a year ago and never looked back. WAY easier to manage and more feature complete out of the box.

Both ran as VM's on my ESXi hosts. 4 vCPU, 8Gb of RAM. vSphere shows in the last month, it's highest CPU usage was 9.68%, memory hovers around 15% used.

Firewall, IPS, Endpoint Protection, and SSL VPN turned on.

Edit: Keep in mind unless you're tossing a lot of RAM and some SSD's at your FreeNAS box, it's not going to have enough speed to make 10GbE really even matter.

When you set it as 4 vCPU, is that as 4 cores, or as 4 threads? Curious what kind of CPU resources Sophos would need, because if I can put some of a 6c/12t or 8c/16t to use for firewall/routing, a larger portion toward FreeNAS for general use and Plex work, and have some left over for at least one or more other VMs for messing around, that would be awesome.

And yeah, I doubt I'll ever really make the 10GbE matter. It'd be nice if it would at least go over the 1GbE threshold, but as it'd be a 1GbE network, I highly doubt it. But really, the end goal is to just treat them as extra LAN connections dedicated to FreeNAS. If I get a board with 10Gbase-T RJ45 ports, I'll just team the two ports and connect them to a standard gigabit switch and not worry about a more expensive switch for a little while. But if the best board for the job ends up having SFP+ ports, I gotta connect them somehow and I don't want external transceivers.

 

[XavierMace]

http://www.virtualizationsoftware.com/virtual-cores-virtual-sockets-vcpu-explained/

 

[destrekor]

http://www.virtualizationsoftware.com/virtual-cores-virtual-sockets-vcpu-explained/

That was as I thought. Thanks. Wasn't sure if ESXi handled it differently as compared to Type 2 hypervisors like VirtualBox.

 

[fiberst]

10Gbase-T works for 10GbE transmission technically, but few fiber optic manufacturers and suppliers provide 10G copper transceivers, so SFP+ optical modules seem to be more popular. When distance between the switch and the equipment, say less than 10m, SFP+ DACs are also the suitable choices. Besides, SFP+ DACs are less expensive than SFP+ optical modules.

 

[sdifox]

are you actually having multiple 10gbps servers that need to talk to each other or just one server to multiple clients? I am asking since bonding 3 gigabit ports (1 for pfsense) should be plenty for home use.

10GBE for pfSense would be a waste since it is unlikely your external link is 10gbps.

You could direct attach ESXi host to your SAN if you are building a SAN :awe:

http://www.cables-solutions.com/category/fiber-optic-transceivers/direct-attach-cables

 

[sdifox]

That was as I thought. Thanks. Wasn't sure if ESXi handled it differently as compared to Type 2 hypervisors like VirtualBox.

Just make sure you check the ESXi hardware compatibility list before purchase.

 

[holden j caufield]

I built a whitebox esx server out of my laptop. Was able to accomplish a lot with the vt-d pci passthrough.

Had a sophos vm, freenass vm (passed through the sata controller to the vm so performance was pretty decent as the vm saw the sata controller and drives as innate), it wasn't enterprise stuff so I used the virtual switch. It's been many years but since sophos needed two nics I even got a usb nic passed that through to my spphos vm. My cable modem isn't fast enough so I was fine with that.

I used to worry about compatibility but you can slipstream drivers in with esxi customizer