My home Gigabit Network

[HVAC]

I have learned much from AnandTech forums over the years, so I thought I would report how it has helped me.

I recently bought a barebones system, CPU, memory, drives, gigabit switch, and a cheap gigabit NIC to replace an aging server. I put it all together and started testing throughput to the server from my desktop (an Nvidia 405 chipset system with an X2 3000+ cpu).

Over stranded CAT5e cable run throughout my house (yes..yes...I know solid is better for permanent installations....but I make my own cables and didn't want to buy two spools), I was getting throughput of ~250Mbps using nuttcp to test. Jumbo frames didn't seems to make a difference.

I thought maybe the cable runs were marginal, so I brought my server up to my desk and direct connected it to the desktop computer. ~250Mbps over a 3 foot storebought CAT5e cable. My thought was that it was the NICs, builtin on the desktop and PCI bus Realtek 8169 chip on the server.

I bit the bullet and ordered 2 Intel Pro/1000 PTs (PCI-E x1).

Now I am getting ~950Mbps using nuttcp over my homemade cables with the switch in the middle, everything set on 4088 Jumbo frames.

My throughput still isn't fantastic using large file transfers, but I can now blame OS file system overhead, disk throughput, or phases of the moon instead of the network.

Edit: I read my post and then realized that I had promised to tell how these forums had helped me. 1) General networking knowledge 2) Component selection 3) Where to buy.

 

[p0lar]

According to some, you'll also need solar flare shields, 32-CPU systems, 4Gb fiber SANs and 10gbe cards to achieve this number. Many are of the opinion that gigabit = 250mbit -- 'it is what it is' or 'that is normal, deal with it', which is patently false and a constitutes a limited frame of mind. Low expectations will always yield low results.

Congrats on the successful implementation. Now, what are you going to do with your new-found bandwidth?

 

[JackMDS]

Offer it together with a $400 PSU, and $300 Killer NIC to the Ripley Museum.

 

[HVAC]

Originally posted by: p0lar
Congrats on the successful implementation. Now, what are you going to do with your new-found bandwidth?

Declare victory and retire in relative obscurity!

No.....I am working toward being able to have a server(s) feed multiple video streams to various clients throughout the house. I have a wife and seven kids, so at some point I anticipate that everyone will want to watch/surf/online game something different.

I can at least try to fulfill their desire.

On the other hand...I could simply kick everyone outside to go play and start working on kid eight.

 

[highwire]

Congrats on your new found free time.

Since gigabit is 100base on 4 pairs instead of 2, simultaneous bi-directional transmission with very tricky coding and modulation, I suspect substantial differences in quality will show up between various suppliers. But cabling may not be as critical as I though it might be, since the frequencies involved are no more than 100base.

Your experience seems to bear that out.

 

[p0lar]

Originally posted by: HVAC
Declare victory and retire in relative obscurity!

No.....I am working toward being able to have a server(s) feed multiple video streams to various clients throughout the house. I have a wife and seven kids, so at some point I anticipate that everyone will want to watch/surf/online game something different.

I can at least try to fulfill their desire.

On the other hand...I could simply kick everyone outside to go play and start working on kid eight.

You know, with bandwidth, it always seems to come down to the p..n; and, with 7 kids, you're obviously good at more than just home gigabit networking!

Double congrats on the networked family too -- many nodes.

 

[robmurphy]

Hi HVAC.

I cannot get my machines above 360 Mbs. The NICS are intel Pro GT in a PCI slot as I do not have any spare PCIE slots.

Could you please answer a couple of questions.

What OS is on the server and the desktop?

Did you make any changes in the registry, like the TCPwindow size?

Thanks in advance.

Rob Murphy.

 

[alaricljs]

robmurphy, your issue is the bandwidth of the PCI slot. End of story.

 

[Madwand1]

Originally posted by: alaricljs
robmurphy, your issue is the bandwidth of the PCI slot. End of story.

I've hit > 800 Mb/s with a Realtek PCI NIC, and > 900 Mb/s with an Intel PCI NIC.

Here's an example using the Realtek, with a nForce3 NIC on the other side; 7 KB jumbo frames were used:

D:\tools\bench\iperf>iperf -c 192.168.0.141 -l 64k -t 12 -i 3 -r
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 8.19 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.0.141, TCP port 5001
TCP window size: 8.19 KByte (default)
------------------------------------------------------------
[228] local 192.168.0.139 port 49190 connected with 192.168.0.141 port 5001
[ ID] Interval Transfer Bandwidth
[228] 0.0- 3.0 sec 307 MBytes 819 Mbits/sec
[228] 3.0- 6.0 sec 310 MBytes 827 Mbits/sec
[228] 6.0- 9.0 sec 310 MBytes 827 Mbits/sec
[228] 9.0-12.0 sec 312 MBytes 832 Mbits/sec
[228] 0.0-12.0 sec 1.24 GBytes 825 Mbits/sec
[260] local 192.168.0.139 port 5001 connected with 192.168.0.141 port 12642
[ ID] Interval Transfer Bandwidth
[260] 0.0- 3.0 sec 305 MBytes 814 Mbits/sec
[260] 3.0- 6.0 sec 325 MBytes 866 Mbits/sec
[260] 6.0- 9.0 sec 325 MBytes 867 Mbits/sec
[260] 9.0-12.0 sec 328 MBytes 873 Mbits/sec
[260] 0.0-12.0 sec 1.28 GBytes 855 Mbits/sec

edit: correction on jumbo frames

 

[p0lar]

Intel PCI NIC.

Intel 8255x chips have interrupt mitigation, which can help drastically when implemented correctly in the driver (fxp in BSD); 10/100 Realteks were garbage, but rumour has it they retooled for the gigabit lines and are vastly improved -- no personal word on that yet.

 

[Madwand1]

Realteks are the ne plus ultra of $6.99 gigabit add-on NICs. (Not counting eBay deals.)

 

[p0lar]

Originally posted by: Madwand1
Realteks are the ne plus ultra of $6.99 gigabit add-on NICs. (Not counting eBay deals.)

I knew I saw a blurb about these somewhere.. Here's an excerpt from a kerneltrap.org interview by Jeremy Andrews to Theo de Raadt (05.02.06), somewhat relevant from guys who take pride in vendor cooperation, proper driver construction, and hardware APIs. If you're crying from boredom, just look the other way.

Jeremy Andrews: Why do you suppose binary blobs have become so common?

Theo de Raadt: Well, I think they really have not become that common at all. All the time we see more vendors that are open. It is just that right now the most common devices are made by a very small set of vendors who are very strictly trying to lock down their market in this way. But it could go very backwards for them quickly, too. For instance, the (newish) Realtek Gigabit Ethernet chips are not too bad at all, and there's lots of documentation. So maybe the Taiwanese products are a little bit later to the market, but they are simpler and robust once they do make it to the market.

 

[HVAC]

Originally posted by: robmurphy
What OS is on the server and the desktop?

Did you make any changes in the registry, like the TCPwindow size?

Server=openSuSE 10.2 running a KDE desktop. I just VNC into it to make changes. I have a user named "share" and have set up Samba to allow guest connections and I force all connections to take on the "share" attributes without providing a password (don't do this unless you are thoroughly firewalled and trust everyone who may possibly use the "share" directory to not hose it.....in my case I am counting on my charming personality to protect me from my children's ability to cause rampaging virtual file destruction ).

Desktop=Win XPsp2 up to date.

I did dink around with registry settings that included a lot of TCP settings. I followed this guide: Here. (Just scroll down to your OS)

I ended up using a jumbo frame size of 4088. The switch I purchased (HP J9029A ProCurve 1800-8G) evidently cannot handle the 9000+ size frame.

 

[HVAC]

Originally posted by: Madwand1

I've hit > 800 Mb/s with a Realtek PCI NIC, and > 900 Mb/s with an Intel PCI NIC.

That's great. I might go back and try combinations of the realtek or the NV talking to an intel card and see whether or not it was the realtek or the NV or the combination that was causing my problem.

I went through extensive sessions trying to adjust the OSes to accelerate the realtek/NV combination.....to no avail.

 

[Madwand1]

Originally posted by: HVAC
I ended up using a jumbo frame size of 4088. The switch I purchased (HP J9029A ProCurve 1800-8G) evidently cannot handle the 9000+ size frame.

This is surprising. I hope you're wrong, because this should be a higher-end switch, not limited in this way. Everyone (but Realtek) knows the magic number is 9000 and should be the one supported.

http://www.hp.com/rnd/pdfs/dat...Switch_1800_Series.pdf

? Jumbo packet support: supports up to 9,216
byte frame size to improve performance of large
data transfers

The manual says "9000". Here are some other notes which also point to the magic number 9000:

http://www.hp.com/rnd/itmgrnews/frame_types.htm

So then use a frame bigger than 9000 bytes and the performance will increase in a linear fashion, right? Well, 9000 is really a glass ceiling for Ethernet. Why 9000? First, because Ethernet uses a 32-bit CRC that loses its effectiveness above about 12,000 bytes. And secondly, 9000 was large enough to carry an 8 KB application datagram (e.g., NFS) plus packet header overhead.

Is 9000 bytes enough? It's a lot better than 1500, but for pure performance reasons there is little reason to stop there. At 64 KB we reach the limit of an IPv4 datagram, while IPv6 allows for packets up to 4 GB in size. For Ethernet, however, the 32-bit CRC limit is hard to change, so don't expect to see Ethernet frame sizes above 9000 bytes anytime soon.

The real deal to understanding Jumbo Frames is the following:

* Understand they are not a standard, so testing prior to implementation is critical.
* Do not use anything above 9000 bytes.
* Jumbos must be supported end to end for any benefit: switch to switch, server to switch. Jumbos are not normally supported on client devices.
* Use TCP Offload Engines with Jumbo Frames on server-based NICs.

Perhaps the issue was with the Realtek NIC? They will cap around 7KB. You could use a direct straight-through cable connection between the gigabit NICs to factor out the switch. If it is a switch issue, I'd take it up with HP. Linksys has a couple of new products in this class (e.g. SLM2008); I'd consider changing to them if the issue was with HP and they couldn't resolve it.

It's not that I think there'd be much of a performance difference between 4K frames and 9K frames; it'd be irking IMO to be stuck with yet another jumbo frame size, esp. due to a product in this class when a $40 Netgear/etc. can handle it just fine.

 

[Madwand1]

Originally posted by: HVAC
That's great. I might go back and try combinations of the realtek or the NV talking to an intel card and see whether or not it was the realtek or the NV or the combination that was causing my problem.

I went through extensive sessions trying to adjust the OSes to accelerate the realtek/NV combination.....to no avail.

You might be stuck on some temporary driver / optimization issue and not be able to resolve this yourself until there's a updated driver released. My above results were with Vista 64. I saw some significant differences with drivers a few months apart. The Intel is a better NIC. In your place, I'd think of calling it a day until I had a want for another add-on NIC in another computer. Then it'd probably be another platform and another time, and better results might be easier to come by.

 

[p0lar]

Originally posted by: Madwand1
Perhaps the issue was with the Realtek NIC? They will cap around 7KB. You could use a direct straight-through cable connection between the gigabit NICs to factor out the switch. If it is a switch issue, I'd take it up with HP. Linksys has a couple of new products in this class (e.g. SLM2008); I'd consider changing to them if the issue was with HP and they couldn't resolve it.

Affirmative.

[p0lar@ipcbl-h1 ~]> man re
RE(4) OpenBSD Programmer's Manual RE(4)

NAME
re - RealTek 8139C+/8169/816xS/811xS/8101E 10/100/Gigabit Ethernet device

SYNOPSIS
re* at pci?
re* at cardbus?
rgephy* at mii?
rlphy* at mii?

DESCRIPTION
The re driver provides support for various NICs based on the RealTek
RTL8139C+, RTL8169, RTL816xS, RTL811xS and RTL8101E PCI and PCI Express
Ethernet controllers, including the following:

<snip>

The 8169, 8169S and 8110S also support Jumbo frames, which can be config-
ured via the interface MTU setting. The MTU is limited to 7422, since
the chip cannot transmit larger frames. Selecting an MTU larger than
1500 bytes with the ifconfig(8) utility configures the adapter to trans-
mit Jumbo frames.

 

[robmurphy]

Hi HVAC

Thanks for the info.

Rob Murphy

 

[HVAC]

Originally posted by: Madwand1

Originally posted by: HVAC
I ended up using a jumbo frame size of 4088. The switch I purchased (HP J9029A ProCurve 1800-8G) evidently cannot handle the 9000+ size frame.

http://www.hp.com/rnd/pdfs/dat...Switch_1800_Series.pdf

? Jumbo packet support: supports up to 9,216
byte frame size to improve performance of large
data transfers

The manual says "9000". Here are some other notes which also point to the magic number 9000:

Originally posted by: p0lar
The MTU is limited to 7422, since
the chip cannot transmit larger frames. Selecting an MTU larger than
1500 bytes with the ifconfig(8) utility configures the adapter to trans-
mit Jumbo frames.

Ooops....I got my products confused. Sorry, HP....I besmirched your golden name when it didn't need....er.....um.....besmirching. (While you are reading this....please continue making RPN calculators.......please!)

But, I am happy with 4088 frame size. I don't really see the need to go to around 9000 as I am already getting about 97% of max theoretical on memory only transfers.

I think if I want more, I am going to have to go berserk on the NIC<->CPU/MMU<->Disk path.