Jumbo Frames - Enable them!

[Carson Dyle]

I never realized that jumbo frames needed to be enabled on network adapters. I found that they were disabled on both my Windows 7 PC and on my Win2k8 server. So I just enabled them. I regularly transfer very large files between the two systems and there's definitely an improvement. Probably 10-20% faster. Damn.

You do need a jumbo frame capable network switch. So check that first.

 

[imagoon]

I never realized that jumbo frames needed to be enabled on network adapters. I found that they were disabled on both my Windows 7 PC and on my Win2k8 server. So I just enabled them. I regularly transfer very large files between the two systems and there's definitely an improvement. Probably 10-20% faster. Damn.

You do need a jumbo frame capable network switch. So check that first.

You are in the rare 1% where it makes a difference. Jumbo frames don't even make my SAN move faster any more.

 

[azazel1024]

I doubt in the less than 1% range. Unless it is not properly generating them or something in between is causing the adapters to drop back to standard sized frames, jumbo frames will/have to increase net yield. Less losses to packet headers.

That said, IIRC the difference between 1500MTU and 9000MTU (9k being the typical largest sized you'll find) is around 4% increase in net yield.

A pretty minor difference.

There is also a reduction in CPU overhead.

One of the areas I noticed the biggest difference is with my tablet when I enable 9k jumbo on the USB3 adapter. My CPU load drops from around 20% down to around 8%.

So, unless there was some specific issue going on before, the most you should see is around a 4% increase in performance (my Intel adapters go from 114 and some change MB/sec to 117.5MB/sec 1500MTU vs 9k jumbo). If you are seeing a huge 10-20% jump, either there is something wrong with the adapters in question causing performance problems at a 1500MTU size, or else you had a big CPU/DMA bottleneck that fewer, larger packets is helping out with.

 

[Carson Dyle]

Yeah, playing with it some more, 10-20% was generous. Seems like (although I could be imagining it) it speeds up single large file transfers, say 1-5GB, more so than very large transfers of say 25GB or more. Almost like the transfers 'burst' faster now.

 

[RadiclDreamer]

You are in the rare 1% where it makes a difference. Jumbo frames don't even make my SAN move faster any more.

Well, it DOES make a difference, but it may not be noticable, but it IS a difference.

For those unaware, jumbo frames allow each device to send larger payloads with each chunk of info. So instead of tiny header and small payload you get tiny header and large payload, basically saving data in the way of headers. That being said, there is a tiny difference.

dmac(6)+smac(6)+etype(2)+payload(1500)+crc(4) = 1518 bytes
dmac(6)+smac(6)+etype(2)+payload(9000)+crc(4) = 9018 bytes

 

[Emulex]

If you have a 10gbe nic you might find the offload acceleration to equal the cpu cost in handling the smaller packets. large packets can stuff up latency as well.

Cheapo realtek nic's wont do great offload. server based intel nic's are very easy with packet acceleration and moderate to all cores for tx/rx ring buffers to maximize flow with minimal cpu overhead!

 

[imagoon]

Well, it DOES make a difference, but it may not be noticable, but it IS a difference.

For those unaware, jumbo frames allow each device to send larger payloads with each chunk of info. So instead of tiny header and small payload you get tiny header and large payload, basically saving data in the way of headers. That being said, there is a tiny difference.

dmac(6)+smac(6)+etype(2)+payload(1500)+crc(4) = 1518 bytes
dmac(6)+smac(6)+etype(2)+payload(9000)+crc(4) = 9018 bytes

Maybe if you are hitting the theoretical limits. The main issue that I see is that not all frames will automatically be 9000 byte pay loads. For example a TCP ack will not be a 9000byte packet. Downloading off the web (assuming doesn't straight up fail) will never exceed the MTU of the net etc. Add in that TCPIP Offload basically removes the CPU overhead. So yeah you are right there is a theoretical improvement but the amount of work to get it working in a lot of environments will out weight the majority of the benefits in my opinion.

If you go back to cpu run cards like what Carson seems to be using, then use the 10-20% will help.

 

[Mushkins]

Last I checked, just enabling Jumbo Frames all willy nilly typically caused more problems than it theoretically solves. Things that support Jumbo Frames might see slight improvements if everything is configured properly, but things that don't support them tend to have issues.

 

[RadiclDreamer]

Last I checked, just enabling Jumbo Frames all willy nilly typically caused more problems than it theoretically solves. Things that support Jumbo Frames might see slight improvements if everything is configured properly, but things that don't support them tend to have issues.

I've personally never had an issue with them, but then again, I've not seen major performance gains either.

 

[VirtualLarry]

Of course, if you're a home user, connecting your PCs to wireless router, and then connecting out onto the internet, if you enable Jumbo Frames on the PCs wired to the router, they will be able to communicate amongst one another using JF (assuming that the switch supports it), but it will jack up their internet communications, unless the router explicitly supports JF. Most don't.

In short, lots of pain for little gain, in most cases.

Now, if you set up a parallel storage-only LAN, using JF-capable gigabit switches, and secondary server-grade NICs in each PC, then it might make sense, if your storage LAN needs are performance-sensitive.


My NAS maxes my AC1200 wifi, which is all I care about, really.

 

[Carson Dyle]

it will jack up their internet communications

What exactly does that mean?

 

[Tsavo]

I've personally never had an issue with them, but then again, I've not seen major performance gains either.

I've had up to 9% on my Intel NICs testing xfer rates with a single 4.7GB file...over and over again, dozens of times. Real world would be less than half that. Might as well be invisible xfering thousands of small files.

But, gains are good gains when the performance floor is so slow to begin with.

Speaking of Intel NICs, 6+years using them and not a single networking farkup.

 

[Gryz]

Extra bandwidth to be gained is small. About 4%.
See here the exact numbers:
https://en.wikipedia.org/wiki/Jumbo_frame#Bandwidth_efficiency

The gain is mostly in lower CPU usage.
Of course if you have an ethernet card that can do some of the processing, the CPU-gains will be less.

There is not only a gain to be had on the end-stations (PCs, servers). Also the devices in between will need to do less work. The performance of routers (and switches too?) are typically denoted in packet-per-second (pps). With 9k frames you have 6x fewer packets to forward than with 1.5k frames. Some cheap routers (or switches) might not be able to forward 1500B frames at 1Gbps (which would require 82345 pps). But they can forward 9k frames at 1Gbps (which would require 13861 pps).

Assuming all switches can forward any frame-size at wire-speed is maybe wrong ? I couldn't find any recent numbers about cheap switches. (I did find numbers on more expensive/professional switches, and of course those can forward at line-speed, even with 64 byte frames).

 

[Mushkins]

What exactly does that mean?

Here's a pretty good read on the pros and cons of Jumbo Frames:

http://blog.codinghorror.com/the-promise-and-peril-of-jumbo-frames/

Simply put, if any device end-to-end does not support Jumbo Frames, the whole connection gets jacked up. Everything end to end has to have Jumbo Frames support, have it enabled, *and* agree on the MTU size, otherwise the packet gets dropped or fragmented, which means extra retransmissions giving you *less* total throughput.

Jumbo Packets can be a benefit in a strictly controlled internal environment, but as far as using them out in the wide world of the internet, the odds of every device down the line playing nice are next to zero.

As a side note, Jumbo Packets are not friendly for a lot of applications. Torrents and VoIP come to mind, where they are focused on sending tons of small bits of data very quickly, and forcing them to use Jumbo Packets introduces additional latency and uses more bandwidth as the applications need to fill the unused space in each packet with filler data before sending.

 

[imagoon]

As a side note, Jumbo Packets are not friendly for a lot of applications. Torrents and VoIP come to mind, where they are focused on sending tons of small bits of data very quickly, and forcing them to use Jumbo Packets introduces additional latency and uses more bandwidth as the applications need to fill the unused space in each packet with filler data before sending.

Just because the MTU is 9000bytes doesn't mean the packet will be 9000bytes [plus headers blah blah blah.]

This is why the "4%" number is a theoretical. That 4% assumes that all the frames are 1500bytes (+headers) before and that all the resulting frames are 9000bytes(+headers) after and utilization is 100%. This simply doesn't happen on a LAN let alone a WAN.

An example would be interactive telnet. Each key press is not going to be built into and stored in a 9000byte frame. It will be build (with some delay due to other protocols) and shipped, possibly as small as a ~64byte frame. Shipping single byte telnet frames in Ethernet packets over a link with an MTU of 9000 will not see any improvement. VoIP frames are similar.

Generally this is the main reason why you only see jumbo frames mentioned in storage. It is one of the few places where the percent of frames is high enough to actually utilize the 9000byte frames.

Otherwise... VoIP won't use it, anything off the Web won't, Netflix won't etc because of the net the incoming frame is rarely larger than 1492 bytes.

If you get fancy you can stack the incoming packets in to one frame, but that adds latency which eliminates that value in a lot of cases.

Basically it is a lot of work from what is not a lot of improvement over all.

 

[Cstefan]

At home I would not be concerned about this. It will help bigger file transfers if your switch can do it but the entire stack needs to support 9000 end to end otherwise it will be worse than 1500.

Even in enterprise networks the overall improvement is 3-5% with the biggest improvement in streaming files, backups and the like. All this is best for a SAN which has the greatest room to benefit.

 

[jsbush]

You are in the rare 1% where it makes a difference. Jumbo frames don't even make my SAN move faster any more.

On your SAN (if truly an iSCSI SAN) I can guarantee you the manufacture recommends enabling jumbo frames. They do make a big difference, when needed, and when configured properly. They can also hinder performance where not needed and when improperly configured.

 

[imagoon]

On your SAN (if truly an iSCSI SAN) I can guarantee you the manufacture recommends enabling jumbo frames. They do make a big difference, when needed, and when configured properly. They can also hinder performance where not needed and when improperly configured.

They only recommend it in exactly one case: Using 8 10GBe connections at once. We don't do that.

And I don't agree. They don't make a big difference and are rarely needed. Heck in our environment it was drop in performance because the TCP/IP and iSCSI offload engines didn't handle 9k as well. Average at ~60% read was down about -.5% Not a lot but not a gain.

 

[jsbush]

They only recommend it in exactly one case: Using 8 10GBe connections at once. We don't do that.

And I don't agree. They don't make a big difference and are rarely needed. Heck in our environment it was drop in performance because the TCP/IP and iSCSI offload engines didn't handle 9k as well. Average at ~60% read was down about -.5% Not a lot but not a gain.

What model SAN do you have?

 

[imagoon]

What model SAN do you have?

Variety of EMC CX-4 and VNX2 units all block. All jumbo frame recommendations for less than 10GBe is for the file / data mover modules. For block they recommend ALUA 4 Round Robin.

I should point out that the SAN switches are never peaked (CPU or port) at this point and the throughput at VNX peeks are 1 gigabyte/sec.

 

[jsbush]

Variety of EMC CX-4 and VNX2 units all block. All jumbo frame recommendations for less than 10GBe is for the file / data mover modules. For block they recommend ALUA 4 Round Robin.

I should point out that the SAN switches are never peaked (CPU or port) at this point and the throughput at VNX peeks are 1 gigabyte/sec.

Which part of my statement don't you agree on? I said they make a big difference when needed. And when not needed they can do the reverse. EMC is of no exception to this; with 1gbps or 10GbE.

If you are peaking at 1GB/sec then you are definitely running a 10GbE or a 10x1gbps LAG, or 10x1gbps MPIO iSCSI fabric.

 

[imagoon]

Which part of my statement don't you agree on? I said they make a big difference when needed. And when not needed they can do the reverse. EMC is of no exception to this; with 1gbps or 10GbE.

If you are peaking at 1GB/sec then you are definitely running a 10GbE or a 10x1gbps LAG, or 10x1gbps MPIO iSCSI fabric.

Yeah pretty sure considering I built it and all.

The part about "big difference" is what I am arguing. Theoretical increase of 4% vs real life result of -% to 1% isn't "big."

The pros are tiny, the cons much bigger. If any one is seeing "30%" increases, they have other issues like switch queues, fabric CPU, network (missing flow control, port problems) etc.

 

[azazel1024]

I'd clarify, the cons IF they occur are much bigger than the pros.

I've only been able to get jumbo frames to misbehave badly once. I've seen or heard of a few other setups where they've caused real problems (I don't mean a fractional slow down because of "insert nebulous reason").

The pros are generally a few percent better network throughput, generally somewhat lower CPU utilization as well as somewhat lower switching overhead.

As for "theoretical 4%", at least for things like CIFS/SMB/iSCSI where most of what you are transfering over the network is going to be "a lot of data" that can be broken in to very large packets, you'll get roughly the 4%. ACK packets are small, but they also are <10% of the packet traffic.

Sure, things like telnet and so forth aren't going to be able to take advantage of it. I can't think of too many networks where you are saturating things because of telnet connections. Generally it is network file transfers, HPC or video streaming that is resulting in networking bottlenecks. One of those three is a generally (but not always) an excellent case of jumbo frames and 2 of those 3 are generally going to ignore jumbo frames (and 1 of the 3 would work poorly with jumbos, as HPC latency is often more of a performance killer than bandwidth, and jumbos will increase latency, at least a tiny, tiny amount).

 

[velis]

My experience with them has been strictly negative. Granted, I'm no network expert, but I did try a home grade router as well as a business class switch and in both cases my throughput went down considerably.
Though I did have a very mixed set of NICs to work with: Intel, Atheros, Broadcom.

My figures were 110 MB / s bidirectional for 1500 MTU and 20 - 35 (one way) / 75 (the other way) MB / s for 9K MTU.
I'm definitely not enabling them again unless I match my equipment first. But with current 1500 MTU performance I really don't feel the need.

 

[Tsavo]

My experience with them has been strictly negative. Granted, I'm no network expert, but I did try a home grade router as well as a business class switch and in both cases my throughput went down considerably.
Though I did have a very mixed set of NICs to work with: Intel, Atheros, Broadcom.

My figures were 110 MB / s bidirectional for 1500 MTU and 20 - 35 (one way) / 75 (the other way) MB / s for 9K MTU.
I'm definitely not enabling them again unless I match my equipment first. But with current 1500 MTU performance I really don't feel the need.

At 110 MB/s, you won't see any appreciable gain going to 120 MB/s if you got all perfect NIC's/switches/routers.

I've noticed that a lot of cheap NIC's that support Jumbo packets have weird packet sizes, and has caused me headaches in the past in terms of reliable throughput. I've purged the heathens long ago, so I get perfect throughput all day with my Intel stuff.

The Broadcom unit in my 2014 Mac Mini is a real champ, though; fits right in with my Intel stuff and doesn't skip a packet.