Slow file transfer over Gigabit

[integramodder]

I have two 64 bit computers on gigabit; one running Windows 7 (with 7200 rpm HD) and one running Server 2008 (Raid 5 Array). They are both running on about 20 feet of Cat 6 cable that I made myself. Both machines are showing 1 Gbps connections on each network.

I was initially getting about 9 Mbps and leveled out to about 6 Mbps. I did some googling and have updated to latest drivers, enabled Jumbo Frames on both machines and rebooted. This did not solve the problem. When it first started it started at 15 Mbps and then went down to 6 Mbps. Then I found some other suggestions and ran the instructions listed here to disable auto tuning and rss. These actually decreased the performance, initially it was about 13 Mbps and then slowed down to 3 Mbps.

Does anyone have any suggestions?

 

[Tsavo]

Check the hardware first.

1. Try a different cable.

2. Saying you have a 7200 RPM drive doesn't mean anything. I've got some older ones that can't pinch out 45 MB sustained. You are getting much less than that, but it can't hurt to test. http://crystalmark.info/software/CrystalDiskMark/index-e.html

Software.

Are you running AV, and if so, what? Does your transfer speed change with the AV disabled?

 

[llee]

What are you transferring? You'll experience drastically slower speeds if you're moving a large number of smaller files compared to a small number of large files. That's why server and IT administrators usually compress their deployments.

 

[pcgeek11]

Try Non-Homemade Cables they do work better. I have Windows 7 and Windows Home Server on a Gbit Network through a Netgear switch using bought CAT5E Cables and I average between 40 - 75 MB transfer rates depending on the sizes of the files. You should be seeing the same or close.

 

[P4man]

Before trying anything else make sure you test the speed by copying very large files (100s of MBs). When copying lots of tiny files performance plummets and that has nothing to do with the network. Copying locally from one drive to another under windows I can get performance as low as a few MB/s or lower if its a ton of tiny files.

 

[Emulex]

try using iscsi or nfs - it will bring out the true light.

nfs udp i mean.

 

[Rifter]

They are both running on about 20 feet of Cat 6 cable that I made myself.

This is the problem. It is very hard to make proper cables yourself. Try store bought cables.

 

[spidey07]

This is the problem. It is very hard to make proper cables yourself. Try store bought cables.

Yep. Almost guarantee it's the cable with such poor performance. It's the number one cause of problems.

 

[integramodder]

Check the hardware first.

1. Try a different cable.

2. Saying you have a 7200 RPM drive doesn't mean anything. I've got some older ones that can't pinch out 45 MB sustained. You are getting much less than that, but it can't hurt to test. http://crystalmark.info/software/CrystalDiskMark/index-e.html

Software.

Are you running AV, and if so, what? Does your transfer speed change with the AV disabled?

1. I dont have different cables but I can make them. If both components are reading gigabit status, is this still an issue?

2.

Client:

-----------------------------------------------------------------------
CrystalDiskMark 3.0 x64 (C) 2007-2010 hiyohiyo
                           Crystal Dew World : [url]http://crystalmark.info/[/url]
-----------------------------------------------------------------------
* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]

           Sequential Read :    49.559 MB/s
          Sequential Write :    46.770 MB/s
         Random Read 512KB :    28.747 MB/s
        Random Write 512KB :    26.458 MB/s
    Random Read 4KB (QD=1) :     0.471 MB/s [   115.1 IOPS]
   Random Write 4KB (QD=1) :     0.877 MB/s [   214.1 IOPS]
   Random Read 4KB (QD=32) :     0.967 MB/s [   236.1 IOPS]
  Random Write 4KB (QD=32) :     0.831 MB/s [   202.8 IOPS]

  Test : 1000 MB [D: 68.2% (67.5/98.9 GB)] (x5)
  Date : 2010/08/27 9:13:38
    OS : Windows 7 Ultimate Edition [6.1 Build 7600] (x64)

Server:

-----------------------------------------------------------------------
CrystalDiskMark 3.0 x64 (C) 2007-2010 hiyohiyo
                           Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]

           Sequential Read :   245.971 MB/s
          Sequential Write :    16.828 MB/s
         Random Read 512KB :    37.034 MB/s
        Random Write 512KB :     8.337 MB/s
    Random Read 4KB (QD=1) :     0.528 MB/s [   128.9 IOPS]
   Random Write 4KB (QD=1) :     0.286 MB/s [    69.9 IOPS]
   Random Read 4KB (QD=32) :     4.164 MB/s [  1016.6 IOPS]
  Random Write 4KB (QD=32) :     1.058 MB/s [   258.2 IOPS]

  Test : 1000 MB [D: 65.7% (2446.2/3725.6 GB)] (x5)
  Date : 2010/08/27 9:42:40
    OS : Windows Server 2008 R2 Enterprise Edition (Full installation) [6.1 Build 7600] (x64)

Both systems have built in windows firewall enabled and Symantec Endpoint Protection v11. I will try running test speed tonight without these enabled.

I also ran JPerf last night:

 

[integramodder]

What are you transferring? You'll experience drastically slower speeds if you're moving a large number of smaller files compared to a small number of large files. That's why server and IT administrators usually compress their deployments.

A 35 GB vitualbox image

 

[integramodder]

try using iscsi or nfs - it will bring out the true light.

nfs udp i mean.

I dont have any idea what you are referring to. Could you explain a bit more?

 

[imagoon]

1. I dont have different cables but I can make them

Don't. Spent the $3 and buy a machine made one.

 

[integramodder]

Yep. Almost guarantee it's the cable with such poor performance. It's the number one cause of problems.

Why are cables a common problem? It does not appear to be anything that requires a lot of complex work to get right - 8 wires on each end. I will order cables today and see how it impacts performance.

Is there a network tool you suggest a I use to test speeds before and after other than a windows file transfer?

 

[MerlinRML]

I'm not going to disagree about the cables because I know better.

An additional problem that I see is that your server disk system can only write 16MB/sec. I'm guessing you're doing a Windows software RAID5 with those low numbers. This disk performance would be very closely inline with the performance you're reporting.

Are you writing the data to the server? If not, is there something else that is writing to the server during that time that's clobbering your read speeds?

 

[DisgruntledVirus]

2 things.

First, cables are more than just 8 wires on each end. If they aren't properly terminated you can have an decrease in the quality of the signal. Expensive cable testers look for this. Machine made cables are much less likely to have a flaw that degrades the quality of signal. Besides, it's fairly cheap to buy one.

Second, I'm having a similar issue. I never had the problem before, and the only change was the HD's in the server. They are the new 4 KB sector drives. You say you are running Server 2008, which IIRC does not support that. I haven't had time to mess with mine yet to see if that fixes it, but it's the only thing that has changed in the systems and I went from 60+ mbps down to <5 mbps.

 

[imagoon]

Why are cables a common problem? It does not appear to be anything that requires a lot of complex work to get right - 8 wires on each end. I will order cables today and see how it impacts performance.

Is there a network tool you suggest a I use to test speeds before and after other than a windows file transfer?

95% of the time, the wrong cable is used #1. If it is a solid core cable, it should not have ends crimped on, it is meant to be terminated to a keystone.

The crimp ends are different for a) Solid core b) stranded. They are further different for a) cat5(e), b) Cat6, c) Cat 6a etc. The wire gauges different.

Many people don't realize that there is a maximum 'straight' length and that the wires need to be installed in a specific order (IE 568A or B)

So 95% of the time we find that users with weird layer 1 issues (errors, runts, CRC's etc) that generally lead to slow or crappy performance even though the link lights are up are easily fixed with a machine made cable, and that is why the stickies say to use them and why several of us refuse to help until it is cabled correctly and to spec.

So nothing personal.

 

[imagoon]

I'm not going to disagree about the cables because I know better.

An additional problem that I see is that your server disk system can only write 16MB/sec. I'm guessing you're doing a Windows software RAID5 with those low numbers. This disk performance would be very closely inline with the performance you're reporting.

Are you writing the data to the server? If not, is there something else that is writing to the server during that time that's clobbering your read speeds?

I would consider 16MB/sec slow for RAID 5 unless it is 10 year old hardware. I can push about 190MB/sec to mine.

 

[imagoon]

2 things.

First, cables are more than just 8 wires on each end. If they aren't properly terminated you can have an decrease in the quality of the signal. Expensive cable testers look for this. Machine made cables are much less likely to have a flaw that degrades the quality of signal. Besides, it's fairly cheap to buy one.

Second, I'm having a similar issue. I never had the problem before, and the only change was the HD's in the server. They are the new 4 KB sector drives. You say you are running Server 2008, which IIRC does not support that. I haven't had time to mess with mine yet to see if that fixes it, but it's the only thing that has changed in the systems and I went from 60+ mbps down to <5 mbps.

Good thought. You can 'force' support by partitioning it correctly. Use an offset that sticks the partition on the 4096byte barriers. Diskpart can do this for you. Once you do that, and stat using ntfs 4K or larger sectors, it will start to fly.

 

[owensdj]

integramodder, are your machines connected to a gigabit switch or are they connected to each other directly using a single cable?

 

[Binky]

Cables
Cables
Cables

A bad cable will still connect at GB speeds (the light goes on), but it will work just as poorly as described above.

Did you follow the 586A or 586B wiring standard?

 

[integramodder]

I'm not going to disagree about the cables because I know better.

An additional problem that I see is that your server disk system can only write 16MB/sec. I'm guessing you're doing a Windows software RAID5 with those low numbers. This disk performance would be very closely inline with the performance you're reporting.

Are you writing the data to the server? If not, is there something else that is writing to the server during that time that's clobbering your read speeds?

This is the other problem I am investigating, I have to check the cache setting in the bios on the card. Its a 5 1TB drives running on a RocketRAID 2320. It hosts some web traffic and does a few other small tasks, but is primarily a mass storage server.

The 35 GB file is being copied on to the server, but even then it should theoretically stay at 16 Mbps...

 

[integramodder]

95% of the time, the wrong cable is used #1. If it is a solid core cable, it should not have ends crimped on, it is meant to be terminated to a keystone.

The crimp ends are different for a) Solid core b) stranded. They are further different for a) cat5(e), b) Cat6, c) Cat 6a etc. The wire gauges different.

Many people don't realize that there is a maximum 'straight' length and that the wires need to be installed in a specific order (IE 568A or B)

So 95% of the time we find that users with weird layer 1 issues (errors, runts, CRC's etc) that generally lead to slow or crappy performance even though the link lights are up are easily fixed with a machine made cable, and that is why the stickies say to use them and why several of us refuse to help until it is cabled correctly and to spec.

So nothing personal.

Valid points. I have ordered cables and they will be here on Tuesday.

 

[P4man]

Im a bit skeptical about the cable being the issue, in my experience they work or dont work (reliably) if the network cards dont renegotiate a lower speed or lose connection randomly, then I think its something else. Of course it doesnt hurt ruling this out especially as network cables are cheap.

YOuve tested harddrive performance, but as others implied, you may want to test filesystem performance. Just copy a huge file or set of files to a different folder and see if you get normal performance. Try it on both machines.

Also have you tried updating your NIC drivers? Do you have another NIC you can test with? Or a third machine (laptop?) so you can find out if the issue is at the server or client? I have seen similar issues with some onboard gigabit controllers.

Lat suggestion: what inbetween these machines? A router or switch or is it a cross cable? If its a router, make sure there isnt something wrong with that, like a crappy firewall.

 

[imagoon]

Im a bit skeptical about the cable being the issue, in my experience they work or dont work (reliably) if the network cards dont renegotiate a lower speed or lose connection randomly, then I think its something else. Of course it doesnt hurt ruling this out especially as network cables are cheap.

Slow performance is a symptom of "Doesn't work (reliably)."

Layer 1 is the #1 cause of these issues. Always check it with a tester / use a known good before you start messing with drivers and settings.

 

[MerlinRML]

This is the other problem I am investigating, I have to check the cache setting in the bios on the card. Its a 5 1TB drives running on a RocketRAID 2320. It hosts some web traffic and does a few other small tasks, but is primarily a mass storage server.

The 35 GB file is being copied on to the server, but even then it should theoretically stay at 16 Mbps...

Once you've replaced the network cables, you're not going to see any significant improvement because of the poor write speeds on your disk array. Not to say that there wasn't a network cable problem there, but it is not the only problem.

I'm not very familiar with highpoint products, but you should check out if there is a management software for the RAID controller. Check to see if there are any problems reported from any of the drives, check write cache settings, etc. You might also check windows event viewer and see if it is throwing any errors accessing the RAID disks.