Network connectivity issue

[brockj]

Ok, this has been an ongoing issue. Between myself and the Telco many, many hours have been spent trying to figure this out. I will do my best to describe the scenario, and if anything needs further explanation please let me know. I am at a loss on this issue...

The issue is workstations cannot fully connect to our exchange server or AS400. By fully, I mean the workstation can ping our exchange server, both via name and IP; however, if I try to browse shares, or connect via outlook it times out. For AS400 sessions it just won't connect.

The location consists of 10 workstations and a server. All machines are XP, the servers are all 2003 standard. The connection to the corporate office is a Point-to-Point T1, with a wireless connection through a secondary provider as a backup. Both ends of the PTP are 1841 Cisco routers with internal CSU/DSU.

When the office is running on the backup line, everything works fine, just slow. When we switch them over to the PTP T1, random people will lose connectivity to either the exchange server, the AS400, or both. And by random I mean, last week one person lost connectivity to both, this week he is working fine either way.

As I said, I have been working with the telco for several months as at times this issue will almost completely go away...for a couple weeks. So far I have replaced the WIC cards on both ends of the T1, had wiring changed, dealt with issues at the Telco and QWest (they own the main line). The wierd things is at this point, I am not even getting errors on the interface anymore. Before I could at least see errors on the interface and yell at the Telco, but everything has been running clean since the last time the Telco was out!

I am completely out of ideas, and any help from the experts here would be GREATLY appreciated! Thanks in advance!

 

[sieistganzfett]

hmm. i'm not familar with "AS400" but am familar with Point to Point T1 lines, and i would actually get a terminal server up and running for what ever software you need at the site that must go over the T1 to access the exchange server and AS400, more than likely, this would fix the time outs and slowness. for exchange, i would set up outlook web access to bypass that issue other issue. even though its T1, its still a lot slower than even a 10/100 LAN. terminal server with all the needed apps on it, located on the side of the AS400 + exchange server should take care of the problems.

 

[brockj]

Unfortunately it is not a timeout issue. With the T1 connected, my ping rates are excellent, 7ms consistent. With the backup line I am anywhere from 10ms-100ms, and on the backup line everything works fine.

 

[nweaver]

so if a machine stops working, is it a long term issue, or just a couple of minutes?

I would set up a network tap on the LAN side of the routers and start taking dumps to see if you can find the issue.

 

[spidey07]

1) Make sure the T1 is running without ANY errors on both sides of the link. Get your telco to run some intrusive testing (have them loop your CSU and run tests on both sides). It sounds like you have a bad T1.
2) Check speed/duplex settings on the router and switch port. Router and switch need to be auto or hard set on both sides. NEVER force one side and leave the other at auto - you will have a duplex mismatch and have terrible performance.
3) Pings are not a valuable tool for this kind of troubleshooting, if you want to use them then make the length 1400 bytes "ping <ip> -l 1400"
4) You can never rule out cabling with something like this so make sure you aren't using home-made patch cables.

 

[sieistganzfett]

no idea really of what to do next for you, but what i mentioned is how i would have set it right off the bat though to prevent this potential problem from happening due to the large amount of data being sent (i assume the data is a lot).. you already blamed the t1 isp each time you found something wrong, but now you ran out of dirt on them. you can try a different ISP for your T1, if your absolutely sure its still them (probably is in some form), especially if the remote site really does and would run fine for days or even a few weeks on end without any of these problems occuring when using the slow backup line as your primary line. what happens when you do a tracert to the destinations that fail over the t1? is it going the correct route or get stopped somewhere? how does it start working again? do all of the workstations get the problem at the same exact time, or is it really only some of them at random like you said? when you reach half or near full capacity of the t1 line, is that when things stop working on the t1 or is this when you switch from the backup line to the primary which is the t1? are you really getting a full t1? have you monitored the t1 connection over the course of several days that contained the problem, like bandwidth utilization, latency, etc. and notice anything interesting or of value from that data? who has the access to the cisco equipment, my guess is the telco, and your stuck unable to check if the config there is perfect.

 

[spidey07]

Masking a network problem by using less bandwidth is not generally a good idea. 10 people is easily supported with a T1 for all but the most intensive applications.

AS400 is transactional based, it uses very little bandwidth.
Compounding good design principles by using a terminal server is never a good idea.
Making it even worse by making it web based smacks good network design in the face.

The OP has a very simple problem and will not be solved by a terminal server. Rather than resolving the actual problem, you just moved it somewhere else.

 

[brockj]

Originally posted by: nweaver
so if a machine stops working, is it a long term issue, or just a couple of minutes?

We have let it run as long as a couple hours, unfortunately we are not able to leave it long term, and during that time normal troubleshooting was done on the PC's (restarts, checking things out, ect.)

Originally posted by: spidey07
1) Make sure the T1 is running without ANY errors on both sides of the link. Get your telco to run some intrusive testing (have them loop your CSU and run tests on both sides). It sounds like you have a bad T1.

We have done the intrusive testing and the Telco reports no errors. I have cleared the counters, and recently have not noticed any errors, on either side of the link.

2) Check speed/duplex settings on the router and switch port. Router and switch need to be auto or hard set on both sides. NEVER force one side and leave the other at auto - you will have a duplex mismatch and have terrible performance.

I will double check all of these, but everything should be set to auto.

3) Pings are not a valuable tool for this kind of troubleshooting, if you want to use them then make the length 1400 bytes "ping <ip> -l 1400"

I can give this a try, I was using ping not only to establish connection, but to rule out a DNS issue as well.

4) You can never rule out cabling with something like this so make sure you aren't using home-made patch cables.

The only cables that are different between the backup and the T1 have been replaced 3 or 4 times, all with purchased/pre-packaged cables.

what happens when you do a tracert to the destinations that fail over the t1?

When I run a tracert everything looks good. I see the three hops I would expect and times similar to my pings.

As I have said I have done just about everything I can think of. If there is anymore information that might help clarify this please let me know and I will get it right away.