Packet Sniffer help

[Dag]

Ok something has been soaking random WAN links in our company for about a week now. What I am looking for is someway to track this traffic to either a IP or a Hardware address. I have tried several free sniffer products, and all I can get is either local subnet monitoring, or they will tell me "Hey, this link has very high latency." No kidding? And here I thought 90% failure and 1500ms pings were the norm.

Anyway, is there a product out there that I can install on my desktop that will let me monitor traffic from say, 10.1.1.1 to 10.10.1.1? The routers are a combination of 3com and Ciscos.

 

[Shadow07]

Are you willing to spend money for a good Packet Sniffer? One of the best one's I have seen is WildPackets Etherpeek. You can capture packets and raw data for analisys. Also, checkout Net3 Group's products. They have a product that will analyze packet captures and give you statistical data that will help you in analyzing what the problem is.

 

[spidey07]

What you really need is a WAN sniffer that actuall monitors the WAN port. If you operate any decent sized WAN a sniffer is absolutely required.

Otherwise you can turn on IP accounting for a specific interface with cisco via the "ip accounting" command on the interface you wish to gather stats. view results with the command "show ip accounting" clear counters with "clear ip accounting". the results will show you IP addresses and how much data they are sending/receiving.

That should help. What models of cisco routers are you using? you could use netflow switching to gather stats depending on model. Also, you can call cisco and they can give you more info.

I doubt you are really dropping 90% of your packets, try pinging with a higher timeout like "ping x.x.x.x -w10000 -t"
10000 is milliseconds and t is continous. But still 1500ms latency is beyond ridiculous and borders on NONFUNCTIONAL. that is really bad.

 

[Shadow07]

Spidey, the sniffer I recommended not only captures Ethernet packets, but you can also specify the node you want to capture data from.

But, if you do want to monitor a Cisco router, then useing the IP ACCOUNTING command will allow you to get some of the stats you need to figure out the problem. However, it will not show you who is broadcasting or flooding the LAN port. (and yes, I know that a router does not forward broadcasts, but you will want to see if a node is, thus creating a DoS attack that is not intentional, or you would think not).

 

[Dag]

Right, the port is not dropping, and the analyzer i was trying (wildpacks netdopler) was not showing loss, but showing HIGH latency. It will suddenly just stop though, and back into the 50-150 ms range.

I am new to useing these though and am looking for something like you said, a WAN analyzer. I may be willing to buy it, but I would really like to try it first if possible.

We have 35 some odd Frame links, all useing either cisco 2600's or 3com netbuilder 2's.

I will try the "ip accounting" though. That should tell me what I need to know at least for now.

 

[spidey07]

Yeah, but if you are not on the segment where the traffic is flowing you won't be able to see any part of the conversation that is actually causing the problem.

DAG, what is the basic layout of your network? Is there only one primary LAN segment with several other LAN segments connected with a router over some WAN links? How many routers are there and what kind of WAN protocol are you using? Maybe we can figure out where best to start sniffing.

 

[spidey07]

with 35 frame relay links you really need to purchase a WAN sniffer.

Cost around 15K but pays for itself almost immediately. Network associates makes what everybody calls a sniffer.

sniffer WAN

Let me know if you need more info and I'll post the part numbers you need. You can't be expected to operate a frame relay network of 35 links without one.

 

[Shadow07]

With any of the CA Network management tools, you cannot go wrong. Another program from them that you might want to check out is NetworkIT. It is the smaller version of TNG Unicenter, and has some unique features that would impress anyone. The nice feature that it has is the Nuegents, which are intelligent plug-ins that will monitor proactively after about 1 tp 2 weeks of network analisys.

 

[Dag]

Ah, part of the problem is EVERYTHING is in transition.

We want to be all Cisco, but have alot of 3com's around still (probably 30 3coms, 5 cisco). All new setups and replacements are cisco.

We have one major hub (10.1.1.1), that has 5 more smaller hubs attached. The smaller hubs have anywhere from 1 to 7 satillite sites on them. These hubs are 512, all single connections are 128, all frame. There are also 10 or 12 non hub sites that come directly back to 10.1.1.1. We are going to elliminate the smaller hubs soon and bring them all back to 10.1.1.1, except the one that goes to Canada.

We run IP and IPX, using RIPIP, and NLSP respectivly. A few sites are pure IP, but not many. Soon they will all be pure IP (6 months maybe.)

There is also only one Subnet per site, they aren't very big, 100 nodes is the biggest I think.

Heh, what did I leave out?

 

[spidey07]

where is the congestion occuring? on random links between each other or simply the MAIN HUB frame relay access ports? How long does it last?

One thing to look out for is IP-RIP. I'm curious if 3com and cisco rip are having some kind of split horizon issue?

 

[Dag]

So far the congestion has showed up in 3 places. One was between a hub site, and a satilite site. 2 was the SAME hub site and a different site. 3 was from the MAIN site (10.1.1.1) and a sigle satilite site. That last satilite site is a cisco. All the others are 3coms. It was the first Cisco we put in and has been there for 8 months now I think.

 

[Xanathar]

Dont forget to look for is the behavior normal? What is the Wan link being used for? Are you monitoring the useage on the links regularly, or just occasionally ping and see the delay as huge?

Transfers between machines will always try and use as much bandwith as they can, So if a remote site is copying files accross the WAN, the WAN will be cogested during that period. The only way to prevent that is to instate rate limiting, which may not be beneficial if X end user is directly opening a document from word, and their machine is at a standstill until the file is done opening.

 

[Dag]

When something like this happens I throw a simple ping /t with a 500 buffer lines in the CMD window. Its not great, but I can watch the trends. When the latency happens, 90% of these time out, the others are 1400-1500 ms. That goes on for a long time, then just stops in its tracks.

The sites each have their own local server. Email and internet is mainly what goes accross these links. Unless there is a user that traveled to a different site. Like I said the sites are fairly small (20-30 users), so it isn't hard to check around to see if anyone is doing a copy, or roaming that day. Whatever it is, it is probably either something they did and didn't know it, or coming from some automated process.

The are all bevaiving now that I know how to track them though =-P.

See!
Reply from 10.254.1.1: bytes=32 time=32ms TTL=254
Reply from 10.254.1.1: bytes=32 time=32ms TTL=254
Reply from 10.254.1.1: bytes=32 time=16ms TTL=254
Reply from 10.254.1.1: bytes=32 time=47ms TTL=254
Reply from 10.254.1.1: bytes=32 time=47ms TTL=254
Reply from 10.254.1.1: bytes=32 time=32ms TTL=254
Reply from 10.254.1.1: bytes=32 time=16ms TTL=254
Reply from 10.254.1.1: bytes=32 time=32ms TTL=254
Reply from 10.254.1.1: bytes=32 time=32ms TTL=254
Reply from 10.254.1.1: bytes=32 time=32ms TTL=254
Reply from 10.254.1.1: bytes=32 time=16ms TTL=254
Reply from 10.254.1.1: bytes=32 time=32ms TTL=254
Reply from 10.254.1.1: bytes=32 time=32ms TTL=254
Reply from 10.254.1.1: bytes=32 time=32ms TTL=254
Reply from 10.254.1.1: bytes=32 time=94ms TTL=254
Reply from 10.254.1.1: bytes=32 time=32ms TTL=254
Reply from 10.254.1.1: bytes=32 time=32ms TTL=254
Reply from 10.254.1.1: bytes=32 time=16ms TTL=254
Reply from 10.254.1.1: bytes=32 time=32ms TTL=254

Reply from 10.2.1.1: bytes=32 time=125ms TTL=28
Reply from 10.2.1.1: bytes=32 time=125ms TTL=28
Reply from 10.2.1.1: bytes=32 time=156ms TTL=28
Reply from 10.2.1.1: bytes=32 time=172ms TTL=28
Reply from 10.2.1.1: bytes=32 time=125ms TTL=28
Reply from 10.2.1.1: bytes=32 time=125ms TTL=28
Reply from 10.2.1.1: bytes=32 time=125ms TTL=28
Reply from 10.2.1.1: bytes=32 time=188ms TTL=28
Reply from 10.2.1.1: bytes=32 time=109ms TTL=28
Reply from 10.2.1.1: bytes=32 time=125ms TTL=28
Reply from 10.2.1.1: bytes=32 time=109ms TTL=28
Reply from 10.2.1.1: bytes=32 time=172ms TTL=28
Reply from 10.2.1.1: bytes=32 time=141ms TTL=28
Reply from 10.2.1.1: bytes=32 time=172ms TTL=28
Reply from 10.2.1.1: bytes=32 time=125ms TTL=28
Reply from 10.2.1.1: bytes=32 time=125ms TTL=28
Reply from 10.2.1.1: bytes=32 time=125ms TTL=28

If is does it again I will show ya.

 

[spidey07]

sounds like a large e-mail or file copy or the dreaded listening to the radio or watching TV over the internet to me. Hopefully the "ip accounting" will help with your immediate needs, once you find suspect IP addresses then just figure out what application it is.

If you really want the top of the line WAN monitoring then use Visual Networks CSUs. I can't tell you how great these things are. You can see EVERYTHING, EVERYWHERE. Perform packet captures on every link, log latency, view top protocols and talkers. Really just about all you could ever ask in a WAN mangement system. I just reengineered our WAN and had ATT install visual CSUs at all sites. AT&T offers this service as Frame-relay Plus.

Or you can install the CSUs yourself.

 

[Shadow07]

I would have to agree with Spidey on this one. It definately sounds like someone is using a multicast program of some sort, whether or not it is Napster, Internet Radio, watching streaming porn, oh I mean training material, is anyone's guess. You will need a WAN network analyser to capture packets and to see who and what is being transfered on your WAN segments.

I have had no experience with the CSU's Spidey is talking about, but if he says they work great, hey, they work great.

 

[Dag]

We use the internal CSU's for the cisco's and Adtran TSU ESP's for the 3coms. The adtrans have some monitoring capabilities, but I haven't explored them.

I don't think it is streaming as that is only possible with a NAT(which me and about 3 others have, but no one else

It could be some video website with autodownloads or somthing though (web cam site or something).

Thanks for all the info. I will look into these programs and see if one is cost effective for us. I also have to find out how to do the ip accounting within the 3coms. They can usally do it, just tracking down the command is a pain.

Thanks again

 

[spidey07]

AHHH,

that's the downfall of using internal CSUs in the router. No chance to actually monitor your WAN other than basic SNMP stats. No RMON for YOU (seinfeld).

If you want to know what is happening on your wan then stay away from the internal CSUs.

 

[Shadow07]

Are you surea but that? I mean, the newer CSU's and routers support SNMP v2 and RMON I and II, I believe.

What series of Cisco and 3COM routers do you use?

 

[Dag]

Here are the Adtrans we use. These are only used on the 3Com's. http://www.adtran.com/data_sheets/html/61200/169/l1-8/index.html

3Com- Superstack II - NETBuilder 442's

We use 2600 Cisco routers with their internal CSU's.