QoS and Frame-relay

[spidey07]

Looks like its time for me to re-evaluate our QoS parameters and queuing strategies. Our frame network is pretty healthy as its built with NO congestion points in the network, meaning all ports can operate at full speed without consequence.

The problem comes from e-mail. we use lotus notes and replicate a ton of e-mail and databases. Some of these databases are over 6 gig in size. notes is pretty good at only replicating changes, but if there is a multimegabyte change or someone has 100 megabytes of mail to download then a 384 kbit frame connection is going to get slammed.

So my question is - what's the best QoS approach to this common problem? priority queuing? custom queueing? CBWFQ?

I'm using priority queueing right now because I run multicast video to all the sites. I'll probably stick to priority queueing unless somebody can enlighten me. Below are a few T1 pvcs...

DC-7204-1#sh traff
Access Target Byte Sustain Excess Interval Increment Adap
t
I/F List Rate Limit bits/int bits/int (ms) (bytes) Acti
ve
Se1/0.335 128000 1000 8000 0 62 1000 -
Se1/4 56000 875 56000 0 125 875 -
Se1/4.511 640000 10000 80000 0 125 10000 BECN
Se1/4.512 256000 4000 32000 0 125 4000 BECN
Se1/4.513 384000 6000 48000 0 125 6000 BECN
Se1/4.514 56000 875 56000 0 125 875 -
Se1/4.515 56000 875 56000 0 125 875 -
Se1/4.531 56000 875 56000 0 125 875 -
Se1/4.532 56000 875 56000 0 125 875 -
Se1/4.536 56000 875 56000 0 125 875 -
Se1/4.543 56000 875 56000 0 125 875 -
Se1/4.544 56000 875 56000 0 125 875 -
Se1/5.521 512000 8000 64000 0 125 8000 BECN
Se1/5.522 384000 6000 48000 0 125 6000 BECN
Se1/5.523 128000 2000 16000 0 125 2000 BECN
Se1/5.524 256000 4000 32000 0 125 4000 BECN
Se1/5.525 128000 2000 16000 0 125 2000 BECN
Se1/5.526 56000 875 56000 0 125 875 -
Se1/5.533 56000 875 56000 0 125 875 -
Se1/5.534 56000 875 56000 0 125 875 -
Se1/5.542 56000 875 56000 0 125 875 -
Se1/5.551 56000 875 56000 0 125 875 -

The small 56k pvs are all backups, the whole network is redundant. I'm thinking about moving notes from the normal queue to the low queue to eliviate some of the pain of large bulk transfers. Interactive apps like our ERP are medium queued and work fine.

Any ideas?

 

[spidey07]

bumpity...though some ideas out there fellas.

 

[ScottMac]

Well, nuts. I'll take a stab at it, though I doubt I'll hit anything you haven't already thought of....

I believe the bandwidth limiters / shapers (ala Packeteer) will allow you to adjust / shift your bandwidth by the clock, maybe your firewall will give you that or some additional capability.

What I was thinking would be to dynamically change the routing from "Point A" to "Point B" such that you route some of the traffic via an intermediate office .... kind of like doing an IMUX thing dynamically over diverse routes.

As a worse case, a clever script that dynamically re-programs your routers to accomplish the same basic thing. The routes / parmas could be pre-set manually (via management, term session, whatever) than shutdown .... then re-activated via script at the chosen time-of-day.

AG Systems actually did something similar for their ATM MPEG CODECs in their session management system. AN SQL database with all of the port/cable/route information was kept on the management station. When "Location A" wanted a session (either point-to-point, or a group), the management station looked up the connections necessary, then "automagically" (via SNMP) created a -* PVC *- for the required / specified duration, then it went back through and tore 'em all down.

It'd be a bit of programming and planning, but historically, it sounds like y'all could probably swing it.

Another option (*really not cheap*), might be to set up VSAT communications with the various major points, with a schedule to open up the bandwidth from A-to-B at the scheduled hours. I set up a VSAT for a guy that normally ran at 128/384, and on a schedule, it would switch to 384/384 or 768/768 for video teleconference / gross data transmission (they could also request a change with one or two hours notice). I don't know your specific deployment, but perhaps the extra bandwidth could be backhauled to other locations in the same neighborhood (Walgreens does this with their satellite system for some stores).

Something like that could work as an extraterrestrial backup link to counter the cable-seeking backhoe. I'm sure you have diverse paths for your critical links, but, ya never know ... some "alignment of the planets" kinda thing could happen, I s'pose...


That's my two cents worth, maybe it'll stimulate some other responses.

Good luck!

Scott

 

[Tallgeese]

I know this kinda diverges from the QOS questions, but after a certain point, you can only shoehorn so much traffic over your pipes.

If your Notes bandwidth needs get too insane...Citrix/Terminal Server options can suddenly become VERY attractive (as opposed to higher monthly bandwidth charges).

 

[ScottMac]

HEY! NOWHERE on the original post did he mention "Reasonable" or "Practical."

Good grief, Bean Counters....sheesh....:disgust:

Let's just keep things in the abstract, where they belong.....

JM.02

Scott

 

[Tallgeese]

Originally posted by: ScottMac
Good grief, Bean Counters....sheesh....:disgust:

Yep...I have been working for accountants too long, my friends

 

[Tallgeese]

For some strange reason, this item leapt out at me:
As an example, we found a user who had sent 17 GIG (yes, GIG) worth of e-mail over a two month period. :Q

 

[Garion]

Stupid question, but couldn't you just lower the priority of traffic going to/from your Lotus Notes servers to make sure there's available bandwidth for other applications?

Does Notes have any way to prioritize replication traffic within the app? IE, e-mail is always prioritized first, then database X, then database Y. How about scheduling of replication? Only replicate certain large database overnight, or only do replication of more than XX MB between 8 PM and 5 AM?

Notes is very cool, but one of the things that I don't like about it is that you mix your e-mail in with other applications, and it was fairly difficult to differentiate between them.

- G

 

[spidey07]

Well, so far the notes replicator (the portion that handles moving data) is set for 15 minutes. This gives a nice happy medium between rapid e-mail delivery and database replication. We like to do things pretty real-time so I probably can't change the intervals, and in our situation it probably wouldn't help.

Putting notes in a low priority queue would work fine. But I'm faced with the decision of "Do I risk starving off e-mail?" That's a though decision to make and why I'm thinking about CBWFQ or custom queuing. Guarantee notes a small amount of bandwidth but leave it low priority.

PS - we move over 20 gigabytes of mail everyday. Just like every corp - e-mail is the true "killer app" in more ways than one.

 

[Tallgeese]

The funniest phrase I have ever heard in my career: "WTH...it's just e-mail"
Spoken by someone who had/has never administered a messaging system before, to be sure. :frown:

Personally my take would be to avoid starving Notes.
Nothing pisses a VP or other wheel off faster than not getting e-mail fast enough.
I'm a bit confused tho, how would "guaranteeing Notes a small amount of bandwidth, but leaving it low priority" happen?
Isn't that a contradiction in terms? Unless some other traffic type(s) shared that same bandwidth pool?

As in (and numbers are for illustration only):

=====================
Pool X (128kb/s total)

Traffic type1 (high)
Traffic type2 (med)
Traffic type3 (low)
--------------------------
Notes (50Kb/s low)

=====================

 

[EasyE]

Well, priority queuing services the high queues first then will get to the lower ones *if* the high one is empty SO it is possible that the low q never gets serviced if there is always stuff in the high q.

It sounds like you may want to reserve a slice of the bandwidth for notes, which CBWFQ should work fine for. You could also rate-limit that with CAR.

You could get fancy and use time-based ACLs to adjust the traffic for times of the day to, or even use time-based ACLs to adjust routing metrics at certain times like ScottMac was suggesting.

As for alternatives, how about going with a web-based type solution so they aren't transferring megabytes of data all the time.
This could be an excuse to setup a extranet (VPN) for your office.... then users could VPN in from home and access it. Or... SSH (Hi ScottMac)

E

 

[spidey07]

Already have a large VPN. Home users get DSL/vpn client, roamers get vpn client or private dial-up.

I'm gonna do some more looking into CBWFQ or custom queueing. Priority queueing works great, but I can't risk starving e-mail. Maybe straight custom queueing would work well, but the configuration is going to get messy with over 100 pvcs.

Need to do some lab work.

 

[EasyE]

If you're on 12.2(2)T or higher and all the sub-interfaces need the queueing, they enhanced the CBWFQ stuff in 12.2T so you can specify a % of bandwidth to use. This way if the policy is pretty common between the subs you can specify like 10% for the email class, 70% for whatever else, and remaining goes to default class and apply the policy to the main interface. There is a restriction though, you can't apply service-policys to individual sub's if there is a policy applied to the main interface. Also in some IOS's you may be limited to the # subs where policys can be applied. A colleage of mine ran into that problem.

But be careful and make sure your subs and FRTS values are set correctly so the bandwidth figured is correct. It uses the mincir value to figure this. By default that is 1/2 of the cir value.

http://www.cisco.com/univercd/cc/td/doc/product/software/ios122/122newft//122t/122t2/ftllqpct.htm


See this cisco link

 

[Garion]

Seems like you're in a catch 22 - You want to optimize bandwidth to allow e-mail to perform well, yet e-mail is one of the bandwidth hogs that you need to control more tightly.

It's been a while since I've worked with Notes (4.x days), but other "enterprise" mail systems seem to work best when e-mail isn't distributed and kept in the main data centers. Do you really need to replicate it, or could you serve it off central servers?

Actually, the same goes for the rest of your Notes databases.. It's quite efficient on bandwidth - Why replicate the same database across multiple sites when that causes replication conflict issues and might never be accessed? I can understand if you're dealing with massive file attachments. After all, how much data can a bunch of guys who make whiskey actually generate??

*grin*

- G

 

[SR]

I'd setup up cbwfq for notes and give it X kb using the bandwidth command and for your video/voice use the priority command. I'm sure know the differences being the smartest network engineer on the board. Anyhow here is a link describing the differences for other people.

http://www.cisco.com/warp/public/105/priorityvsbw.html




How does everyone like the changes to Cisco's web site?

 

[spidey07]

Thanks for the suggestions guys.

Probably high time I upgraded our routers from 12.0 to 12.2.