My frustrations as a network engineer

[Pheran]

I don't think that is true.
I agree it's a pain. And requires more configuration. And sometimes it might not work. But there is a solution.

Yes, I'm completely aware that if you NAT DNS this can be made to work. It's just stupid and ugly, as spidey already mentioned.

 

[xSauronx]

My favorite is when I go into a customer to deploy a PBX and find their internal network is non-RFC1918. I've seen 100.10.10.0/24, 172.168.10.0/24, and more. I once had a Mitel network tech propose using 172.168.x.0/24 as their voice vlans.

one of our customer uses 129.102.52.x for some god damn reason, and i cant talk them into changing it because they use it for the wireless links and monitoring units for all of the city utilities and apparently it would be a HUGE pain in the ass to change all of that.

bonus: ALL of the addresses (probably 80 or better) are statically assigned. every.fucking.one.

im no network engineer, but jesus tapdancing christ every time i have to work on that network it drives me nuts

 

[imagoon]

Yes, I mean that the NAT box also translates the DNS packet.
And yes, it is ugly.

But NAT at first was intended as a tool to delay renumbering. Only (slightly) later, when PAT was invented (small change), NAT was used as a tool to lower the amount of public addresses needed. I don't think anyone is using (complex) NAT configs anymore to delay renumbering. But if you'd want to, you can do it. I guess the trade-off is just how many addresses you need to renumber. Even then, personally I would always chose to renumber.

Remember, when NAT was invented (in 1994) DHCP was not everywhere yet. Renumbering was even more painful then, than it is now.

That would be one heck a of a kludge that would break all sorts of things like DNSSEC, certificates, HTTPS to name just a few. (Assuming the certs are not wildcard.) I would expect email / MX records to be fun also... IE if the srv record indicates that only certain IPs are allowed to send email and yours are all translated.

 

[imagoon]

I was thinking (for a while actually) that a good b1tch thread might be interesting ... no whining ... just aggravations, deviations of Best Practice conventions, NetIdiots gone wild, the bosses kid's networking ...etc. Do you think a sticky on this thread (or starting one similar) would be worthwhile?

(also assuming the other mods have no issues with it ... ).

I would love to see the equivalent of the Server Room "What did you learn today" thread in this forum someday like Ars Technica. Granted Danger Mouse over there definitely gives that thread some class.

 

[theevilsharpie]

one of our customer uses 129.102.52.x for some god damn reason, and i cant talk them into changing it because they use it for the wireless links and monitoring units for all of the city utilities and apparently it would be a HUGE pain in the ass to change all of that.

Are you sure they don't own that block?

 

[Cooky]

The place I work doesn't seem to understand the importance of the network.
Higher up's have the mentality it's as easy as home networking w/ a Linksys router.
The server team is two to three times the size of the network team.
Apps team is even bigger, but I'll accept that, since we have all sorts of apps that need maintained & developed, etc.
What they don't seem to realize is EVERYTHING rides over the network, even voice & video.
Yet they scream & yell when there's an issue, but they're not willing to pony up when budget is submitted.

 

[Pheran]

Are you sure they don't own that block?

Unless they are also a music school in Paris, no they don't own that block.

 

[freegeeks]

Many small organisations i guess don't have a dedicated IT function. They simply call their ISP to sort out any and all networking issues. It makes sense for obvious reasons in a way, since a company with 10 employees doesn't need a mass IT function.

Is MPLS more or less standard now, and have older stuff like frame relay completely gone?

ATM is the way of the dodo. It is still there on the access for lots of DSL products but even the uplink from the DSLAM is ethernet now. New DSL products like VDSL2 are ethernet

 

[drebo]

Yeah, ATM was a great idea, but the implementations tended to be expensive and not worth the trouble. Especially since the tech basically matured along side MPLS and MPLS is much, much more flexible.

 

[Gryz]

If you maintain things, the trick is to not work in first-line support. But second-line, or even further from the first-line away. First-line is a very ungrateful job. When you do your job well, and there are hardly problems, nobody will realize you are there. And in the few cases something goes wrong, you get blamed extra hard.

But when you work in the backline, things change. Problems occur. People get blamed. Problems don't get solved. Things heat up. Things explode. Then you, in 2nd or 3rd-line support get involved. You solve the problem. You are the hero. The opportunity for gratefulness, even fame, is much higher.

Is this just my experience/opinion, or have people seen this in other places too ?

 

[ScottMac]

Yeah, ATM was a great idea, but the implementations tended to be expensive and not worth the trouble. Especially since the tech basically matured along side MPLS and MPLS is much, much more flexible.

ATM was well ahead of MPLS, by years. The technology of the day is what made the case for ATM. Using fixed length containers (cells), it greatly accelerated switching, and provided configurable and predictable latencies.

RSVP was just getting off the ground, and could only support QOS in a single span (to do end-to-end t had to be manually configured per span), where ATM could provide complete and reliable end-to-end QOS as a matter of applied policy.

Back in the day I worked in the Interoperability Lab at Anixter. We set up side-by-side demonstrations using models of customer's networks versus the same network with an ATM core (and in some cases ATM to the desktop). With tools like Chariot and Spirent NetBits we could objectively evaluate performance, and ATM won every time.

Fore Systems (now Marconi) made some great stuff. Cisco sucked at ATM, so they hated it (a chicken and egg situation), Nortel had some nice edge equipment, but their core sucked. Fore Systems is gone, Marconi bailed on ATM (probably a wise move), Nortel died, and Cisco got their GigE and moved forward, though it took Extreme kicking their ass in performance to get 'em moving.

Certainly there are better technologies now, mostly because technology (primarily processing power) has advanced. But back in the day, ATM beat everything, and we could prove it.

 

[Gryz]

There's enough reasons why ATM wasn't great. Too much overhead, too much complexity. Give me Ethernet or PoS any day. But there was one thing that ATM did well in those days. And that was a way for network engineers to traffic-engineer their large-scale flows.

Two out of the three largest ISPs in the US used a core of Fore ATM-switches to build their network. And had (cisco) routers at the edge of that large ATM cloud. It worked well, even if you had dozens, or even hundreds of routers with a full mesh of ATM PVCs. From the router side, I think we could have made it work with 1000+ routers in a (double) full mesh. Those two ISPs were quite happy with that setup. The 3rd one hated ATM, and was the place were PoS was first deployed.

Those two ISPs each had $100M+ of Fore equipment in their network. Cisco didn't like that. As it was equipment in the core of an IP network, they thought that money belonged to them. At the same time, a startup called Ipsilon was promoting a box that did some sort of flow/tag-switching with ATM and IP. Telcos (and ISPs) were very interested in Ipsilon's technology, because Telcos understood Connection-Oriented, and hated ConnectionLess.

So cisco came up with tag-switching. The first goal was to do traffic-engineering. ISPs en telcos picked up the idea quickly. So did the research world and other vendors. And the IETF. It was quickly renamed MPLS, and it became a community-effort. Ipsilon never took off. And Fore was dead within a few years.

Personally I think MPLS is overly complex. And a technology like MPLS-VPNs could have been done in a much simpler, elegant, more scalable way, without MPLS. Some technologies have been invented, just to give MPLS a purpose. Such as MPLS-VPNs. I never liked MPLS-FastReroute (why not let the IGP do fast reroute ?) But it is probably good that ATM is gone.

 

[Railgun]

What I noticed even more that with these added complexity, the knowledge and networking troubleshooting skills on the customer side are pretty poor.

Believe me...my experience has been the providers are completely inept, from not having a clue how to even navigate the equipemnt they're supposed to support to waiting for some callback for three hours for circuit down issues.

Not all are bad, but the bigger ones are the problem. It sounds as though you're not one of those engineers and hope that you work for one of our providers.

 

[Pheran]

Believe me...my experience has been the providers are completely inept, from not having a clue how to even navigate the equipemnt they're supposed to support to waiting for some callback for three hours for circuit down issues.

Not all are bad, but the bigger ones are the problem. It sounds as though you're not one of those engineers and hope that you work for one of our providers.

I have to agree with Railgun, my experience is that ISPs are usually terrible. That's not because they don't have any good engineers working for them - no doubt they do, in fact on rare occasions I've gotten to talk to a few of those engineers, and those calls are a pleasure. But they usually have a huge layer of bureaucracy and more-or-less clueless first-level support in front of those engineers. So I've had experiences where I can't see an ISP-provided loop because the engineer is looping some obsolete leg of the circuit that got replaced two years ago, and I have to prove that to them over the course of HOURS.

 

[imagoon]

ATM was neat tech but it is less than necessary today. I could see a certain potential for a larger cell ATM say 1500 bytes but I think it would be one heck of a battle considering IP / Ethernet is the main man at the moment.

As for MPLS, yes it is a bit complex and I personally think the ISP's could have implemented with the back end a bit more hidden but for what it does (private channels over public networksm with out overlap or access to other channels) it does decently. I have seen some ISP's handle MPLS by making the customer have just a flat IP range and then let them handle IP's (IE there is no MPLS config on the local router, just a 'black box' that gives you an ethernet port and a pseudo layer 2 to all the other routers in your network.

 

[Pheran]

ATM was neat tech but it is less than necessary today. I could see a certain potential for a larger cell ATM say 1500 bytes but I think it would be one heck of a battle considering IP / Ethernet is the main man at the moment.

The funny thing I've been seeing lately is that technologies that ATM had in the 90s are now coming to ethernet. Heard of TRILL/FabricPath? It's just a routing protocol at layer 2. ATM was doing that 15+ years ago, it was called PNNI. Of course, having that kind of stuff adds complexity to the network, which was one of the downfalls of ATM.

 

[ScottMac]

Alrighty. I set it as a sticky, let's give it a whirl for a bit and see if it stays active.

 

[Railgun]

Just like every other profession, there are a lot of clueless people out there. I was once speaking with a "senior network engineer" who was asking me if I could see the MAC address of his server in my ARP cache. Mind you, this server was a couple of states and several router hops away. Don't even get me started on trying to troubleshoot VPN connections with the people who support the network in hotels. One of them had no idea that there were IP protocols other than TCP and UDP (e.g. ESP).

Meh...I asked something similar of someone and they said the only Mac they were familiar with was the Apple they were using.

 

[imagoon]

So here is my frustrations for the day:

I know a worldwide organization that lives with static routing and ASA style mesh VPN for every location handling everything from data replication to VOIP. Also the Internet is split tunneled which can make sense if you used the Cisco tools to keep the config consistent but they don't. When you connect via the VPN agent, different VPNs (ie differing ASAs) have differing access to differing segments due to the variation in static routes and split tunnel config. The frustrating part is when send a suggestion up I get a response that my professional experience doesn't apply to this company. I am not always sure how I should take that.

 

[freegeeks]

ATM was well ahead of MPLS, by years. The technology of the day is what made the case for ATM. Using fixed length containers (cells), it greatly accelerated switching, and provided configurable and predictable latencies.

RSVP was just getting off the ground, and could only support QOS in a single span (to do end-to-end t had to be manually configured per span), where ATM could provide complete and reliable end-to-end QOS as a matter of applied policy.

Back in the day I worked in the Interoperability Lab at Anixter. We set up side-by-side demonstrations using models of customer's networks versus the same network with an ATM core (and in some cases ATM to the desktop). With tools like Chariot and Spirent NetBits we could objectively evaluate performance, and ATM won every time.

Fore Systems (now Marconi) made some great stuff. Cisco sucked at ATM, so they hated it (a chicken and egg situation), Nortel had some nice edge equipment, but their core sucked. Fore Systems is gone, Marconi bailed on ATM (probably a wise move), Nortel died, and Cisco got their GigE and moved forward, though it took Extreme kicking their ass in performance to get 'em moving.

Certainly there are better technologies now, mostly because technology (primarily processing power) has advanced. But back in the day, ATM beat everything, and we could prove it.

cost of interface cards was one of it's biggest downfalls. I was an ATM backbone engineer in the beginning of 2000 and the cost of STM-16->STM-64 cards was prohibitive to say the least. We were transporting ip Video On Demand over ATM, the business case didn't make any sense to be honest. Large overhead, complex, expensive. It had its place in the 90's and in the beginning of the millenium but nowadays it doesn't make any sense

 

[freegeeks]

ATM was neat tech but it is less than necessary today. I could see a certain potential for a larger cell ATM say 1500 bytes but I think it would be one heck of a battle considering IP / Ethernet is the main man at the moment.

As for MPLS, yes it is a bit complex and I personally think the ISP's could have implemented with the back end a bit more hidden but for what it does (private channels over public networksm with out overlap or access to other channels) it does decently. I have seen some ISP's handle MPLS by making the customer have just a flat IP range and then let them handle IP's (IE there is no MPLS config on the local router, just a 'black box' that gives you an ethernet port and a pseudo layer 2 to all the other routers in your network.

that is the normal setup. CPE to PE is just normal IP routing, the mpls part starts at the PE router. There are few use cases where mpls to the CPE makes any sense. And the black box is in many cases just an ethernet demarcation unit. When you are delivering fiber, in most cases you don't want to plug your fiber directly into the customers CPE.

 

[imagoon]

that is the normal setup. CPE to PE is just normal IP routing, the mpls part starts at the PE router. There are few use cases where mpls to the CPE makes any sense. And the black box is in many cases just an ethernet demarcation unit. When you are delivering fiber, in most cases you don't want to plug your fiber directly into the customers CPE.

I have had seem quite a few cases were the terminated MPLS on T1's and the CPE was required to have the circuit id data, and share routing data with the ISP as well. I have seen that more than the "black box" approach actually.

 

[freegeeks]

I have had seem quite a few cases were the terminated MPLS on T1's and the CPE was required to have the circuit id data, and share routing data with the ISP as well. I have seen that more than the "black box" approach actually.

strange setup. I have worked for quite a few ISP and CE to PE is just normal ip routing (ospf, BGP, ....)
We only use mpls to the CPE in special cases like when fast-reroute and in mobile backhaul (which is probably the number one use case). I don't see a compelling reason to deliver mpls to the CPE in most user cases. When you have unmanaged routers, the customers CPE needs to support mpls and by my experience and also by other people in this thread, the customers wants a cheap box on his circuit

 

[drebo]

MPLS to CE has a place. Split tunneling of Internet Access and IPVPN, for instance. I've been considering doing it for Hosted PBX access and ATA access.

Some more advanced customers that want to run their own routing protocol between their sites for various reasons (multiple paths, etc) might want it that way, also. Also, L2 VPNs delivered over non-Ethernet solutions can benefit from having MPLS all the way at the CE. Just depends on the provider's implementation.

For me, I mostly just run BGP from PE to CE and leave the label popping to the PE.

 

[freegeeks]

give me your top troubleshooting tools for voip, currently I use cisco ip sla and iperf. Any other tools (preferably open source / free). I have mixed feelings about cisco ip sla, results are not always consistent