My frustrations as a network engineer

[freegeeks]

I'm an IP/MPLS engineer working for an ISP. I configure and troubleshoot MPLS services (L2 - L3) for business customers. My customers are sometimes very small networks (just a few sites) to pretty big (hundreds of sites).

As everyone noticed who works in this business there has been an explosion of services running over the WAN. Everything from cloudstuff, thin clients (citrix, ...), voip, "insert buzzword of the week here"

What I noticed even more that with these added complexity, the knowledge and networking troubleshooting skills on the customer side are pretty poor. More then 50% of time I'm basically troubleshooting stuff on the LAN of the customers (not my job but well, if something is not working, it's the network taht is always blamed first). I pretty much do everything from getting their routers connected (they choose to have unmanaged), troubleshooting voip (consumes most of my time nowadays), performance (99% of the people I deal with have no idea how to optimize tcp for LFN and just complain that the network is slow), the concepts of qos (I probably explained 1 zillion times that qos is something that kicks in when there is congestion in a network). Iperf and wireshark are unknown to lots of "engineers" I deal with. And it's not only the customers ICT, it is also their integrators (don't get me started about that).

Sometimes I wish I was a barista at starbucks

end-of-rant

 

[drebo]

I've started implementing VRFs for my hosted pbxs so that I can tunnel my customers' LANs across and avoid NAT. Why? Because my customers insist on running their offices on $50 consumer routers and don't want to pay for a managed router.

 

[Pheran]

Just like every other profession, there are a lot of clueless people out there. I was once speaking with a "senior network engineer" who was asking me if I could see the MAC address of his server in my ARP cache. Mind you, this server was a couple of states and several router hops away. Don't even get me started on trying to troubleshoot VPN connections with the people who support the network in hotels. One of them had no idea that there were IP protocols other than TCP and UDP (e.g. ESP).

 

[freegeeks]

I've started implementing VRFs for my hosted pbxs so that I can tunnel my customers' LANs across and avoid NAT. Why? Because my customers insist on running their offices on $50 consumer routers and don't want to pay for a managed router.

try customers who are trying to setup multi-homed internet access with routers that don't support BGP (no sir, static routes are not a preferred solution for this setup)

 

[spidey07]

Heh, I get paid to come in after all these folks and clean up their mess. It's a great gig. Most of my clients are service providers and fortune 100 companies.

Networking is the "voodoo" of IT, black magic. Thing is it really isn't that hard.

 

[drebo]

try customers who are trying to setup multi-homed internet access with routers that don't support BGP (no sir, static routes are not a preferred solution for this setup)

I had that situation with one of my customers.

He wanted load balancing and failover between a connection he bought from us and a pre-existing connection from AT&T. He didn't have an ASN or IP allocation. I told him his existing PIX 515 wouldn't do what he wanted.

He proceeded to send me two different application load balancers (both of which were well past end of sale).

Finally, I was able to talk him down and asked him to come up with a simple PBR policy and then he contracted me to implement it for him. URRG.

 

[drebo]

Just like every other profession, there are a lot of clueless people out there. I was once speaking with a "senior network engineer" who was asking me if I could see the MAC address of his server in my ARP cache. Mind you, this server was a couple of states and several router hops away. Don't even get me started on trying to troubleshoot VPN connections with the people who support the network in hotels. One of them had no idea that there were IP protocols other than TCP and UDP (e.g. ESP).

The worst are the dumb customers who go to hotels where they block or haven't properly allowed PPTP. They'll call every single time they're at a hotel where it doesn't work. As if this time is somehow different than the last 10 times.

Thankfully, SSL VPN solutions are becoming much less expensive to deploy.

 

[Pheran]

Networking is the "voodoo" of IT, black magic. Thing is it really isn't that hard.

The basics aren't that hard, but I've certainly troubleshot a few things that ended up being pretty goddamned complicated, up to and including Linux kernel bugs and Microsoft network stack bugs. The thing about networking is that it ties everything together, so it's hugely helpful to have a wide background in system administration and programming doesn't hurt either.

 

[m1ldslide1]

It's called "job security". Now stop complaining and start hoping nobody invents a GUI that automates your job

 

[freegeeks]

I had that situation with one of my customers.

He wanted load balancing and failover between a connection he bought from us and a pre-existing connection from AT&T. He didn't have an ASN or IP allocation. I told him his existing PIX 515 wouldn't do what he wanted.

He proceeded to send me two different application load balancers (both of which were well past end of sale).

Finally, I was able to talk him down and asked him to come up with a simple PBR policy and then he contracted me to implement it for him. URRG.

PBR is the devil :awe:

 

[freegeeks]

Just like every other profession, there are a lot of clueless people out there. I was once speaking with a "senior network engineer" who was asking me if I could see the MAC address of his server in my ARP cache. Mind you, this server was a couple of states and several router hops away. Don't even get me started on trying to troubleshoot VPN connections with the people who support the network in hotels. One of them had no idea that there were IP protocols other than TCP and UDP (e.g. ESP).

I'm sure you would be surprised about the answers if you ask about ARP to a lot of network engineers. I'm sometimes troubleshooting with customers, they put an unmanaged firewall on the link and when I do a ping and tell them something basic, probably layer 1 is not OK because I don't get an arp reply, they just say, it's normal, I'm blocking ICMP on my firewall

Then I know it's going to be another long day

As a matter of fact, security engineers are the worst, most of them are totally clueless about networking, they just click in their fancy GUI

 

[drebo]

My favorite is when I go into a customer to deploy a PBX and find their internal network is non-RFC1918. I've seen 100.10.10.0/24, 172.168.10.0/24, and more. I once had a Mitel network tech propose using 172.168.x.0/24 as their voice vlans.

 

[freegeeks]

My favorite is when I go into a customer to deploy a PBX and find their internal network is non-RFC1918. I've seen 100.10.10.0/24, 172.168.10.0/24, and more. I once had a Mitel network tech propose using 172.168.x.0/24 as their voice vlans.

LOL, yeah I know the feeling

 

[mammador]

Many small organisations i guess don't have a dedicated IT function. They simply call their ISP to sort out any and all networking issues. It makes sense for obvious reasons in a way, since a company with 10 employees doesn't need a mass IT function.

Is MPLS more or less standard now, and have older stuff like frame relay completely gone?

 

[Pheran]

My favorite is when I go into a customer to deploy a PBX and find their internal network is non-RFC1918. I've seen 100.10.10.0/24, 172.168.10.0/24, and more. I once had a Mitel network tech propose using 172.168.x.0/24 as their voice vlans.

I can top that one. Many years ago (before I worked there), someone at the UK division of one of my former employers decided it would be a great idea to allocate a class A network (yes I really mean a /8) to every one of their offices - completely flat, no masks. Where did they get that much IP space, you might ask? Well, conveniently IANA had a bunch of class A's that were marked "reserved", so why not use them?? Of course they were only reserved because they hadn't been allocated yet. As soon as those blocks started getting handed out to the regional registries, huge swaths of the Internet became unreachable because of these idiotic office allocations.

 

[Gryz]

My favorite is when I go into a customer to deploy a PBX and find their internal network is non-RFC1918. I've seen 100.10.10.0/24, 172.168.10.0/24, and more.

And why is that a problem ?

You can do mapping both ways.
Nowadays, everybody knows 10/8 is private address space. And thus everybody is familiar with "map official external ip-addresses to internal 10/8". But NAT is a lot more flexible. You can say "map external 100/8 to internal 10/8" just as well. So suddenly all "real" 100/8 address look like 10/8 on the internal network. And there is no confusion between anymore between the customer's "fake" 100/8 and the real 100/8 out on the Internet. It just takes a little extra configuration on the NAT boxes.

RFC1918 is from 1996. I think NAT is older.
(Yep, I was right: http://en.wikipedia.org/wiki/Cisco_PIX#History )
The first examples of what you could do with NAT showed some crazy examples. Like 3 merged companies, all using the same IP addresses that they had just "hijacked". With the proper config, you can make it all work.


BTW, good to hear there is still a huge shortage of capable network engineers. As Mudslide said: awesome job security !

 

[Pheran]

And why is that a problem ?

Because if you are sitting on that subnet and you need to reach whatever is on the Internet at that IP, you are totally screwed. NAT is not going to help at all. And everywhere else in the company with native routing to that subnet is in the same boat.

Yes, if you are merging with someone who did something this dumb you can do tricks with destination NAT and route filters, but it's a big pain.

 

[ScottMac]

I was thinking (for a while actually) that a good b1tch thread might be interesting ... no whining ... just aggravations, deviations of Best Practice conventions, NetIdiots gone wild, the bosses kid's networking ...etc. Do you think a sticky on this thread (or starting one similar) would be worthwhile?

(also assuming the other mods have no issues with it ... ).

 

[spidey07]

Plugging linksys router in with a bid summary or default route into the data center.

Linksys does proxy arp, and because he's not really doing anything, he has more than enough time to respond to every arp request he sees. That is all.

 

[drebo]

And why is that a problem ?

You can do mapping both ways.
Nowadays, everybody knows 10/8 is private address space. And thus everybody is familiar with "map official external ip-addresses to internal 10/8". But NAT is a lot more flexible. You can say "map external 100/8 to internal 10/8" just as well. So suddenly all "real" 100/8 address look like 10/8 on the internal network. And there is no confusion between anymore between the customer's "fake" 100/8 and the real 100/8 out on the Internet. It just takes a little extra configuration on the NAT boxes.

I'm not talking about an advanced network where cohabitation exists or where you're doing a bunch of 1:1 NATs from internal or external or NATing across an IPSec tunnel or anything.

I'm talking about a simple, one-site location with a single PAT where the network admin had no idea what he was doing.

 

[cmetz]

Heh, I get paid to come in after all these folks and clean up their mess. It's a great gig. Most of my clients are service providers and fortune 100 companies.

Networking is the "voodoo" of IT, black magic. Thing is it really isn't that hard.

+1.

Also, I've seen that many companies won't pay what it takes to hire someone competent, and then can't figure out why their network only mostly works.

Is MPLS more or less standard now, and have older stuff like frame relay completely gone?

Frame is done, put a fork in it. I don't think you can even buy frame switches new anymore, telcos are in attrition mode now.

ATM for virtual circuit purposes never did work all that well. Telcos like the concept, it's just that ATM was technologically retarded.

So for modern-speed interfaces, MPLS is very much the standard way to get a virtual circuit now, obosoleting both Frame and ATM.

 

[Gryz]

Because if you are sitting on that subnet and you need to reach whatever is on the Internet at that IP, you are totally screwed. NAT is not going to help at all. And everywhere else in the company with native routing to that subnet is in the same boat.

I don't think that is true.
I agree it's a pain. And requires more configuration. And sometimes it might not work. But there is a solution.

Suppose you configured your internal network as 100/8.
Now you can not reach "the real 100/8" on the Internet. As you say.
First you need an official IP-range. Say we get 177/8.

On the NAT box, you can configure translation like this:
any IP address in the 100/8 range coming from the outside, translate it to 10/8 going in.
any IP address in the 100/8 range coming from the inside, translate it to 177/8 going out.

Suppose you misconfigured your whole internal network to use 100/8.
Suppose www.something.com is at 100.1.1.1.
Now an internal PC, with "fake" address 100.1.1.1 wants to connect to www.something.com.
The PC does a DNS request. DNS request goes out through your NAT box.
Reply comes back: "www.something.com A 100.1.1.1".
NAT box intercepts the reply. And changes it into: "www.something.com A 10.1.1.1".
PC receives "www.something.com A 10.1.1.1".
PC sends a TCP syn (or whatever) to 10.1.1.1.
The routing inside your network points 10/8 (or 0/0) towards your NAT box.
NAT box receives the TCP syn.
Now the NAT box translates the TCP syn to have: src addr: 177.1.1.1 and dest addr: 100.1.1.1.
That packet gets sent over the Internet. And reaches www.something.com.
Return packet is sent to 177.1.1.1 by www.something.com.
NAT box translates 177.1.1.1 to 100.1.1.1 when it gets into the internal network.

I hope I made myself clear.
This is an unusual config now.
But when the first PIX boxes arrived (before rfc1918), this was an intended scenario to help people who had misnumbered their networks. It's a lot less common nowadays, I guess. Because everybody knows you need to apply for official addresses. But in 1994-1996 I bet there were much more people making that mistake.

 

[spidey07]

If the host is 10/8, it will never send to, nor reach any NAT device/router. You would have to have the nat router intercept the DNS query from the client, which for any modern network simply wouldn't happen.

Now if the NAT happens between the client and DNS server, sure. But DAMN that is ugly. May as well re-address at that point.

 

[Gryz]

Yes, I mean that the NAT box also translates the DNS packet.
And yes, it is ugly.

But NAT at first was intended as a tool to delay renumbering. Only (slightly) later, when PAT was invented (small change), NAT was used as a tool to lower the amount of public addresses needed. I don't think anyone is using (complex) NAT configs anymore to delay renumbering. But if you'd want to, you can do it. I guess the trade-off is just how many addresses you need to renumber. Even then, personally I would always chose to renumber.

Remember, when NAT was invented (in 1994) DHCP was not everywhere yet. Renumbering was even more painful then, than it is now.

 

[jersiq]

I've got you from the other side. I was at a Wireless ISP who purchases L2 services from a "telco." Most of the time, we troubleshoot their equipment. <cough>Verizon Business<cough>
In the end, my favorites were the "protected" paths that rode a single linear physical link.