Is corporate IT always a mess?

[Insomniator]

My first job out of college was for a small (medium?) sized shop. ~100 virtual Windows servers all on the same domain, few behind a DMZ. Everything worked and we knew where everything was. Changes, patching, reboots etc could be just coordinated and done whenever we wanted. ~10 IT people total.

Job for the past couple years is the complete opposite. We have ~1200 servers, multiple domains, linux/unix/windows 2k - 2k12, multiple DMZs, 200 different owners, apps that can never be down etc. If the current director asked us for a simple list of our servers, we couldn't even give him a confident sheet that has everything, or even what everything we know we have does. Doesn't seem like anyone knows what the hell is goin on.

What is more representative of your average place?

 

[ultimatebob]

The bigger and more poorly managed the company is, the more it will look like the latter option.

The IT department where I work now probably has no idea how many servers we have. Most of the QA testers and developers have their own servers. I'll bet that most of those "rogue" boxes aren't being patched properly.

When I used to work at IBM, they cracked down on behavior like this and made everyone consolidate their development and test servers into big server rooms with draconian security policies that basically prevented people from getting their jobs done without a sysadmin's help. I'll bet that the developers all use VM's for stuff like this now

 

[kt]

Yes, corporate IT is always a mess. It's simply due to corporate politics.

 

[trmiv]

Insomniator and ultimatebob are pretty much describing where I work, and I'm pretty they don't work where I work. So I'm going with "yes" as the answer to the title of this post.

 

[Exterous]

Yes, corporate IT is always a mess. It's simply due to corporate politics.

Eh I don't know if its always politics. I need to beat up on a production server so I clone it and beat up on that. Now there is another server that no one else really needs to know about (assuming I can network it if I even need to) to do their jobs correctly. Hell, some virtualization products now make that automatic.

"How many servers do we have?"
"375...wait 376....er...340...."

Also some servers may be setup to do a simple job and they do it so well that you forget they are doing it. Someone asks you for a list and you forget to put it on it because you haven't touched it in 3 years

There's also the aggregation complexity. You have multiple domains managed by other groups who then give control to additional groups down the hierarchy to their own folders. That group then create more folders and gives local groups access to those. Your top level group isn't going to know if all hundred thousand AD accounts are needed and active. Your lower groups may not know or understand what systems are intertwined

And the always dreaded IT guy who managed stuff leaves after X years. "What is this\does this?" "I dunno - Bob managed that."

 

[vi edit]

Try doing this for a government where one part owns servers and application space. And another part owns the Domain OU's, cabling, networking and structured cabling part. And part that owns most of the infrastructure and AD space has an SLA structure with response times about 10x slower than the server & app side.

GOOD TIMES!!!!

 

[AznAnarchy99]

Also you run the risk of breaking something when you try to clean shit up.

 

[yllus]

Probably. Corporate IT is usually informed at the very last minute about planned product rollouts, corporate mergers/takeovers, or other major events that would mean significant changes to the company's systems. There's no human way for them to keep up when they're never invited to the table about major company decisions.

 

[kstu]

Probably. Corporate IT is usually informed at the very last minute about planned product rollouts, corporate mergers/takeovers, or other major events that would mean significant changes to the company's systems. There's no human way for them to keep up when they're never invited to the table about major company decisions.

They are also one of the first and hardest hit when it's time to make cuts since they are an enormous cost center.

"But we support the profit centers" always seems to fall on deaf ears.

 

[Kaido]

The bigger and more poorly managed the company is, the more it will look like the latter option.

You can see this in film production too...the movies where one dude (or a crack team of people) is running the show vs. the ones that are "design by committee". Same in car design:

The best shops are run by benign IT dictators...all of the information flows through them, but they don't make boneheaded decisions like requiring all developer authorizations to go through a sysadmin. But you still have one guy calling all of the shots so that you don't go into crazytown with a committee. And the bigger you are, the more carefully the decisions have to be made because if you're blasting out a single change & affecting thousands of computers, you have to make sure you're not blowing up somebody's workflow or killing functionality.

The mess is what drives me nuts. I demand full traceability...I want every wire, every outlet, every piece of hardware to be labeled & stored in a system that I can reference in the event of an issue or audit. It's not rocket science, but it is a lot of work up-front to make it happen, and so many places just have no idea what they have in inventory or what their full policy set is. Look at how many places have been hacked...the CIA, Target, IRS, Home Depot, you name it. It's really difficult to have a large company & also be crazy-organized in regards to IT.

 

[Gatecrasher3]

From what I have seen, unless protocols are put in place right from the start, it turns to shit shortly after, and near impossible to turn things around...

 

[ctbaars]

~100 virtual Windows servers a.... ~10 IT people?
~1200 servers, multiple yada yada ... ~1 IT Insomniator?
??
+1 Kaido : 5/7 post
Committees are the worst.

 

[BoberFett]

The bigger and more poorly managed the company is, the more it will look like the latter option.

The IT department where I work now probably has no idea how many servers we have. Most of the QA testers and developers have their own servers. I'll bet that most of those "rogue" boxes aren't being patched properly.

When I used to work at IBM, they cracked down on behavior like this and made everyone consolidate their development and test servers into big server rooms with draconian security policies that basically prevented people from getting their jobs done without a sysadmin's help. I'll bet that the developers all use VM's for stuff like this now

It's very difficult to have a flexible environment that isn't "a mess". It's those hated draconian security policies which prevent some developer (who insists that he's smarter than those idiots in IT) from setting up his own server that then goes unmanaged, doesn't get updated and serves as an attack vector for the exploit du jour. Flexible and safe are essentially opposite concepts in computing. Why do you think Apple went for almost total lockdown with iOS?

 

[BoberFett]

You can see this in film production too...the movies where one dude (or a crack team of people) is running the show vs. the ones that are "design by committee". Same in car design:

The best shops are run by benign IT dictators...all of the information flows through them, but they don't make boneheaded decisions like requiring all developer authorizations to go through a sysadmin. But you still have one guy calling all of the shots so that you don't go into crazytown with a committee. And the bigger you are, the more carefully the decisions have to be made because if you're blasting out a single change & affecting thousands of computers, you have to make sure you're not blowing up somebody's workflow or killing functionality.

I think you mean benevolent dictator?

The mess is what drives me nuts. I demand full traceability...I want every wire, every outlet, every piece of hardware to be labeled & stored in a system that I can reference in the event of an issue or audit. It's not rocket science, but it is a lot of work up-front to make it happen, and so many places just have no idea what they have in inventory or what their full policy set is. Look at how many places have been hacked...the CIA, Target, IRS, Home Depot, you name it. It's really difficult to have a large company & also be crazy-organized in regards to IT.

I'm trying to force my company to get more organized there before we get too big to implement more rigid standards. Right now we've still got more one off solutions than I'd like. We've been in acquisition mode for the past couple years and had some painful growth from the IT side. The business of course wants everything cheap and fast, so good of course is out the window...

Fifteen years ago, IT at this company was fully run by a single guy who had a server sitting next to him in his office. That guy is still my network administrator, but a whole department has grown around him. Change is difficult, but if we don't get organized we could very well find ourselves in a nasty HIPAA situation someday.

 

[Vdubchaos]

Less is more (better)

I find this to be the case with lot of things in life......

Whenever you have more people involved, it simply gets out of control and complex.......and in the end FUBAR. Then you add desperation, job justifications, walls these people put up and other BS.......ahh the all mighty "humans"

 

[kt]

Eh I don't know if its always politics. I need to beat up on a production server so I clone it and beat up on that. Now there is another server that no one else really needs to know about (assuming I can network it if I even need to) to do their jobs correctly. Hell, some virtualization products now make that automatic.

"How many servers do we have?"
"375...wait 376....er...340...."

Also some servers may be setup to do a simple job and they do it so well that you forget they are doing it. Someone asks you for a list and you forget to put it on it because you haven't touched it in 3 years

There's also the aggregation complexity. You have multiple domains managed by other groups who then give control to additional groups down the hierarchy to their own folders. That group then create more folders and gives local groups access to those. Your top level group isn't going to know if all hundred thousand AD accounts are needed and active. Your lower groups may not know or understand what systems are intertwined

And the always dreaded IT guy who managed stuff leaves after X years. "What is this\does this?" "I dunno - Bob managed that."

You just described corporate politics in a much longer way.

 

[nakedfrog]

Is corporate anything ever not a mess?

 

[Brovane]

Yes, corporate IT is always a mess.

I had a conversation earlier this week from a corporate auditor about some sensitive information (customers SS) in plain txt files out on a ISSOFT drive. My name came up because when I consolidate the ISSOFT drive 3-years ago I was the one that moved the data, several TB's of applications. I think the application is no longer used but the client support team which managed the application was outsourced a year ago. I told the auditor I just moved the information from one location to another. Of course the out-sourcer is clueless about the application. So I just repeat these three words, "Not my Problem". Not sure what the corporate auditor is going to do about these txt files, but it is "NMP".

 

[Red Squirrel]

Yep it's often a mess, mostly due to politics. I worked at a hospital the job was more politics than actual IT work.

Worse is when you have people who barely know anything except for buzzwords who make decisions.

"We need to disable hyperthreading on the RSA VPN, because it will make the network faster!"
(I'm not even joking I had an IT manager basically suggest this)

When migrating a SAN.
"Can't you make it copy faster?"

 

[MagnusTheBrewer]

Eh I don't know if its always politics...

And the always dreaded IT guy who managed stuff leaves after X years. "What is this\does this?" "I dunno - Bob managed that."

If that isn't corporate politics/policy, I don't know what is.

 

[Slew Foot]

if you think its bad in the corporate world, just wait until you see the government. The problem comes from the fact that smart people that know what they're doing dont progress into management, but rather ass kissers and backstabbers.

 

[PowerEngineer]

Yes. At least in my personal experience, corporate IT is always a mess.

Is politics the cause? Well, it often is -- at least in the sense that most corporate IT departments want to assert control over every piece of digital equipment throughout the company with a seemingly heightened desire for those systems and/or devices they know nothing about and/or have no expertise to maintain. Once given control, they will start dictating what technologies you can use (to conform to their limited expertise), redesigning your vendor-provided systems (because they know better than the vendor), and downgrading your support to what they decide to provide (your 7/24 critical control system gets to stand in line with desktop PCs in accounting).

For what it's worth, most everyone working in positions similar to mine have the same complaints.

:'(

(feels good to rant a bit...)

 

[sdifox]

If the corp culture sucks, IT has no hope. I was involved in an AD exercise where 5000 accounts exits but the place had 1500 people and no org chart.

So I asked why the list by dept hasn't been forwarded to the dept to vet. The answer I got was it wan't their problem.

So I froze all them extra accounts. They had to all fill out requisition forms to reactivate the ones they need.

Oh and apparently anyone can edit the ERP setup...

 

[IndyColtsFan]

Corporate IT has been taken over by pencil-pushing PMs with no technical knowledge and managers who love buzzwords like ITIL, ITSM, etc, but don't understand the impact it has on their core function - that is, serving the business. I recently jumped back into the corporate IT realm and I think I might have a concussion from all the face palms. I think I'll jump back to consulting as soon as I can.

 

[sdifox]

Corporate IT has been taken over by pencil-pushing PMs with no technical knowledge and managers who love buzzwords like ITIL, ITSM, etc, but don't understand the impact it has on their core function - that is, serving the business. I recently jumped back into the corporate IT realm and I think I might have a concussion from all the face palms. I think I'll jump back to consulting as soon as I can.

Please, consulting has way more facepalm.