Dunno, exactly. I just know we were running a 5.1 cluster w/o issues for a long time. Then replaced vCenter with a new box running 5.5 and BOOM. 6 Host KPs in two days.
:shrug:
It said it was supported. Like I said. n00b.
Strange. I can reproduce that PSOD with a stand alone ESXi host. I am fairly certain that is a host bug and has nothing to do with vCenter itself and I am a bit doubtful that the upgrade order of vcenter vs hosts would mask that issue. I mean I wouldn't upgrade to vCenter 5.5 until I had put ESXi 5.1u2 on the hosts first via vCenter 5.1.
So here's the breakdown.
Our vCenter environment is not functioning. It spikes to 100% cpu and just stops working. I've tried all the usual support things, nothing is wrong with the database server (and all other database on that server are fine). It's the vcenter procoess vpxd.exe that is at 100%. I should also point out that I've been using vsphere though versions 3-5 and I'm a current VCP holder (going to take my vacp soon).
This is preventing all backups from running as they need vcenter to initiate the backups. I log a ticket with vmware in the morning two days ago. I get a call shortly after and the tech starts to do his thing. I tend to let techs try what they think is right. I've worked phone tech support and I remember those asshole admins who think they know everything, yet had to dial support. After a few hours he is no closer to the problem, but it's his time to go home, so he transfers me to a new tech in another timezone. This tech of course has to start over, because in IT the other guy is obviously an idiot. He suggests we reinstall vcenter. I allow it because I honestly don't know what the hell is wrong. Nothing is solved.
Fast forward another 2 hours and it's this guys time to go home. I get transferred yet again, this time to india (I think) (around the world in one phone call?). This guy quickly realizes he needs help and involves a senior engineer. The senior engineer checks all the things I checked before calling (but I understand) and then suggests we reinstall vcenter. I'm hesitant because we have done this before. He explains that the last guy probably didn't uninstall it properly. I relent and we reinstall vcenter. This causes no relief. He continues working and eventually settles in that the problem is a new ESXi host we added to the cluster a few days ago. We can't remove it, because we can't login to vcenter. He gets another engineer and we go to the database and SQL out the ESXi host. At first CPU usage drops, but then the problem comes right back. At this point it's almost 3am my time, I've been on a call well over 15 hours. I suggest we shelve it until the morning, obviously they need time to work on the issue. They download some logs and dmp files from vcenter and we call it quits. I go get a burger at steak and shake, it was pretty gross and the only food I had to eat after 11am. Bedtime is around 4:30am.
I get into work about 9am the next day (yesterday), the day of my wedding anniversary. vCenter is still down, my boss not terribly happy. I call vmware and they begin their work again around 10am. New tech as the old tech can't be reached. He again explains that everyone else must be an idiot and begins the process a new. He decides it must also be a host issue and we go though a tedious process of rebooting all esxi hosts. To my surprise (and I'm pretty sure also unrelated) vcenter starts to run slightly better even though the cpu usage stays pegged around 80-90%. He then suspects it's a storage issue and brings in a storage expert. They work on the system until about 5pm (My wife brought me lunch so I could eat!) and nothing was resolved. They downloaded the log bundle for vcenter and all hosts and left me so they could analyse the issue.
The result is I still can't not run backups. I still can not manage my VMs though vCenter and I still have no idea what is causing it. I suspect today will be more of the same.
To the support representatives credit, they have been very polite and have tried very hard to fix my problem. I really appreciate their hard work. At this point I could have rebuilt the server. The reason I haven't is because I don't want this problem to pop up again, I want to know the root cause.
In any case, I made that post simply because I needed to vent to someone and my phone was handy.
I'm just a vCenter n00b, and I understand Linux guys hate reinstalling stuff, but if you'd just installed a new vCenter instance and pointed it at the old DB, wouldn't it have:
1) Probably fixed it (assuming you're right and the DB is fine)
2) Taken you a couple hours, tops.
?
Yea, this is 5.5. I've done at least 16 upgrades to 5.5 for various clients in my last job.Have you upgraded to 5.5 yet? I think those support multiple vCenter servers in an HA thingy.
If you haven't upgraded to 5.5, do your hosts first. Upgrading to vCenter 5.5 with a bunch of 5.1 hosts exposed this bug for us. Random hosts started KPing. It was a couple days of hell.
Not a fan of vMWare.
My Hyper-V instances with System Center VMM as the management interface (30 across 5 servers + redundancy boxes) have been rock solid.
Feel your pain though. Have worked on seriously messed up vCenter environments before. Problems with vMWare generally mean a pain in the ass for a while.
To be fair, I've only ever needed to call vmware support twice in the last 10 years (Both because of 5.5 oddly). For the most part the environments I've worked in have been rock solid. My last employer was all hyper-v and it is also pretty nice to use.
Our current environment is 30 hosts and about 425 virtual machines (mostly windows). It's the largest I've ever worked in. The last VMWare environment I worked in was 8 hosts 200 virtual machines (all redhat) and we went from 3.5 all the way to 5.1 without ever having a single instance of downtime or incident with VMWare. I'm going to be doing helping my replacement there do their 5.5 upgrade soon. I don't anticipate any issues.
I would think running ProcMon and perhaps using procexp to suspend the hung task on the old server might at least lead you to whatever the vCenter box is doing to give you an idea of where the hangup is.
As to the comment about HyperV being rock solid and having troubles with VMWare... I'm in a completely different boat. That is not to say I don't like HyperV as a product, but just that I feel that VMWare has a far better layout in terms of networking and storage.
I setup one HyperV environment that was a simple two host configuration, and last I left it was perfectly fine. But I have dealt far more with VMWare and the configuration is much much easier. (I believe the HyperV boxes I had to rebuild 2-3 times because the network configuration of the host kept getting borked and lost on reboot. I finally realized that configuring the vSwitch and then setting a static IP for accessing the host was the way to go.
All in all, seldom do you see issues with VMWare, but they are not immune to a few bugs.
I did a little of that, it really is vxpd.exe using all the CPU. If you disable the vcenter service, the server runs about 8-10%. The probably hasn't recreated itself yet on the new vcenter. So far we are running great. I'm starting to think it might have been database corruption.