VM keeps crashing

[Red Squirrel]

I have this one VM, it's just CentOS 6.5 with rtorrent and rutorrent, nothing fancy. Yet, it keeps crashing and acting really slow all the time. It takes up to 10 minutes just to load the rutorrent webUI. Eventually it just crashes completely like this:



Is there a log somewhere I can check to find out what's going on?

 

[moddestmike]

I'm assuming you're running XFS by default? Could be related to that maybe? I dont think dumping is enabled by default tho. I think this may help you in the vent it crashes again, this is all provided you havent gotten it fixed by now.

http://lost-and-found-narihiro.blogspot.com/2012/11/centos-6-how-to-enable-dump-core-files.html

 

[Red Squirrel]

For file system? It's EXT4, at least I'm pretty sure... I'd have to double check. I did not do anything special so it's whatever the default is. Whatever EXT is at these days.

Though I'm starting to suspect the VM got hacked via a potential exploit in the torrent software, because while troubleshooting I noticed that my SSH fingerprint changed and I have not changed anything such as the IP or DNS name. I shut it off for now till I can figure things out after Christmas time is over. I need to do a full audit of my environment as if that VM got hacked then it means anything else on the same vlan probably got hacked too... I don't really have anything very involved as far as security goes for each individual system on my network... perhaps I should.

 

[TheRyuu]

What happens if you have the VM use the e1000 controller instead of vmxnet3. vmxnet3 (which I'm assuming you're running based on the panic there) isn't really useful outside of large scale enterprise stuff where you have A LOT going in/out. For anything else the others (e1000/e1000e) may even be faster. The same generally goes for the pvscsi driver as well. The latter is slightly harder to change if you're doing it for a boot drive but still pretty easy. VMWare tools are also optional so don't feel like you need to install them unless you need a feature they provide since most of the useful stuff is integrated into the kernel now.

As far as getting hacked I suppose that's possible. I assume you didn't do any hardening (like running a kernel with grsecurity which you absolutely should be doing with any kind of server setup, I would even say you should definitely be running it on a desktop as well although I can understand if some options there would be tuned towards performance). Even if you don't use RBAC a grsecurity patched kernel is still extremely useful to have (it helps to compile it yourself but you still gain something by using a pre-compiled one from your distro if they have it, just be mindful of the sysctl entries if they have that enabled). I know there are Debian (which of course I will always recommend using) packages like that if you want to go that route to make it pretty easy. It goes without saying you were keeping everything up to date I hope.

If there's nothing that valuable on there the safest thing would probably be to just nuke it if you're that worried about it. You could always hook up the virtual disk to a seperate VM (used only for this) or a live CD or something to get something off of it (or at least to investigate). Then the next time you set it up again (preferably with some hardening!) you could snapshot it before you use it so if anything happens again all you have to do is go back to that.

 

[Red Squirrel]

Never heard of having to recompile the kernel just for security, why can't they just add those features built in? Recompiling a kernel is pretty much a software engineer task not a system administrator task, I don't think too many people compile their own kernels so it seems odd to say that it's a requirement. Either way I don't think it would serve much purpose in this case. If the torrent software does have an exploit that got attacked then the security lies in that app's hands in what the exploit allows the hacker to do (ex: remote code execution). I was more or less keeping stuff up to date but if I recall rtorrent was installed from source and not through yum so it makes it much harder to keep that up to date. TBH never actually attempted to update a package that's installed the ./configure way, what is involved? Not even sure how to uninstall, and guessing I can't just download the latest and reinstall, as it will probably just install two instances.

I regularly run yum update on all my systems though.

Have not had the chance to mess with this too much given it's Christmas time and every non holiday day I've been working, so I just turned it off for now.

No weird things on my other servers such as SSH brute force attempts so I'm thinking I'm fine. I really should run this VM on another vlan though... just need to figure out what to do for storage if I do that, since giving NFS access to my file server would defeat the purpose.

 

[Crusty]

I was compiling my own kernels long before I became a software engineer. It's not hard at all... and very much a sys-admin task. I certainly wouldn't hire a sys-admin that didn't know how to recompile a kernel.

The reason these things aren't compiled in by default are probably for performance reasons.

Most software that use autotools come with a 'make uninstall' command that will purge the binaries it's installed. Then you just need to remove the source directory you extracted to remove it from your system entirely.

 

[TheRyuu]

Never heard of having to recompile the kernel just for security, why can't they just add those features built in? Recompiling a kernel is pretty much a software engineer task not a system administrator task, I don't think too many people compile their own kernels so it seems odd to say that it's a requirement. Either way I don't think it would serve much purpose in this case.

As I said for the less paranoid if you use a debian or rpm based distro I'm sure you can find precompiled versions that make life easy. That being said it's not terribly hard to compile and install your own kernel on debian[1]. I would certainly try a vanilla build before getting adventurous with any patches but that goes beyond the scope of this thread I think.

If the torrent software does have an exploit that got attacked then the security lies in that app's hands in what the exploit allows the hacker to do (ex: remote code execution).

The point is mitigation. You can't expect software to not have any bugs at all and expecting only the application itself to handle security is a naive way of thinking. A grsecurity patched kernel can defeat entire classes of exploits.

I was more or less keeping stuff up to date but if I recall rtorrent was installed from source and not through yum so it makes it much harder to keep that up to date.

If that's the case then you could even build it with the various compiler hardening flags:

-D_FORTIFY_SOURCE=2 -fno-delete-null-pointer-checks -fno-strict-aliasing -fno-strict-overflow -fstack-protector-strong --param=ssp-buffer-size=4

And additionally as a PIE (position independent executable) but that's probably going beyond the scope of what we're talking about here.

The reason these things aren't compiled in by default are probably for performance reasons.

That could be part of it but some (most?) of it definitely is politics. Also it's not like you can't disable the hard performance hitting features (UDREF, memory sanitizing) on places where you would want to (desktops?). Most of the other stuff has a minimal/small performance hit.

[1] http://hardforum.com/showthread.php?t=1616814 (this is kind of old but covers the basics)