NFS randomly hiccuped, now everything is screwed up

[Red Squirrel]

On one of my machines NFS randomly took a hit, now everything is screwed up. VirtualBox will not open at all, and if I try to do anything having to do with the NFS shares such as try to cd to the folder, it just locks up that SSH session and I can't do anything.

I tried to restart NFS from another SSH session but it just fails. How to I get out of this mess?

[root@borg network]# cd vms_lun1
^C^C^C^C^C^C^C


^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C

^C
^C
^C

^C
^C
^C
^C^C
^C

It just stays stuck like that. I can ctrl+c all I want but nothing happens.

I'm guessing this is probably what's stopping virtualbox from opening, since it tries to read a folder in there or something.

I need a way to forcibly unmount everything, maybe that will help. umount -f never works though, it still say resource busy.

 

[theevilsharpie]

If a lazy unmount doesn't work, the only other way I know of to clear a hung NFS session is to reboot.

 

[Red Squirrel]

I just found out about -l command when searching stuff on google and it seems to be more powerful than -f and it did work. -f was not working. I was able to unmount, but if I try to remount it just hangs. Is there any logs somewhere I can check? NFS does not seem to have much logging, if any other than dmesg stuff once in a while.

In dmesg the only thing I have is this: (on client)

nfsacl: server isengard.loc not responding, timed out
nfs: server isengard.loc not responding, timed out
nfsd: last server has exited, flushing export cache
NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
NFSD: starting 90-second grace period

I tried restarting NFS on the client but no go. One of the services also fails when I do a restart. I really don't want to have to restart NFS on the server as that will screw up all the other machines that are working fine now.

[root@borg network]# service nfs stop
Shutting down NFS mountd:                                  [  OK  ]
Shutting down NFS daemon:                                  [  OK  ]
Shutting down NFS quotas:                                  [  OK  ]
Shutting down NFS services:                                [FAILED]
[root@borg network]#

 

[Red Squirrel]

Well I gave in and just rebooted the server and now everything works. It's always a battle to get that server back up from a reboot though as randomly my config files like resolv.conf get overwritten by stuff. Super annoying. I can't wait to get stuff migrated off that box, FC9 is just ancient and I need to retire it. All the shares are working now though.

 

[dave_the_nerd]

resolv.conf shouldn't get rewritten unless you're trying to use static DNS with a DHCP server.

That's weird.

 

[theevilsharpie]

resolv.conf can be overwritten by NetworkManager, as well as the configuration scripts that various distros use. Unless you're using a positively ancient distro, there should be no reason to modify resolv.conf by hand.

 

[Red Squirrel]

resolv.conf can be overwritten by NetworkManager, as well as the configuration scripts that various distros use. Unless you're using a positively ancient distro, there should be no reason to modify resolv.conf by hand.

How else am I suppose to change DNS? I usually uninstall the networkmanager crap, it does nothing but cause grief by ignoring or overwriting config files. Seems even without it being there some other process likes to randomly overwrite config files though. I find the newer distros are actually worse for this.

 

[Crusty]

Well, you're supposed to use networkmanager.

 

[Red Squirrel]

How does one "use" networkmanager, it's just there, and it just overwrites stuff, there's nothing about it that can be "used". It's garbage unless you're running a workstation and using the GUI. Everything I read online when I want to do something like change an IP or do bonding or any other network config tells to edit files. NM just overwrites these files. Though even without NM there seems to be certain distros that have other processes that overwrite stuff. Pain in the ass. Config files should never be overwritten by anything. It defeats the whole purpose of having config files.

 

[MrTransistorm]

On my home server (Ubuntu Server 12.04 LTS) resolv.conf is automatically generated. In fact there is a comment at the top of the file that warns against editing it manually. Instead, my DNS server settings are in the /etc/network/interfaces file. In fact the man page (on Debian-based distros) for resolvconf states that this where to store those settings. For example:

# OpenDNS nameservers
dns-nameservers        208.67.222.222 208.67.220.220

Your distro may be different.

I don't use NetworkManager, but here is some info about its DNS setup. Note that by default it won't configure any interfaces that are defined in /etc/network/interfaces.

 

[Crusty]

http://blogs.gnome.org/dcbw/2014/06/20/well-build-a-dream-house-of-net/