My XDm 40 stays loaded with 16 rounds in the mag, one in the chamber, cocked (of course) and holstered with an extra mag next to it 24/7 at home. My USP 40 is also loaded up to max (13 rounds ni the mag, one in the chamber) hammer down and holstered. (Been tossing between engaging the safety and not based on mood and after reading up on how hard it really is to truly make swiping the safety off an instinctual action.) Spare mag also loaded and carried with the USP everywhere I go.
Never an issue with any of the mags, and I agree with a lot of the other posts in that a gun is worthless if not loaded. Although I don't keep my 10/22 loaded, nor my AR15 (but I do keep a few mags loaded for the AR.)
One thing to really strive for is to have plenty of spare mags, and rotate/test fire them routinely. I only have three mags for my USP due to supply issues right now, but have a dozen for the XDm and even more for the AR. Label them, and cycle through them if you want to be extra safe, but these days, any good factory mag is probably gonna outlast you. Or at least outlast the time before you decide on using a different boomstick for defense duty =)