Lived there as a kid and have gone there frequently all my life. I've never been a victim of crime there and I've been pretty much all over it.
If you're at Union Square as someone else has said don't walk around the area west of there. Its not that is ridiculously dangerous, just dirty and depressing. Chinatown is interesting for a quick walk through on your way to North Beach which is one of my favorite areas to take visitors. Catch a ferry to Alcatraz from there - its well worth the couple of hours.
I don't really care for The Haight. Too many beggars, although it is cleaner than it used to be. There's really nothing there unless you just want to see where the 60s counter-culture happened. The Castro is interesting for a quick walk around. Its not the solidly gay village it was in the 70s and 80s tough - Techies are slowly taking it over. Speaking of them Noe Valley just south of there is their turf - nice gentrified area full of restaurants and shops.
If you get tired of all that Golden Gate Park is really nice. The California Academy of Sciences, de Young Museum and Japanese Tea Garden are all fairly close together and its a quiet place to unwind a bit.
Its a great city - just be prepared to the horrendous traffic/parking situation, lots of bums and street crazies and dog shit (and sometimes worse) on the sidewalks. High density living makes all that more noticeable. Oh, and bring a light jacket - its usually in the high 60s, often overcast and very breezy in the summer.