Boy, there are some people here who clearly have not/don't play any sports.
I played two division one sports in college, so perhaps I am biased towards them, but first a few observations...
Horse-riding is not a sport. Horse racing is not a sport. Anything involving a live animal is not a sport. Does it take skill to ride a horse? Yes. Does that make it a sport? No.
Golf, as much as some people may like it, is not a sport. It is a contest of skill, like darts or bowling, but when people have other people that carry their clubs around, and take carts from place to place...it isn't a sport. Yes, I know on the PGA tour they don't use carts, (their opposition to Casey Martin using a cart on tour was simply to try and make it look more difficult than it is to play, and they looked like a bunch of A-holes for their stance IMO BTW)..again, the skill and practice required to be good are certainly factors, but I still say it isn't really a sport.
Soccer, as much as Rome hates it, is a legit sport, but the athletic skills necessary to play aren't all that demanding or difficult to improve.
Hockey certainly rates highly on the list of sports that are difficult to master, but being a really good goalie, or defenseman, is different than being a really good scorer, which is the most difficult part to master.
Football has too many positions that aren't really difficult to play, though you could argue that "mastering" the QB position is right up there in difficultly level. I was a HS QB, and though I considered myself good, I certainly wouldn't say I had mastered the position.
I would have to say that two parts of baseball are the most difficult to master. Hitting and pitching. I know everyone talks about the difficultly in hitting, which is mostly true, but I would like to add that "mastering" pitching is extremely difficult as well. You can't just say "all you have to do is throw hard", because there are plenty of guys who throw in the 90's that stink, or never even get to the majors.