First off you singled out a football position which requires arguably the least amount of specialized skill. What about playing as a quarterback? The mental and physical skill set required to play that position is far different than that of a tackle's. What about playing as a wide receiver? One must develop the ability to run routes and have the hands to catch the ball consistently. Rugby players in turn have to develop their own skill sets to become successful. If you are going to compare the two it isn't fair to single out one football position.
Secondly a 190lb safety absolutely has to be able to tackle a 260lb guy. Look at the size of TEs and WRs these days. 220-260lbs isn't out of the ordinary anymore.
Thirdly anyone who thinks that wearing pads makes football less physical is mistaken. As pads have advanced and become more protective players have stepped up the physicality of their game accordingly. Having players wear helmets entirely changes the way you can hit someone. How common is it to see a rugby player run full speed and initiate a head to head hit? While it's illegal to do in the NFL is still happens fairly frequently during the season, and if you think wearing a helmet means it doesn't hurt then you are woefully misinformed. There is growing movement to take helmets out of the game, as a way to slow it down and have less physical hits.
http://online.wsj.com/article/SB10001424052748704402404574527881984299454.html
This whole argument is stupid anyways. These are two completely different sports. It's like saying hockey is harder than soccer. Football has a set of rules and players are chosen to maximize the team's competitiveness under those rules. The same follows for Rugby. Could a professional football team beat a professional rugby team in rugby? No way. Could a professional rugby team beat a professional football team at football? Nope not a chance. Two different sports, many different types of athletes, this is a waste of time.