Yeah I can understand this point of view, but it could be applied to almost any label. If I was at a party and said, "Hey have you met my Maori friend, Rangi?" then there's absolutely nothing offensive about it. But if my car got broken into and I said, "Fucking Maoris!" then that's offensive. There is absolutely nothing offensive about the word Maori though. I think Oriental is the same - it's not the word that's offensive, it's the cockturtle using it in an offensive way.
Yes, that would be true for the most part. However, the history of the word is also important, as is the meaning/connotation of the word.
Look at it this way, China has always called itself the center of the world. The rest of the world is either North, South, East, or Westerners.
When you center the world around China on a map, North Americans become Easterners. So are Canadians and South Americans. So if you lived in China, I think you may be offended to be referred to as an Easterner or the Chinese equivalent of "Oriental" with no regard for your American identity - you'd be no different than a Mexican or Canadian in their eyes. But of course, this would depend hypothethically on the situation and how someone uses the word.
That's the other reason that historical usage aside, knowing the meaning of the word can create offense in the eyes of some, because of overbearing generalizations and ethnocentric world views. It gets complicated when you educate yourself ya know.
Ornamentals would know. ^_^