Well, as they say, practice makes perfect. Downshifting with a near perfect rev match is very rewarding. Here's a nice little vid on heel-toe shifting.
http://www.youtube.com/watch?v=cPj9XXW25GA&feature=related
I've driven a car with a broken clutch cable once. It was the first car I ever owned and I had owned it for a year or so. I drove to work and when I came out the clutch went to the floor and did nothing so rather than call a tow truck I started it in gear to get going and then rev matched to shift. Up shifting was easy, just get the engine speed matching the speed of the vehicle and ease the shift lever out of gear and then let the revs fall while pushing it into the next gear. No clutch wear there! Of course the starter motor takes a lot of abuse as do the gears if you get the engine speed wrong.
Honestly, you are worrying about something that is so minor it is almost insignificant. The clutch takes far more abuse from starting out than it does from downshifting if you get the revs close to where they should be. I'm pretty good at this and wouldn't hesitate to do it in any manual transmission car I've driven. I also do this all the time on my motorcycle, a vehicle that you really want to have in the correct gear for the corner and which is much more easily upset by sloppy downshifting.