Well... that may change somewhat in the future. DirectX has gone through, what, about 1 major version per year so far? DX1 came out with Win95, so it was being written 8-9 years ago. This means everything *has* to be backwards-compatible, since you only get a new line of graphics cards every 12-24 months, and you can't stop supporting old games and game engines every year. This also leads to the phenomenon where all the games on the market are a full generation behind DirectX -- basically every game released this year (except for a few, such as TR:AOD, Max Payne 2, and Halo) has been a DX8.1 game. We're just now getting to see DX9 technology in action. Developers have to write to the lowest common denominator at some level, so with a short generation time you end up having to support much older video cards for a long time or risk problems -- lots of people bought BF1942 only to find it wouldn't run on their TNT2 (which is a DX7 part!)
MS has said they want to stick with DX9 (with slight expansions, such as adding support for PS/VS 3.0 and probably more HLSL stuff) for the forseeable future. Most likely we won't see DX10 until Longhorn ships, if even then. I think they've finally gotten the API to a point where it has all the features people have been asking for, and shader technology is extensible enough that today's (and probably tomorrow's) cards can't really use it to its full potential. It makes little sense to put out a new version now. So we may see more of a "standardization" to DX9 than with some of the previous incarnations, as there is a fairly large jump between 8 and 9, and there will likely be an even larger one between 9 and 10. Comparitively, DX5-7 are very, very similar -- they add some neat features like multitexturing, but few graphics cards at the time were able to take advantage of them.
If pixel/vertex shaders are really the wave of the future, then expect that within a year or two you'll start seeing games that *require* DX9 support in hardware. Companies will provide a DX8 path for a while, but frankly, games that get much more graphically intense than Doom3 and HL2 will most likely not work on anything below a GeForceFX or RADEON 9XXX-series card. There are too many shader effects that you just can't replicate in DX8.