Double sided, double-layered DVDs hold 18GB but that is still far less than your 20GB/platter HD. The DVD needs 4 surfaces to get 18GB but the HD does it only in 2 which is roughly twice the density.
I'm no storage expert but I feel the advantage to the HD is the enclosed environment allows for very close bit packing which translates into an incredible amount of bits/sq in. Optical drives have to read and write under much more variable circumstances and distance from head to media.
I still haven't seen any of these mythical double-sided, double-layer DVDs despite the spec being around since 97. The fact is that double-layer is easier said than done, and just like double-sided DVDs was only included in spec to appease a certain faction so they could actually finalize it.
And you're dead on the variability aspect, in a sealed HD case you can push tolerances to the max, but with a disc you have to account for dust, scratches, variations in spin rate, laser alignment, etc. Not to mention that a very large portion of data on DVD and CD are devoted to error correction and framing.
finally, HD mfg'ers can push ahead with new technology as soon as it's perfected, on the other hand optical discs are stuck in standards that appear to change about every 15 years.