I don't buy many video cards, my current is a TNT VANTA of all things, but I have used enough to compile the following list.
Worst:
Every ATI card I have ever owned or used. They have been a consistent pain since 1990.
Why? In the Windows 3.0-3.1 days, the drivers caused random and frequent int13 breakpoints in code whenever Microsoft debuggers were running.
My last ATI card, a circa December 1999 Rage Mobility Pro, was immediately obsolete due to ATI's incredibly crappy driver update policy. If its not the latest greatest, you are out of luck. It doesn't matter if they are still selling the chipset.
Honorable mention: Diamond cards for the same driver update reasons.
Hats off to nVidia, I will NEVER EVER EVER buy any other brand thanks to their driver support. As a programmer who updates OSes faster than hardware, a video card that works no matter what machine or OS I throw at it is more important than faster frame rates.