Only once have I exceeded 4GB of memory on my machine. My wife scanned an 8x10 picture at 2400dpi. The resulting image was H U G E (19,200 x 24,000... or 460 megapixels.) I opened it in the Gimp to resize it to something more normal sized and it took like half an hour, because it was hitting the disk for virtual memory.
Other than that, for regular use (10+ tab web browsers, Excel, Powerpoint, etc... all open in the background while gaming and such like that) 4GB is plenty right now.
On the other hand memory is cheap right now and people are never going to want to use less of it, so if the price is right, it's not bad to have more.