Adding to what BrightCandle said: some apps are already coded to take advantage of GPU when available. For example, I know some Adobe apps and other media apps, do, as do some engineering apps.
OSX provides some frameworks/libraries that make use of the GPU, and even provide a way to have some code run on the main CPU AND the GPU. Even so, there is a lot that doesn't yield easily to being run on the GPU. Apple 's latest MacPro is a GPU monster, with one GPU pretty much dedicated to computational tasks. To show it off, they released a new version of Final Cut Pro, their video editing app. The GPU provides a huge speedup for some operation, very little for others.
I'm less familiar with the windows side, but I expect similar issues exist.
Some code is easy/worthwhile to run in parallel, but a lot isn't. Of the code that yields to parallelization, some of it yields to GPUs, but pretty much all of it is going to be easier to do on a multicore CPU.