Check out ebay for xeons v4 (2696, 2697, 2698, 2699 v4). Some are quite cheap. Strong cores, lots of cache, high bandwidth, reasonable price ->hard to beat. you can get 18-22 core cpu for good 500$, sometimes even cheaper.
If you are fine with OpenCl, you can also try R9 280/290x or get a used...