He talks about several things there. First, about CPU speeds going down? The only reason for clock speeds to decrease is that there's no money in high-clocked CPUs these days. All the money is going into mobile devices, which need to run off a battery and use little power. So dropping clock speeds is very helpful to do that. If you gave Intel a few hundred million, they could surely make a 200-300W+ 5GHz+ Skylake...within a few years, after building a completely new process designed for high clock speeds and higher power consumption.
The second thing he talks about is fewer FPUs. We tried that with Bulldozer. It was a disaster for CPUs. But here he's not talking about CPUs; he wants to build an "IPU" device for machine learning. If it's a neural net he's aiming for, he hasn't gone far enough. What you really want is memory with extra, dedicated compute components linking memory cells. Taking this to the extreme, I think memristors linked by analog (!) compute circuitry may be the best way to implement a neural net.