I have found what many others have already discovered and its probably correlated at least to the low FPS, especially since changing graphics settings has little effect. The game chooses a single core and maxes it the hell out at 100%. People suggested disabling core 0, which is usually the one maxed out, but the game just picks another one and maxes that one out instead. The game is behaving almost like a single threaded game, so unless you have an OC'd 6700k like the guy in that performance review video, you are going to get even worse dips than he did. Even an OC'd 6700k falls below 60fps at times. Single core bungelope this game is.
Quoted from dsogaming.com
Forza Horizon 3 sadly suffers from single-core CPU issues. As we can see, one of our CPU cores was maxed out during our playthrough, while all the other five cores were below 70-80% usage. As a result of that, we were CPU-bound in pretty much the entire game.
In order to find out how the game scales on various CPUs, we simulated a dual-core and a quad-core CPU. Unsurprisingly, there were no performance differences between our simulated quad-core and our hexa-core systems. On the other hand, the game was completely unplayable on our simulated dual-core system (while it managed to run the game at 30fps, there were severe stutters that made it unplayable). And given the fact that the game relies heavily on a single CPU core, we strongly suggest disabling Hyper Threading.
What’s really interesting here is our GPU usage. As we can see in the following images, our PC system was unable to offer an enjoyable experience due Forza Horizon 3’s single-core CPU issues. Our GTX980Ti was used at 50-60%, suggesting that our CPU was bottlenecking the GPU. It’s pretty clear that Forza Horizon 3’s multi-threading capabilities are awful, something that definitely surprised us. Oh, and in case you’re wondering, we were unable to run Forza Horizon 3 with constant 60fps at 1080p, even on Medium settings and without MSAA (while driving through Surfer’s Paradise).