The card was designed as a 56 ROP unit and the attendant limitation that it works best with 3.5GB of VRAM.
But nVidia decided to put 4GB on it AND WRITE DRIVERS FOR IT to let it "use" 4GB.
This is very true. The card is has in fact 3.5GB of VRAM and if the remaining 512MB are getting any use, performance will drop hard. It takes ~16ms of total GPU render time to deliver 60FPS, during that time GTX970 can read/write ~0.5GB from that 512MB segment at the cost of giving up ALL bandwith to the fast 3.5GB segment. So there is simply no way the card can deliver decent framerate if those 512MB are actively used (as happen in texture mods for games and will continue to happen with console ports and other games in the future).