Eh, I guess I can't say since I'm not a programmer, it just seems odd that it would be THAT much more difficult. The only thing 'different' is the input. Everything else is already done in a 3D world normally. The headgear is simply 'mouse look' in most cases and your body moves independent of that (ala arrow or wasd). Is programming for the hardware really that difficult? Most of it is handled in premade modules (at least that people are using). Practically everything else is typical for 3D games. There hasn't been a single thing I've done that hasn't been done on a computer monitor at some point.
I get that one area that is difficult is movement and not making people sick, thus all the stand in place items.
The prices aren't just a function of how difficult the game is to make. The developer has to make money and the VR market is tiny. If I make a great VR game, it's going to sell like shit no matter what (because there's like half a million headsets out there tops), and the people who are going to buy the game are going to buy it no matter what (because there's so little content), so why would I not charge $40? It also gives a lot of headroom for steam sales, which is where I expect a large percentage of purchases happen these days.
From what I've seen, one of the hardest things in VR development is the newness. Normal games have had 30+ years of generally accepted "rules" for how things work. Many of those things don't apply in VR and developers have to spend a lot of time creating the "rules" of VR by trying different things to see what works and what doesn't.
The headset is not just mouse look. A mouse only provides rotation information. You can move the headset, aka the camera, in any direction and do things that you don't have to worry about in normal games. For example in a normal game a 2 foot fence is enough to be a barrier but in VR I can physically move my head and go beyond it. Developers have already figured out ways to mitigate this (like fading the screen to black when you move your head someplace it's not supposed to be), but that requires effort.
Probably even more of a challenge than the headset itself is the motion controllers. In a normal game they can limit what you're able to interact with by simply not making it interactive. In VR this isn't really acceptable because when I see a coffee mug on a table I want to pick it up and throw it into the trash can. Also the physical interactions of picking up game objects with the controller and having that feel natural is not easy. Additionally, the ways you can interact with things in a normal game are extremely limited. Just think of the differences in sword combat between a normal game and a VR game. They would have almost nothing in common in terms of how you design the enemies, their AI, their reactions to attacks, the parry/blocking system, etc.
Performance is also more critical in VR than normal games. Very few normal games run at 90fps all the time, and even with VR runtime aids like ATW/ASW, good performance is important and difficult to ensure all the time.
Programming is just one part of game development. Art, design, sound, and testing are all significant parts of development and all have unique challenges in VR. If making VR games was no different than making regular ones, there would be a lot more normal games quickly ported to VR for easy cash in a content starved market. The fact that it's taking so long for Doom VR and FO4 VR to materialize should be an indication that it's not an easy task.