All these things you say are true.
I think there has been a shift in the userbase of modern games. When i was young, videogames were for nerds. Never mind the D&D rpgs and NWN and the gold box SSI games and Fallout, just playing with a machine was for nerds. People couldn't wrap their head around "load * ,8,1" but would rather play football.
The people who designed games were nerds, the people who played them were nerds.
You'd have a few jocks playing coin-op arcade on sunday, but it was occasional, and they were more into pinball than videogames.
Games of this era required some weird mental gymnastics to play; there were no tutorial, generally required external per-case knowledge (like most rpgs)
Then Halo changed everything. Halo, COD, MOH, these games appealed to the jock and to a ton of casuals that: did not own a desktop w/ gpu, didnt want to read or interpret stats, only wanted an hour of fun, and had no knowledge of non-pc game culture; and to top it off, they were *easy*.
The games that i played were hard. Boy were they hard. Each game was a challenge, to figure out how to beat it. NOT all games i bought, did i manage to beat.
No such thing for Halo.
Because of this new userbase, who burns through content (because that content cannot try to *stop* the player), games were now made differently than before. No more of 3 friends sitting in front of a pc trying to solve a Day of The Tentacle riddle for 2 days, but smooth flowing "immersive" gameplay that does not try to punish you, doesnt force you to learn any new skill. Because some money guy paid a team of asians to design that content, and by god the players need to see it.
I speak with people at work, and nobody can name one hardcore game they play. Maybe Dark Souls at best, but mostly it's all casual, easy, for kids games that anyone can utterly beat.
I still have not beaten Xaero in Q3A on Ultraviolence, much less Nightmare.
So the challenge aspect was removed because to casuals, difficult isnt fun.
Which is BS, because i am PROUD of my lap times in WipeOut 2097, but thats another thing casuals will never experience.
Changing the typology of content also changed the associated costs, making game production an expensive affair, governed by money people, not gamers.
One other thing is that the wow factor is no longer there. In 1998, (?) GT2 came out and it was incredible. Ive lived through the b&w tv era and videogames were alien technology; but tech has stopped advancing so fast, and so games aren't as "amazing" as they used to be. Thats fairly reasonable, but it does help compound the problem.
And because of the money involved, there's no more of "just a bunch of guys" making an amazing game. Rebels, who didnt care if you liked it or if you ragequit. Developers who hid things, who made it difficult on purpose for the purpose of fun, not for the purpose of frustration. And who could add any crazy subject because there was no censorship.
Like many cool things, games used to be a niche product, then they had success. Too much success, and the morons started buying them. And morons, they dont know a good thing, they just buy crap.