Definitely missing out. No two ways to look at it, in my opinion. Without story, even one told without having you read words, it's just pointless killing with no meaningful objective (in many games). Story provides immersion, and different emotion or satisfaction that simply completing the action sequences can't give you. I definitely see it a lot, though. Particularily playing MMO's with friends, whenever we are questing, they are all off running from waypoint to waypoint while I'm still reading the quest details, so I get into the habit of just skipping them myself. Or people asking how to skip cutscenes in virtually every game ever made.
That said, story is not the most important part of a game. A game can have a bad, downright boring, or uninteresting story, and still be great fun to play.
Curious, do you feel the same way when watching a movie if there's no action on the screen, you fast forward, or at least want to fast forward? I ask because my fiance sort of always does this (same with video games, not interested in the story, or at minimum not interested in games that have stories). If there's no action within the first 10 minutes of the film, she asks me, "what's the point of this?" I tell her "for plot and character development." By the end of the movie, she usually likes it, but she doesn't like having to wait for action to start. Funny, she loves to read fiction books, though.