Getting on to some of your comments about dexterity-based challenges, there are two answers to this: 1. one of the defining qualities of most games is that they present a challenge to a specific ability or class of abilities that humans have, or can learn to have. To say that an action game shouldn't require too much dexterity is like saying that a computer chess game shouldn't be so good at chess that some players can't beat it.
But like I said, that is only one
of the defining qualities of games, and one of the issues with it is that the rest of the game (the art, the storytelling, the music, and the other, non-challenging aspects of play) are off-limits to anyone that can't get past the challenge of it. This is somewhat of a problem if you ever wanted your mom or girlfriend, for example, to appreciate a game like Shadow of the Colossus, which has a lot to offer artistically, but only people that can actually face up to the platforming and exploration challenges can experience them.
Part of this is the raw "game-playing talents" that gamers build up over time, and which games are designed to cater to. Another part is just insider knowledge, the kinds of tricks and tactics that gamers are familiar with and that game designers assume are part of your arsenal when playing their game. (Would any non-gamer think to break open a crate to get a health refill?) But this is the same problem that "challenging art" has in other media -- movies or books that require close attention and perhaps also a knowledge of the history of literature and movies to appreciate. Should arty movies be more accessible, or should people that don't like/get them seek out movies they do like?
But the other answer: 2. that games should aspire to having multiple solutions to problems, and those solutions should be diverse enough that different skillsets are required. (see "difficulty tuning"
In addition, and this seems to be a pretty common approach to games nowadays, is that games should be relatively easy to beat in the narrow sense (complete the story, get a feeling of closure), but with plenty of additional, harder challenges that players can seek out as they desire them. Recent Mario games generally require about 50% completion to get to the final boss, for example. Many other games have things like "achievements" that provide extra challenges.
But while this is progress, the spectrum of abilities is huge, and there are casual gamers that can't get more than 10% into a Mario game, and probably non-gamers that can't get past the first level -- though part of this may depend on their willingness to keep trying, and part of that may be based on social expectation that they shouldn't bother because they're not Gamers.
So I don't know... I see some value in both sides. First, accesibility is good and should be a consideration when building games. Second, not all art needs to be available to all audiences, and it might be better that a diverse selection of art is made, so that each person can get a good supply of something they like.