http://www.talula.demon.co.uk/games.html is an interesting article with an informal study of the economics of open source gaming. The conclusions seem quite plausible. It'll be a cold day in hell before we see much in the way of real open source gaming(yeah, yeah, Battle for Wesnoth is kinda addictive and ID's back catalogue is more or less available, if you can snarf the .WADs but generally speaking the point is clear). Engine and library suppport, though, are logical areas of strength, and hopefully will allow the "modder" types to become game creators much more easily. This would be to the benefit of all involved and isn't too implausible. Imagine the following hypothetical situations: With a modicum of cleanup and some scripting language hooks, and similar bits and pieces, the open source Quake 3 engine becomes generally usable for any mildly to moderately talented developer. This implies that any game company, no matter how indie, has neither the need, nor the excuse, to use an engine that is worse than Q3 assuming their planned game is close enough to the code. This is good for them; because they don't need to cobble together some in-house monstrosity or pay huge licencing fees to get a real engine, and they can spend the money on story guys, artists, and the like.
This is good for gamers; because they have to deal with fewer horrific engine bugs, have a wider selection of independent game production houses to choose from(who spend more on making games fun, and less on making their lousy engines not crash too much), and; because the underlying engine is open, will be effectively platform independent. This would help deal with GreyMittens' point. Sure, some OSS users are basically just stingy, and absolutely won't pay for anything. Such people would just pirate the closed source games anyway, as they already do. OSS people who don't have any problems with, say, buying CDs of artists they like, would presumably have a similarly positive attitude towards buying the work of game designers they liked, and since the design would pretty much just be the art, story, meshes, scripts, video, sounds, music, etc. that makes the given game what it is, on top of an open engine, they wouldn't have to install any closed code. I know of a fair number of people who won't load unfree code onto their systems, and I am among them; but I don't know anybody who refuses to look at, or listen too, video and songs that are licenced under anything but a permissive Creative Commons licence. It is very important that the programs that we use to handle the media are open, for technical and ethical reasons; but as long as the media files do not restrict our fair use of such tools, it is hardly necessary that those files be free as in beer.
Obviously, the ideal, general purpose, set of open engines and libraries is still in the future(if anywhere); but if such a framework were to exist it would effectively make the distribution of, and market for, games look a lot like the one for music, albeit with a diffirent creative class. This does mean that there would be the abuses of the gaming equivalent of the RIAA, SonyBMG, etc. but it would also make possible the use of game analogs to the various alternative relationships(which generally seem to work pretty well) between the creative producers and the fans.