I dont know your knowledge level on computer architecture, but Ill explaine it as simple as possible.
It really has nothing to do with the amount of memory the card has, other than the amount of memory usually defines which generation of hardware the video card has.
See, older cards were optimized to run older games. There were standards that most older games were programmed by and most cards were made to do those things best.
Every generation things are done newer better and DIFFERENTLY. So newer video cards must be optimized to do the new things best.
Due to cost cutting measures or incompatibilities or other reasons, older optimizations are eliminated from newer generation hardware in faver of new optimizations. Because the older optimizations are gone, older games which tool advantage of them run slower. Newer games which can use the current set of optimizations run faster. Similarly, older cards which lack newer optimizations run newer games slower.
Easy?