According to my observations, if a Core is unstable due to over over-clocking it will do one of two things, or both things separately, namely either showing up artifacts in your games and/or other application types, or will simply hang/freeze the whole system, and nothing will respond anymore, not even Ctrl+Alt+Del, forcing one to hard-reset their system. Artifacts coming from an unstable Core are also visually easily recognizable. They are mostly little dots filling either the whole screen (severe instability) or small portions of random textures/polygons during game-play. They can be white, green, red or yellow (usually white, sometimes red or yellow, and very rarely if ever green, but possible). A popular example of Core-related artifacts can be seen using ATiTool's Artifact scanning tool. It will usually show yellow dots if unstable (shown on-top of the rotating 3D cube).
As for the Memory, if unstable, will usually cause various screen corruption, distortion, strange lines appearing in games as well as in mere Desktop activities, or even during early Windows booting (loading screen for example). In-game, when severe instability is the case, it will distort the very polygons themselves, will deform the shapes of certain (if not all) surfaces on-screen, will change or shift colors, sometimes making the game look 16-Bit colored rather than 32-Bit (just a visual impression). The famous "huge triangles" symptoms are caused by Memory over over-clocking (instability). As far as I know Memory instability, even severe, cannot actually crash or hang/freeze a whole system (Windows not responding), but can crash an application back to the Desktop (Core instability can do that as well however).
And don't quote me on this, but, again as far as I know, severely "damaging" (just pushing too far and too often) a GPU Memory can result in permanent artifacts much faster than a Core would suffer permanent issues. In other words it would be more difficult (again, not proven by me, only observations) to permanently damage a Core than a GPU's Memory, although permanently damaging a Core is still possible but is done so more via bad over-voltage selections and manipulations than anything else, since a Core will just hang a system if it lacks "juice".
Hope it helps.