What version of Memtest86 are you using? I experienced this very same rebooting thing when I was using Memtest86, latest version 3.5, with 2 x 2GB of G. Skill, on an Asus P5Q-Pro. Searching the Internet, I found a Memtest86 Wiki page that describes an incompatibility with 4GB of Ram.
I then...