Some people like to push everything at once (cpu & memory) while oc'ing while I prefer to treat each 'area' separately. The first thing I do is push the CPU using fixed settings (like locking Vcore at a fixed point) while the memory 'follows' at just under spec settings. With this in mind and looking at your settings (compared to mine), the first thing I would do is drop your DRAM speed (currently 805/1610) down a notch, set a fixed Vcore (I'm using 1.296, which gives me about 1.285 in Windows) and resume testing. Leave your memory timings on auto for this initial testing and note the values in BIOS (all of them, as I show in this screenshot of my BIOS):
Some BIOS will not show the set values until after a reboot so do that before noting them if your BIOS is like that. My system kept trying to set lower timings (than spec) for the speed they were at (1440), which were not stable, so I ended up pulling the data for my memory timing from BIOS (by setting each speed manually, rebooting and returning to BIOS to note the values). My default XMP/1600 timings are: 9-9-9-27-128-12-6-5-6-20-1T and they are currently set as in the above screenshot. I left some settings on auto because they were the same as the spec settings so no need to manually set them. I tested the system at 1440 with the XMP/1600 timings set and the system was stable. After noting the timing differences between JEDEC spec #4 (685/1370) and the XMP/1600 values, since I was at 720/1440 I set my initial timings to: 9-9-9-27-114-10-6-5-6-20-1T. While it passed all tests (Prime95, IBT and various other benchmarking tools), the system hung at the shutdown screen. I rebooted and then set the timings to: 9-9-9-27-117-10-6-5-6-20-1T and tried again. Same thing, passes tests but hangs at shutdown. So I set my current timings: 9-9-9-27-120-10-6-5-6-20-1T and everything is working fine.
I am telling you all of this because before I could start working on the memory as detailed above, I first set it to below spec (a safe and known stable speed) and pushed the CPU up to where I am comfortable with my current cooling (have hit a max temp of 83*C running IBT on Very High with a ThermalTake Frio cooler). Since my memory speed options are (excuse the blur!):
My next step up in speed would be to push it to 1800, which is over spec for this memory. Since this computer is also a work system I would rather not push it that high. Because of that, my current options to push the memory higher would have to be upping the CPU speed. With my temps hitting 83*C, I would need to replace my cooler with a better one to do that (which I am considering if I can sell off my Frio).
In your case, I would do as I stated above: lock Vcore to around 1.28, set memory speed/timings one notch below top speed (2:8 ratio) and start pushing the BCLK up.
Then we take it from there! I hope this helps.