Your graphics card should be able to auto detect and transmit the full 2560x1600 resolution to your monitor.
Is there an option on the AMD Catalyst settings to change the resolution to 2560x1600 or on windows control panels in display settings to change the output rsolution to 2560x1600?
If the options above is not available.
1. How long is the mini-DisplayPort to DisplayPort cable?
2. If it works stable at resolutions at 1280x800 then that DisplayPort cable may not be made to transmit 2560x1600.
My suggestion is to try another Mini-DisplayPort to DisplayPort cable (6ft).
There are alot of DisplayPort cables out in the market that cannot transmit beyond 1920x1200 and many more that are not VESA compliant.
DisplayPort 1.1 cables with copper at 32AWG must be less than 2 Meters (6.5ft) long to transmit full 2560x1660. Also DisplayPort cables that are not VESA Compliant are using one of the cable's pin that makes the monitor send power back to your card and may damage your hardware - VESA Compliant DP cables do not use/activate this pin.
For more information regarding DisplayPort technologies visit:
http://www.displayport.org/