Ok, first do not run 96GB, please. So the 64GB at 3200 sets.
Second, mixing and matching speed can effect timings, as each set may need different amounts to work as well as different voltages.
Third, if stable at lower MHz, you may just have a weaker memory controller on one of the dies. You...