3U is 5.25" or about 133 mm externally, less internally, and then subtract the motherboard offset. That rules out any tower HSF with 120mm fan. I just got a Xigmatek tower with a 92mm fan and it measures 135mm tall, so I'd also guess that few, if any, towers with 92mm fans will fit. Going smaller than a 92mm fan isn't going to be particularly quiet.
That mostly leaves HSFs with horizontal, down-blowing (or angled) fans. You can't use one that just fits, because it would be up against the top of the case and you'd have poor airflow. So you may need something relatively low profile. I'd try to keep it under 100mm. A Noctua NH-L12 would probably work well, but I haven't used one.