So I finally took everything apart today as the noise was just driving me nuts. After I removed the fan, I noticed the damper going out to the duct was open. Upon closer inspection, it looked like Mr. Contractor installed the damper upside down so that it was open by default and with air rushing past it, it just rattled back and forth. Grrr. Took everything out, installed damper correct way and viola, no noise.