Noise is noise, it shouldn't matter what size one is than the other.
Now a bigger fan should put out more cfm at a slower fan speed so it should mean that the bigger fans while putting out the same amount of air as a smaller fan moving alot faster, would be alot quieter.
Ie. Two 50cfm fans, one 80mm, and one 120mm. The 80mm fan should be humming along alot faster than the 120mm fan and therefore the 120mm fan should be alot quieter. This is just a correlation and not an actual comparison of any "real" fan.