I've have a case on mine about 50% of the time. So far I haven't gotten any scratches in the few months I've had the Note 4.
Is yours scratching on the silver chamfered edge or on the painted sides? I thought about getting the white Note 4, but figured scratches on that white paint would be a lot more visible than on the dark gray model.