How can I tell if it's genetic? I've talked it over with my wife - I'm getting cosmetic surgery after I'm at my goal. I'm was 100 pounds overweight, I'm thinking the loose skin alone would justify it.
Cosmetic surgery is not the solution until you have dieted down to 10% bodyfat or less. I mean a 'real' 10% too, not what most people say is 10% (but is really like 16%).
I, too, carry chest fat naturally. And, if you really study the anatomy of it, it isn't just chest fat, it is armpit fat, which pushes towards the chest when your arms are down, thus shoving fat to the chest area, increasing it further.
I can assure you, if there isn't some gland growth (hard lumps - don't confuse them with your RIBS) then it is just fat and you will be able to get rid of it be dropping bodyfat levels.
Now, we can't chose the order in which fat comes off, and we can't choose which order it comes on, but despite what you read, it does not accumulate the same way always. Meaning, you could be 20%, drop to 10% and then go back up to 20% and depending on the foods, exercise and diet, your body may look far different when comparing both at their 20% levels.
Best advice I can give anyone is to buy calipers. That is the ONLY way to know if you are dropping fat from week to week as often visually changes blur together over time.