OK, so as far as the premise of the OP, I'm going to approach this by generalizing based on human racial differences. Despite our wanting to be something more, people are just animals, and conditions in different regions around the world created racial groups that are very obviously different from one another. This might "trigger" someone, because politically correct culture likes to shy away from admitting race is a real thing, but it is. Just like male and female humans have differences outside of just sexual organs. Sometimes it feels like I'm talking to 5 year olds with this stuff, you know?
Anyway, if we are admitting race is real, it just takes a quick glance to see that certain races have different qualities. Different levels of melanin in the skin, some differences in body type, to where people of one region might be taller or shorter, have slightly different bone densities, small differences in fast-twitch and slow-twitch muscle fibers, slight differences in things like tendons, and on and on and on. I'm not going to want to open a conversation on what groups might have better or worse qualities, I'm just pointing out they're not the same here.
OK, so if we look at human aggressiveness, I think the biggest factor is testosterone levels. That's obviously one of the big differences between men and women, and explains part of the differences between the physical aggressiveness of men and women. Out of the human racial groups where men have higher testosterone levels, on average, those men are usually larger as well, in physical size. Racial groups with smaller, shorter men, usually have lower testosterone levels as well.
I guess an unknown for me, is if someone was belonging to a racial group that typically has larger men with higher testosterone levels, and they just so happen to be unusually short for whatever reason, would that person also be likely to have a lower testosterone level? I don't know if there's been good scientific studies on this yet... If anyone is aware of a good scientific study regarding a link between size/height and testosterone levels, I'd be interested to give it a read.
Anyway, genes effect things. There's been more research recently on human genes... Like, recently some interesting stuff about how subtle differences in genes effecting IL-15R-alpha result in measurable differences in athlete's muscle endurance.
Even if there isn't a link between height and testosterone levels, the shortest racial groups have lower testosterone levels, which would drive short guy aggressiveness down on average anyway.