Sure, their background is relevant. The problem is that this background is often used as a reason for why we should believe it when a person of color says a particular white person has done something for racist reasons, when in fact, no one's background qualifies them as mind readers. If you were raised in a black family where your parents told you that white people are generally racist, and that they would never accept you as equals, then one day you find yourself walking down the street in a white neighborhood and you notice some white guy eye-balling you, this background might explain why you infer that eye-balling as racist. Yet it doesn't make it actually true because there are a 10 reasons someone might be looking at you. Just like there are other, and more obvious, reasons someone would discard a banana than trying to make a racist statement. The question is, how much over-sensitivity do you tolerate based on someone's background? I think there's got to be a limit. Were these people repeatedly pelted with banana's by racists when they were kids? That might make it more understandable, but I suspect not. I suspect, rather, that that have been told to see racism wherever they look, and that is the real background here that is making them so sensitive. And that, in and of itself, is a larger problem.
Your last question is just silly. I could say the same thing about any number of things you or anyone else talks about on this board - it doesn't directly affect you. Which doesn't mean you have no opinion on it.