You can't assume that, but nor can you assume the black guy had as easy a time as the white guy, either. That's the whole problem with privilege, really: it's an assumption that people in the same positions have always had the same opportunities, challenges and freedoms. Yeah, that black lawyer could have grown up in a middle-class suburb, sheltered from want and danger... but that's statistically less likely, and he's still going to be the one the police pull over because his Mercedes is "too nice."
The fight against privilege is not a war between cultures -- it's an attempt to acknowledge people's actual advantages and disadvantages, whether they're individual experiences or collective problems like institutionalized racism. Many white people have had to struggle to rise up the social ladder, but we can't pretend that American society doesn't regularly force minorities to climb that ladder with one hand tied behind their back.