That's not being fair, that's rationalizing your own problems and projecting something not truthful onto the rest of us.
To be truthful, many of us are not racist at all.
When you've watched people die violently for no other reason than their skin color, as I often have, when you've been beaten and left for dead because of your skin color, get back to me on how that tracks to what you think you know.
I predict you'll either learn to hate the skin color of those responsible for the violence, or you'll perhaps learn a little bit more about racism, discrimination and prejudice than you know now.