If it makes it easier for you to understand, think of it as small-r "racism" rather than capital-R "Racism". It's not equivalent to lynching somebody, but it's still an assumption based on race that harms someone. One doesn't have to harbor ill intent in order to exhibit racist behavior.
One shouldn't think of oneself as the center of this situation. Your dubious appeals to evopysch relate to what's going on (or not) in one's brain, while one's behavior that is observable to others just says "black people are subordinate".