I think that's part of what's tricky about looking for a sort of logical defense of racism. As if in order for someone to be acting in a racist way the racism has to be logical when it's not. When it comes down to it there simply isn't any real difference between races genetically, or people of different sexual orientations, or people with an ability to do magic or see compare to those who don't. There's the difference itself (one person can see and the other can't) but this doesn't lead to more fundamental differences (the person who can't see is also incompetent or whatever). I think what more often happens it people see the superficial difference and add more important differences to it.

But that also makes it much harder to find a really logical thought process behind the racism, because I don't think there always is one. It can just be a feeling.