The problem is some people are too concerned with not being labeled a racist or bigot