Those who are against the affirmative action ruling don't really want diversity