Bill O'Reilly: "Skin color really wasn’t much of an issue" in the past