Were African-Americans Always the Minority Hardest Hit?

Until the 14th Amendment, most African-Americans were denied U.S. citizenship. But what about those races who had been granted U.S. citizenship at one time, only to lose it -- races to whom the 14th Amendment's birthright clause did not apply?