Both parties “live in different realities” on DEI, according to Ashley Allison