Riggleman: People have called me 'traitor' for calling election fair