Love is Blind star explains why cast members were afraid to leave the show