Fans banned from Tokyo Olympics