Luxury Real Estate brothers recently arrested for sex trafficking had own room at Hamptons Castle