Jennifer Garner was told to think about retiring