Pence's Secret Service agents called family to say good-bye on Jan. 6