Former Prime Minster tells anecdote about a moment at a party with Queen Elizabeth II and some ‘sli