Jimmy was an admired advisor to the Royal Family