Royal Family 'were eager to catch up with Duke and Duchess of Sussex'