Grandfather went and boarded the plane after all the prisoners had already left