Most of the castle's buildings were destroyed