Man concerned that ancient castle wall will fall on his house