Tree Swallows moved in

We put this birdhouse up last fall I was happy to see tree swallows move in, I was not happy to see the house sparrows tried to take it over, I removed the nest and one house sparrow egg, the swallows came back now they chase away the house sparrows.