Minister: Migration is a "small part" of housing crisis